Crossing detection works when the baby crosses the lines defined in the settings of the crossing detection. It is a virtual fence and not motion detection. The area inside the polygon and the area outside the polygon is not sensitive to motion. The baby needs to cross the area.
To improve the situation, you could try making the area of the polygon smaller.