You should limit the cost calculation because, during optimization, some parameters may be supplied with values that drive the system into instability. The resulting large cost value can cause the optimization method to fail to converge due to the limited range of floating point numbers.
When limits are used, they must occur before the integration of the square of the error so that onset of saturation is numerically reflected in the cost function. In this way, onset of saturation is reflected in the cost value and gives the optimizer a slope to follow down.