Me and F.G. del Cueto have been experimenting with cloth stuff some years
ago and we have encountered exactly the same thing: instability of the
model. Try decreasing the calculation time steps and/or using a higher order
integration method like Heun or 4th order Runge Kutta.
