I tried double precision in the shader, which only works for sqrt, and it had no effect. So that might zero it in on the cos/sin.
I actually think the float type has higher precision than the NVidia card, but I can't easily confirm that. But the ellipsoid math is producing an incorrect result.
Note that the angles are all between -PI and PI. Most experiments are between 0 and PI or 0 and PI/2.