params.FullScreen_RefreshRateInHz = 60;
This line is after D3DPRESENT_PARAMETERS, should it be 60Hz or 30Hz to work on a NTSC TV?
if the framerate is 60, then on the screen the line 1,3,5,7....will be the first frame, line 2,4,6,8...will be the second frame. Will that be the reason cause the tearing?
Well, experimentation is the key but I would have thought that 60hz would be the right setting for NTSC. The XDK docs go over all that tho, so reading those would give you a better idea.
I would suggest messing around with FullScreen_PresentationInterval and SwapEffect as well. For example setting FullScreen_PresentationInterval to D3DPRESENT_INTERVAL_ONE_OR_IMMEDIATE can cause tearing if the hardware cannot finish drawing the frame in the time specifed by the refresh rate. This would happen if the emulation was eating into the cpu time put aside for drawing the image on the tv screen. Setting it to D3DPRESENT_INTERVAL_IMMEDIATE can also cause similar problems but in some cases fixes screen tearing but causes the sound to go out of sync.
Its all a matter of messing around until you find something that helps but it could turn out that the xbox is not powerful enough, MAMEs frameskipping code is going crazy, etc. in which case no amount of tweaking the display params is going to help you.