I've experienced this problem with mixers. I think you will find the issue is the way a lot of modern receivers work, especially 2.4ghz ones. They output all channel pulses simultneously, whereas older receivers squirt out the channels in a sequential fashion, one after the other at a 50hz (20ms) frame rate.
The x or v-tail mixer usually has just one microprocessor built in to do the mixing, and it's programmed to read one channel, then the next, mix the two together and output to the servos.
With the newer receivers, the two channels arrive at the same time, and the microprocessor can't read the pulses accurately, hence the twitching.
Some manufacturers have realised this and modified the designs of the mixers to suit (e.g. fitting twin processors, or using devices which can read more than one input at the same time), but you need to ensure you have a mixer that conforms to this.
Either that or just use the mixer with an older receiver.
Other methods of mixing? Use an inbuilt v-tail mixer in the TX (many have them) or make a mechanical mixer on the boat, which is quite involved.