AR10 ??

[m6scout]

Shooting on the water might not be the stable platform but I still think a semi-automatic .308 is better than a .223.However a rifle you are confident with and can hit a target with is better than one you can't.I would also rely on my flares,fire them off at first sign of danger and even send a few the bad guys way.
There is not much cover against full auto rifle fire on a boat,so I can see your point in stopming the threat before gets close.Taking out the driver and the engine may be a good option.

[buggsb]

For this application I'd personally prefer a .308. While the 5.56 has it's place, I still acn't help think of it as a glorified varmint cartridge.

If you go .308 there are plenty of proven platforms that will be both accurate and reliable, such as the FN, H&K, M1A and of course AR.

[Solomon]

With the AR-15 platform, you have a ton of caliber choices, too. 6.8SPC would do the trick nicely for what you are describing.