speed problems - fileserver - mediaserver setup

[banjomacman]

I'm having speed problems and was hoping to gather opinions on my best course of action.

My setup
1.66 mini
2.0 c2duo mini
2 crystalHD 1gen appleTV
OWC Mercury Qx2 quad enclosure

I'm currently using the 1.66 mini as:
Fileserver - Mercury unit connected with USB
Plex server
Sabnzbd
Sickbeard
CouchPotato
Transmission

I'm running into buffering and pausing problem on the 2.0 c2duo and appleTVs. Does it seem like I'm just hoping for too much out of the 1.66 or should I consider doing the eSata mod? Any thoughts/recommendations welcome. Thanks. Mike.