Internal/External Routing Issue? Gameservers

[#1]

Hey,
I'm running a left4dead2 server from inside my home network for myself and a few friends. The server runs it's own software firewall (on Ubuntu Linux), which port 27105 TCP and UDP is open, and the port 27015 is forwarded to that machine from the router.
When I play alone from a different machine on the same network, with friends outside of my network things are all fine and dandy. When myself, and another user play from inside my network everyone's connection to the server experiences extremely high latency, 999+ms. Any ideas?
Router Model: TG582n
Firmware Version: 10.2.5.2
Help appreciated!
Thanks

[#2]

What happens if your friend plays from the internal network and some others connect externally? Do it play fine then or not?

[#3]

Hey Chris,
Thanks for your reply.
If one user on the internal network plays (connecting via the external IP) it's fine. If two users from the internal network play (connecting via external IP) then that's when things freak out.
The same applies regardless of whether external players are playing.
Regards,
Chris.