Separate network thread?

[ZenVirZan]

Hi all,
I'm writing a game with multiplayer and networking, and I was wondering what is the best way to receive messages sent from a server.
Currently I'm using the main thread for the game, and creating another thread for a network loop which blocks with #gets() and interprets the results, but this seems to react very slowly to new information sent from the server and the multiplayer aspect is very laggy; because the game I'm making is effectively a MOBA, this is unacceptable and game-breaking. Is there a better way of doing this? I know non-blocking IO would probably be better already, but I don't understand how non-blocking IO works, so if anyone could enlighten me as to how that works, that'd be much appreciated also.

 

[Xilef]

This is my kind of thread!

Could you describe your system a bit better? Are you using an API? What language? What do you have access to, etc, etc.

I'm going to assume it's C++ considering you mentioned gets().
If you're having a network thread you need to detect all direct IO with it, so you don't even use gets() to read string.
If you want to keep it blocking then you need to have one 1 listen and read point, if you have multiple then there will be a dependency. If you definitely want to keep blocking but want multiple listen points on the thread, then I suggest you test the socket before listening using the select() function.

There are also two things you should keep in mind, the send buffer and receive buffer. On Windows the send and receive buffer will give you a lot of latency between messages, basically WinSock will wait until it's send buffer is filled with UDP messages before sending them all, so then your server will get a 2 second delay before a load of packets comes through, then another 2 seconds before the next batch, the same happens with listening, so Winsock will only send a receiving buffer to your program when it is filled.
You can set the send and receive buffer sizes so they are just bigger than the length of an average packet, you'll only have minimal packet delay, if you were to set the buffer sizes to 0 then you MUST have a high-speed listen/receive network thread to capture all the messages, a buffer of size 0 will make the operating system instantly pass all packets at your application, read-or-not, so it can overwrite pending messages.


If you are using C++, I am going to plug and suggest my network wrapper library, which is in use at my own university currently; github.com/felixjones/xiSocketLib
It will reduce the amount of code you need to write for the socket interfacing, although currently it is only IPv4 so it's not quite ready for full use yet.

 

[ZenVirZan]

Sorry Xilef, I can't believe I forgot to mention I'm using Ruby with the default libraries (Not RGSS). Sorry about that. Currently I'm using TCP rather than UDP as well, because I haven't ever used UDP before.

I was using #gets with a delimiter, and I ended all of my messages from and to the server with it. I'm assuming fixed packet sizes is a better way to go?

PS: I think my brackets on my #gets confused you, my bad.

 

[Xilef]

Currently I'm using TCP rather than UDP as well, because I haven't ever used UDP before.

That is massive mistake #1, TCP is terrible for games, especially for MOBAs that require fast response. Unless you are developing a slow MMORPG where latency isn't an issue (WoW for example) always use UDP for games.

With TCP packet sizes don't really matter as what you're streaming will be split into TCP chunks anyway, however with UDP (Which you should be using) packet sizes do matter, small packets create overhead and packets above 500 bytes you might as well forget about.

What I suggest is: Switch to UDP, you'll be 400% better off with UDP.
Send a variable sized UDP packet and listen expecting a 500 byte UDP packet.
This last point is for when you have things pretty much working well: Try and shove as much data as you can in a UDP packet, redundant data is still good as long as you can pad out the packet as much as you can with useful stuff (Don't go sending 500 bytes for the sake of it).
Make sure you set the operating system buffer sizes to 512 bytes, just over 500 bytes, so when you do start bloating your UDP packets you'll notice the speed increase. If you want to be daring, set the buffers to 0 bytes while you develop and then set them to just over the size of your largest packet type later on (But as I said, you need a high-speed thread pulling the packets from the OS otherwise it will drop them).

If you don't have access to select to test the sockets before listening, then just go non-blocking, non-blocking will only fail with the OS buffers being size 0 if they aren't on a high-speed thread, non-blocking is fine with an OS buffer of size [packetLen + 4] or whatever.


I'm guessing this is the API you're using: ruby-doc.org/stdlib-2.0.0/libdoc/socket/rdoc/Socket
That API is veeery similar to BSD/Winsock, so you should have access to setting socket options for OS buffers and setting the blocking flag.

Just please promise me you'll switch to UDP!!!!


EDIT: To answer the original question, yeah use threads when you're non-blocking with an OS buffer size of 0 or use threads when you have a complex game, but there's no reason to use threads if your game loop is in no hurry to grab what's on the socket.

 

[ZenVirZan]

I already knew TCP was bad for games - Considering I've been busying myself with all of the extra little details, I figured I'd swap later, and I guess this time is now. I'll keep my TCP connections for the server listings and stuff that doesn't require low latency, and move all of the in-battle messages to UDP.

Due to the nature of UDP and its lack of streaming, how do I differentiate between two connections coming from the same IP address? I'm not sure how UDP works. If I bind a UDPSocket and listen to it, how do I send back the appropriate information? With TCP, you just #print back on the returned stream, but you can't do that with UDP from what I can tell.

EDIT: I am already using a thread with a network loop of blocking #gets and it seems to have a delay of about half a second. Would that be due to inefficient thread usage or is that just TCP? The game sends instantly, and the server receives and replies instantly too.

 

[Xilef]

I already knew TCP was bad for games - Considering I've been busying myself with all of the extra little details, I figured I'd swap later, and I guess this time is now. I'll keep my TCP connections for the server listings and stuff that doesn't require low latency, and move all of the in-battle messages to UDP.

Due to the nature of UDP and its lack of streaming, how do I differentiate between two connections coming from the same IP address? I'm not sure how UDP works. If I bind a UDPSocket and listen to it, how do I send back the appropriate information? With TCP, you just #print back on the returned stream, but you can't do that with UDP from what I can tell.

EDIT: I am already using a thread with a network loop of blocking #gets and it seems to have a delay of about half a second. Would that be due to inefficient thread usage or is that just TCP? The game sends instantly, and the server receives and replies instantly too.

That half a second delay sounds very much like the OS socket buffer I described, check to see if you can do setsockopt for your network socket.

And with two connections from the same address, you can do what TCP does, TCP has a listen server and a client connector, the client can choose to bind to a port or pick a random port (Picking a random port is the best option for this to work), but the listen is on a fixed port. Because the same IP address can't communicate from the same port anyway, the port can be used to identify two different connections.

Here's an example:
Server is listening on port 27000
Client A starts on random port, it is given port AAAAA
Client B starts on random port, it is given port BBBBB
In TCP, both clients will connect to the server, the server will go "Oh, AAAAA has connected, I'll now send directly to AAAAA"
With UDP, you can say "I just received a packet from IP:AAAAA, I will store that information and send my reply to IP:AAAAA"

This solution doesn't even break down with network address translation (NAT), because the routers will map a port in the sense of X:Y you can guarantee that when the client sends on X, the server will receive on Y and when the server replies on Y the client will receive on X. (This problem is totally resolved with IPv6).

In my XI Networking Engine, I had it so the server will tell the client what port they are free to communicate through to simplify and obscure the client identification method, this is what a lot of games currently do (The ones that ask you to forward a range of ports when you want to run a server).
So the Client A would communicate to the server on the server's public port 27000, the server will then reply to Client A saying "Okay, I have opened port 27001 for you, please use that port so I know who you are" and then Client A will send it's packets to that port instead. That method moves the client identification method onto a socket level rather than a port level so you no longer have the risk of NAT routers bugging out (Which they sometimes do).

 

[ZenVirZan]

So I need to do something like this:

• TCP Server runs on port XXXX
• TCP tells client to use YYYY for UDP which is assigned unique for each new TCP connection
• UDP sends from client to server to client on YYYY

Is this what I should do? I will be using TCP either way.
For most games though, I don't need to port forward a range of ports. Do I have to use a port range for UDP?

 

[Xilef]

No you don't need a port range for UDP, you don't need to forward ports at all (Except for the server) and that's true for ALL IP protocols on IPv4 networks, so you'd be doing that for TCP anyway.

If you want to use TCP for the connection stage, go ahead, that part can be as slow as you need it (There will be a 3 second delay while the TCP connection is made before the streaming begins, remember that). Personally, I can't even tolerate this delay as I've seen players just quit a game when it takes more than 1 second to ping a server, I go UDP all the way.

If you do assign a UDP port for each player then you will need to open a range of ports on the server, otherwise the NAT router will translate the port that the server actually uses to something different than the port you put in the packet to tell the client to use.

What I suggest you do is not build a game first but build a UDP client:server chat system, that will highlight all the simple problems to you first so you can solve them before you go too deep into building your game's networking engine.
QuakeWorld is a good example of this, Quake was built only testing on IP LAN networks and highspeed direct line WANs, the amount of bugs that appeared for dial-up users was insane, the entire networking engine was scrapped 100% and re-written as QuakeWorld (A separate binary entirely!) to solve all the problems in Quake that should have been easy to spot, in-fact Quake was the first client server game network model and this failure with the WAN networking demonstrated every reason why a pure client:server game will always fail.


Just for the love of all things true, use UDP for the actual game packets.

 

[ZenVirZan]

Just for the love of all things true, use UDP for the actual game packets.

I definitely will, I promise :biggrin:

. . . otherwise the NAT router will translate the port that the server actually uses to something different than the port you put in the packet to tell the client to use.

I'm not sure how this translation works or what you mean, sorry. :\
Would you mind explaining it to me?

 

[Xilef]

I got a spare 50 minutes so I will explain the problem:
Network Address Translation (NAT) is a small solution to IPv4 address exhaustion.

Instead of each machine having an IP address, the network is split at the router-modem to have an external IP address and internal IP address. 192.168.x.y is typically the internal address scheme.

External is what you use to communicate to the internet, but what if you have 2 machines internally trying to send on the same port to the outside world?

If the internal addresses are mapped to the external IP then you'll end up with IP address A and B internally talking on port 8888 but externally it would be IP address C talking to the outside world on port 8888 twice, which machine does the reply go to?

NAT maps external ports to machine services, so port 8888 on internal address A may become port 1111 on the external address C and port 8888 on internal B would become 2222 on external C, then when the server replies on those ports the NAT can work out which machine originally sent the packet based on the port it came back on and then it can send it to address A or B on port 8888.

Port forwarding disables this for specific ports and you have to decided which internal address all packets for those ports go to, this is done on servers so sending to port 4444 will arrive on a single machine on port 4444 rather than get translated to port 5555 and the router have no idea where it is going to (so would drop the packet).

You will encounter this NAT issue on IPv4 with internet protocols, it's a guarentee unless you are lucky enough to have IP addresses for your home network (which you shouldn't, we ran out of IPs last year).

IPv6 solves this problem 100%, in the future NAT will become less common.

Edit: so in your case the server will say via TCP "HEY, talk to me on port 1234 plz!" and open up port 1234, the client will get the packet and start sending to the server's IP on port 1234, without realising that the router secretly translated the server's address and port 1234 to it's own external address and port 66600, the connection will fail.

So you'll need the server to have it's TCP public port forwarded to it AND all the client ports you want it to use, this is common for all secure network games, you could always take the minecraft approach and contain an application style player id system and have them all talk through one forwarded port, but everyone knows how easy it is to hack a minecraft packet and crash the game for all players AND the server...

 

[Nuri Yuri]

Interesting. And how do you ask the NAT to redirect the port 80 (for exemple) to your computer ? (When someone from the outside ask to connect to the port 80.)

 

[Xilef]

Interesting. And how do you ask the NAT to redirect the port 80 (for exemple) to your computer ? (When someone from the outside ask to connect to the port 80.)

You don't ask NAT, you tell your router to forward the port 80 to a single machine. You basically must have 1 machine to accept port 80 once you forward it, then you'll be able to get past NAT and directly access port 80 for that machine.
Again, it's needed only for the server side, a client can use any old port they want as long as the server can reply on it.

 

[Injury]

I got a spare 50 minutes so I will explain the problem:
Network Address Translation (NAT) is a small solution to IPv4 address exhaustion.

Instead of each machine having an IP address, the network is split at the router-modem to have an external IP address and internal IP address. 192.168.x.y is typically the internal address scheme.

External is what you use to communicate to the internet, but what if you have 2 machines internally trying to send on the same port to the outside world?

If the internal addresses are mapped to the external IP then you'll end up with IP address A and B internally talking on port 8888 but externally it would be IP address C talking to the outside world on port 8888 twice, which machine does the reply go to?

NAT maps external ports to machine services, so port 8888 on internal address A may become port 1111 on the external address C and port 8888 on internal B would become 2222 on external C, then when the server replies on those ports the NAT can work out which machine originally sent the packet based on the port it came back on and then it can send it to address A or B on port 8888.

Port forwarding disables this for specific ports and you have to decided which internal address all packets for those ports go to, this is done on servers so sending to port 4444 will arrive on a single machine on port 4444 rather than get translated to port 5555 and the router have no idea where it is going to (so would drop the packet).

You will encounter this NAT issue on IPv4 with internet protocols, it's a guarentee unless you are lucky enough to have IP addresses for your home network (which you shouldn't, we ran out of IPs last year).

IPv6 solves this problem 100%, in the future NAT will become less common.

Edit: so in your case the server will say via TCP "HEY, talk to me on port 1234 plz!" and open up port 1234, the client will get the packet and start sending to the server's IP on port 1234, without realising that the router secretly translated the server's address and port 1234 to it's own external address and port 66600, the connection will fail.

So you'll need the server to have it's TCP public port forwarded to it AND all the client ports you want it to use, this is common for all secure network games, you could always take the minecraft approach and contain an application style player id system and have them all talk through one forwarded port, but everyone knows how easy it is to hack a minecraft packet and crash the game for all players AND the server...

Not exactly apart of the discussion but...

NAT and PAT in a nutshell...
One to one, many to one.

I still can't believe they didn't have the foresight to see usable address exhaustion as an issue...

 

[ZenVirZan]

Thankyou very much Xilef, that post was fantastic. I really appreciate it.

So I am using port 4545 for the TCP server at the moment, already forwarded. If I did want to take the single UDP port approach, obviously I'd have to forward, say, port 4546. According to this:

"(Address)-restricted-cone NAT
Once an internal address (iAddr:iPort) is mapped to an external address (eAddr:ePort), any packets from iAddr:iPort will be sent through eAddr:ePort.
An external host (hAddr:any) can send packets to iAddr:iPort by sending packets to eAddr:ePort only if iAddr:iPort has previously sent a packet to hAddr:any. "Any" means the port number doesn't matter."
and what you said, all I should have to do is tell the client to communicate on a unique port, and then get the client to use that unique port when sending messages to the server to differentiate between connections from the same network, and NAT should handle the rest, right?

 

[Xilef]

NAT will handle the client stuff perfectly fine as long as the server replies to the port that the client sent on, the server can send on any port and NAT will handle that case, but the server must listen on a public port, NAT breaks here, you need to forward all ports that the server listens on.

Also; UDP and TCP have separate NAT tables, each one has a 16 bit table (I think most routers encode them in 17 bits, maybe I am wrong there) so you would open TCP port 4545 to connect clients and UDP 4545-4555 for a 10 player game or UDP 6000-6010 or whatever block of free ports you want.

Edit: Injury the address exhaustion problem was known for the 80s, NAT and variable address schemes were developed to delay the problem until a solution was fully tested and deployed, sadly IPv6 was ignored until around the early 2000s and was only tested recently, we are now waiting for ISPs to deploy IPv6 to their customers, but ISPs are waiting for software developers to support IPv6, which involves some costly changes to old software that should have been done in the 90s.

When the rest of the world catches up with china and bites the bullet and switches on IPv6 then all worry will disappear.

Remember that the internet was developed for nuclear war in the 60s with the idea that cities can communicate with each other instantly, it was never designed for individual people to have connections, plus Ethernet which is now the backbone to it all was developed for closed networks where broadcasting to every machine available was a good idea (bad idea when every machine in the world is available), the internet right now is a mash of these two technologies, we are correcting the mistakes of the past (which has actually crippled the speed of the entire internet to 50% of it's potential).

 

[ZenVirZan]

NAT will handle the client stuff perfectly fine as long as the server replies to the port that the client sent on, the server can send on any port and NAT will handle that case, but the server must listen on a public port, NAT breaks here, you need to forward all ports that the server listens on.

If that's the case (which I had already gathered from past experience), how does a single-port UDP server differentiate between computers on the same network? I'm keen on making my server single-port, as unprofessional as that may seem.

The way I see it, is client connects to serverip:serverport and tells the server clientip:clientport. How can you possibly send a different clientport if that is the same port of the server in order to connect successfully? That's the part I'm having trouble understanding - Minecraft does it with ease, having hosted multiplayer successfully myself, no special firewall exceptions required. It works fine with TCP, but I just can't get my head around how you pull this off with UDP.

 

[Xilef]

In TCP the order is actually multi-port with some crazy networking level bindings to make sure it all works

TCP the order is:
Open Server on TCP port 27000

Client connects to server:27000 on ANY TCP port
Server accepts the connection and opens ANY TCP port
The Client and server now talk through ANY TCP port

Client2 connects to server:27000 on ANY TCP port
Server accepts the connection and opens ANY2 TCP port
The Client2 and server now talk through ANY2 TCP port

UDP you don't have the network level on your side, the multiple port system would solve this:
Open Server on UDP port 27000
Remember to open UDP ports 27001 to 27011 for 10 potential clients

Client sends packet to server:27000 on UDP port ANY (ANY = 62000)
Server opens UDP port 27001 and sends this data to Client's ANY port (62000)
Client receives the reply on their ANY port (62000)
Client and server now talk through UDP port 27001

Client2 sends packet to server:27000 on UDP port ANY (ANY = 42000)
Server opens UDP port 27002 and sends this data to Client2's ANY port (42000)
Client2 receives the reply on their ANY port (42000)
Client2 and server now talk through UDP port 27002

For the stupid Minecraft approach:
Open server on UDP port 27000

Client connects to server:27000 on ANY UDP port (ANY = 12000)
Server sends a random number (4522) to Client's ANY port (12000)
Client receives the reply on their ANY port (12000)
Client now puts the number 4522 in-front of all packets they send to server:27000
The server will always reply to whatever port Client sends on

Client2 connects to server:27000 on ANY UDP port (ANY = 32000)
Server sends a random number (5722) to Client2's ANY port (32000)
Client2 receives the reply on their ANY port (32000)
Client2 now puts the number 5722 in-front of all packets they send to server:27000
The server will always reply to whatever port Client2 sends on

 

[ZenVirZan]

If you want to send a message to server port 4546 for example, do you have to send it with your own port 4546? If you don't, then I understand.
Does the router figure out which computer to send externalip:port1 and externalip:port2 on its own depending on UDP messages sent from internal computers in the past? Moreso, when externalip:port comes in, how does the network know which computer to send it to, without forwarding (on the client's side)?

 

[Xilef]

I think I see what you are asking.

So with TCP you're using to "connect to server:port" and you let TCP deal with all the random ports that get opened.

With UDP it's:
"Open socket on outgoing port, send to server's incoming port"

So as an example, my socket library has this for UDP:

xiUDP * const udpSocket = xiUDP::CreateOnPort( xiSocket::PORT_ANY ); // Client binds to any random port

if ( udpSocket  ) {

    xiSocket::addressInfo_s serverData;

    serverData.port = <span style="color: #0000dd;">27000; // We want to send data to this server's external port through our PORT_ANY socket

    // serverData.address.protocolV4 = { 123.456.789.0 }; // Server's target IP address

 

    udpSocket->SendBufferToAddress( <span style="color: #666666;">"My Packet", myPacketLen, serverData ); // This will now send from client socket PORT_ANY to server's socket port 27000

}

The server side would be listening on socket port 27000, it will get a packet from the client's external IP address at some random port that has been NAT'd and such (57494 or whatever)
The client's router has remembered that the client's PORT_ANY (Which the OS chooses randomly, in this example let's say 11111) has been NAT'd to 57494, so when the server replies to port 57494 the client's router will go "Oh, wait, I've NAT'd that port, that belongs to this internal machine's address: 192.168.0.2 and this port: 11111 (PORT_ANY's random port).

The ONLY thing that need's forwarding is the server's 27000 port, if that were not forwarded to the server machine's address then the server's router would NAT it and have no clue where the packet was intended to go to so it would drop it.


TCP's connection is actually similar to this, you must connect to the server's forwarded port, but you can use any port that would survive NAT'ing, the TCP protcol will open up new, random ports for both the client and server when the connection is accept()'d and then they can stream down those sockets until they are done. I described before how to mimic this in UDP (Open up a block of ports on the server and send the intended port to the client).


EDIT: If you didn't gather from the example above;
You do not have to send through port 27000 for it to be received by port 27000, you can send from whatever port you want, the sending and replying can survive a NAT as with UDP the server will get the information of what port the client sent it from after NAT'ing.

I can send from port 55555 to port 66666 and the 55555 would get NAT'd to 77777, the server will receive into port 66666 (If it is forwarded) and would be told that it came from 77777, so it would reply to 77777 which would NAT to 55555

 

[ZenVirZan]

I can send from port 55555 to port 66666 and the 55555 would get NAT'd to 77777, the server will receive into port 66666 (If it is forwarded) and would be told that it came from 77777, so it would reply to 77777 which would NAT to 55555

I FINALLY UNDERSTAND :D
Hoorah, apologies for it taking so long. I get it now, though.
When you send a datagram, you only specify the target port, so NAT will specify its own client port when sending the message, which the server then reads as the client port to send to. This whole translation thing had me confused as hell, but I got there.
Many thanks Xilef, you saved me hours upon hours of headaches :thumb:

EDIT: UDPSocket.recvfrom(size) doesn't behave like IO.read(size) in the sense that it must read to the size and then stop, does it? What benefit is there of specifying a maximum length, because if it can read less and tell where the packet ends, what is the difference with setting longer possible length?