XI Networking Engine

[Xilef]

Hello everyone.

Alright, so this project is a network server with a client API, mostly inspired by Wyatt's thread of what is wanted in a new NetPlay system, however I'm not doing the Play system, I'm doing the Net. Hmm.

First screenshot (Running in the background of an instance of Battle World RPG)

I now have both a client and a server running in the background of BWRPG, so the next step is to get them both swapping variables, then it's back to architecture design (Suggestions please!)

Current Progress:
Todo:

• Network Engine architecture design, read the red text below

Finished:

• Variable stack
• Console logging
• Config reader
• UDP packet sending
• Port to BSD Sockets
• Packet architecture

!Feature Wish-List is being built! Reply to this thread with what you want out of server software, imagine you want to make an MMORPG or online FPS game, what features do you want? This will help me design the architecture to be flexible and give me some test cases. Even if you're not making an online game currently, any suggestions of what you would want as either the developer or the player would help.

I'm developing a local server-client system right now, for offline single player, and then later I'll be plugging in BSDsock and WinSock which will, if done right, enable online multiplayer.

So things that this project will produce:

• Multi-platform server - Windows, Linux, Mac OS X
• C++ (And maybe later C) client library
• C Windows DLL (And maybe later a Linux SO and OS X DYLIB)

The plan is that someone can write a hook system that pushes variables between their development platform and the client API.

Example:
Someone could write a system that pushes RPG maker's RGSS variables to and from the client API, so the client can tell the server what state RPG maker is in for that player and then the server can run a script/library/dll/something that knows how to process that variable, before sending it back to the client who will push it back to RPG maker.

Actual:
I'll be using this in my custom RPG engine to write the singleplayer + multiplayer framework, so a lot of the systems will be based around my primary need for a single-player, offline server model for managing maps and battle scenes, later on I'll be adding real-time multiplayer where players can connect to a network server and join a custom map.

Planned Features (List to be expanded upon below)

• ☑ Config file reader - Used first for loading map data
• ☐ DLL Loader - For defining how the server deals with variables, one could write a DLL Loader that gets information from a web server
• ☐ Lua script Angelscript interpreter - Can be used in stead of the DLL Loader for the server logic, or used only for simple stuff like dialogues and menus
• ☐ Movement prediction (Anti-lag)
• ☐ Player management (So the Root admin is the server itself, able to kick anyone, all other management levels are custom designed, so you can have classes like admin, moderator, super mod, player, newbie, your best friend)
• ☑ Variable read/write - Read and write player data, server state, stuff like that.
• ☐ Hearable, spacial hulls for selecting which players get packet updates

Nice things to have

• RCON - Remote connect from another computer or as another client, perhaps log in via your smartphone to check on your server, who knows? Maybe this will have to be custom coded.

Serious things that need considering
Because the server software is open for modding with custom scripts, there will be servers who could modify the game in harmful ways, so there might be a server with a custom DLL that asks you to register, but they might write code so the email, username and password you write is store in plain text for them to read, or even worse, a server that asks for your email but then automatically pushes it to their web-server to sell to spammers.


What kind of games could use this?
The RPG genre is what I'm going for, so it will be built for RPG games, that includes action RPG (Multiplayer Zelda?), dungeon RPGs (Diablo), MMORPGs (World of Warcraft or Ragnarok), latency compensation will be added (It's a requirement) so it can still work for FPS games or MOBA games.
Generally, I'm designing it so it can be used with any game you want to make, be it 2D, 3D, a multiplayer text adventure, a fast-paced FPS game or an RTS, so if you can script it to use the API, it can be done.

Maximum player connections at the moment is 65536 256, if you're building an MMORPG that will have more than 65536 256players then I suggest splitting up the load with multiple servers.
EDIT: It's now 256 until the system is running fine, then I'll bump up the specs to a higher number.

Don't expect
I won't be personally writing an RGSS plugin for RPG Maker, I'm going to see if I can work with a scripter later on to write a basic system for multiplayer RPG Maker maps.

EDIT: I have been testing out hooking with RGSS, I can probably create an API for it, but hooking it with RPG maker I'll leave for the next guy to do.

I am building a sand-boxed, single player iteration first, meaning no networking for the first version, this is to build up the architecture. Network sockets will be added afterwards, so no instant miracles, this requires thought and planning.

Important note:
This system will most likely be used as part of my university projects when I return to university, as such an open source version is improbable, which is why I am going for a library/API approach to hooking it up.

Example Config file-format

npc

{

x = -96.000;

y = -128.000;

angle = 225;

model = "models/players/soldier.obj";

texture = "SOLDIER";

script = "dialogue/npc/soldier1.script";

}

Cool ideas:

• Writing up a new multiplayer API for emulators - a new kind of Kaillera
• NetPlay++ - RPG Maker NetPlay with a new network architecture and a C++ portable server
• Porting an open source software to use this networking engine
• A community of DLL/Scripters who create custom scripts for server management that can be used for any game engine that uses this networking engine or create DLL plugins for different game engines so they use this networking engine

Please be aware that this networking engine is going to be core to Battle World RPG's level, world, scripting and eventing systems, it's development and testing will be based around Battle World RPG's development road-map and currently I'm at the point where the client|server is required, this project may be affected by development direction changes with Battle World RPG.

 

[Xilef]

I'm getting the first real test version done today using my battle system as a test bed, at the moment the Server and it's logic library are communicating, next I need to get the client to pull that data to render it on their end.

 

[Xilef]

As an update to this project (It's not dead as it's an important module of BWRPG!) I've finished up the console input and command binding for C++, so building a server library is now possible.

I've changed the design a bit, my idea of having a statefull client side that is dump to any network processing is still around however I'm changing how the server treats states, it now stores all player states and on the fly messes with them and tells their owners when things have been changed, so that's an extra layer on server-side.

The original idea was to have all states server side and commands sent to client, but if a slow system, like RPG maker, has a state machine for speed purposes then might as-well move the entire processing to the server in the entire binding, the only issue is that a state-machine reading bind needs to be created, but that could be beneficial as it means the machine won't need to be built on the client-side, it can be defined on the server-side, so that removes the need for client-side definitions.

So that's network layer 1 done, the top layer.
Layer 2 is the processing, the client as I stated should have no processing, with it all being on the server, the command binding is done I just need to set up a packet->command system for this part, the scripting system needs to be plugged in too so once that's done it's time to construct the actual messaging system.

I've also merged the client and server for now, the server will be extracted at a later date but for now it makes a lot of sense to just keep client:server on the same page while things are developed.

Still taking in suggestions on this, I'm bumping the thread entirely to get some notice, read the writing in red on post #1 and make a reply please.

EDIT: Forgot to say, I ported this to BSD sockets the other day on Mac OSX, so OSX servers are now possible.

 

[Xilef]

This mini-project is still alive, just reporting that I have game packet reading done. I have a packet standard set and I now have a class that will read the contents of a packet and build a structure with all the packet data in it, next thing to work on is making it go the other way.

So we have:
Bytes -> Packet Builder -> Packet for read/write

And I need to do:
Packet Builder -> Packet for read/write -> Bytes

I have been thinking about how to release this too, at the moment it is part of my library collection (xiLibs) but it has no dependencies with any other of my libraries, so what I might do is release a DLL, LIB, SO, A for debug, release, 32bit and 64bit as Windows, Linux and OSX.

I am still calling out for feature requests, the packet format is settled but actual network features are still open for wish list ideas.

Spoiler: Packet Format

So we have two type of packets, Control and Game. Control packets are sent by players who are not part of the game server, they are short packets for getting information on the server and for requesting a connection, game packets send commands between client and server, the idea is that these commands only mean something once a connection is granted from the control packets.

Game packets can be Reliable or Unreliable, Unreliable can be lost on the internet and the game state should be fine, Reliable should be used if the client or server needs information. Reliable messages can be lengthy and split into chunks, Unreliable must be short messages.

Packet Types
PACKET_CONTROL
PACKET_RELIABLE_MSG_BLK_CHNK
PACKET_RELIABLE_END_MSG_BLK
PACKET_RELIABLE_ACKNOWLEDGE
PACKET_UNRELIABLE

The reliable messages work in chunks, so you can send a chunk of multiple commands spread across multiple packets, every time a reliable packet is received, the receiver will send back an acknowledged packet with the last received packet id, so both ends can build a picture of what chunks are still missing.

Eg: Server sends chunks A, B, C and D, the client received A and then sends back "Got A", the client then receives C and sends back "Got C", the server will then presume B was lost and re-sends B, the client then received D and sends "Got D", but the message never reaches the server so after a while the server thinks "I know they have ABC, it's been longer than the ping time both ways, I guess they never got D?" so it will resend D, the client will think "Hey it's D again, I already got this so I'll resend that acknowledgement"

Unreliable messages are literally:
"I AM HERE, I AM MOVING TO HERE, I HAVE THIS GUN"
and if the server receives the message, all is well, if it doesn't, oh well - the next packet will show more up-to-date information so that out-dated, lost packet means nothing, these are 1 way messages with no acknowledgement.

Control Types
REQUEST_CONNECTION
REQUEST_CONNECTION_ACCEPT
REQUEST_CONNECTION_REJECT
REQUEST_SERVER_INFO
REQUEST_SERVER_INFO_REPLY
REQUEST_PLAYER_INFO
REQUEST_PLAYER_INFO_REPLY
REQUEST_RULE_INFO
REQUEST_RULE_INFO_REPLY

These are generally short, one-way packets. A client will typically send a connection request, after a timeout period it would resend it 3/4 times until it gets an accept or reject message, if it's an accept message then it will reliably send the make connection game packet with more information tied to it, if it is rejected then it shows the rejection reason to the player.

Server info is literally a "hey, I'm playing the game "Awesome Game" and it's version 300, tell me about your server" and the server will respond with "This is my address, my name, my loaded map, how many players I have (Up to 255) and the maximum number of players I can have (Up to 255), and my game version is 310", if the server doesn't know what Awesome Game is, then it ignores the info request.

Player info is "Tell me about player X" and the server will say "Player X had the name Y, their colour is 210, their score is 6780, they've been on this server for 4000 seconds and their address is 192.168.0.4:54000"

Rule info is "Tell me about rule 'Team Attack'" and the server replies "You just asked about Team Attack, it's value is 'ENABLED'" and the client will get this message and then ask for the next rule: "Tell me about rule 'Cheats'", etc, etc until a list of rules has been completed. This is done in small packets so the server isn't clogged up with building a big list of rules (Which could be infinite size) and then sending them all down a pipe, in stead it's a quick job on the side to handle this client's rule request.

Game Message Types
MESSAGE_CONNECT
MESSAGE_DISCONNECT
MESSAGE_EXTRA
MESSAGE_EXECUTE_COMMAND

This connect message happens when the player actually wants to join the server and has permission from the server to join, it will usually follow with extra information about this player. Disconnect is the same for when the player is leaving the server.

The extra is reserved for something else right now.

Execute command means that there's extra data in this message like "Move me to X Y Z" or something. These messages are lumped together in chunks so a lot of commands can be batched up like "Put me here, give me this gun, make me face this direction and start running forward"

In terms of the API, the idea is that you focus on the message contents for your game and decide if it's reliable or unreliable rather than try and carry extra information in the other packets.

EDIT: I posted about my idea for an RGSS binding here -
arpgmaker.com/viewtopic.php?p=909837

The idea is open for comments and suggestions and I will expand on it in the future once the network engine is in a fully working state.

 

[Injury]

I actually really like this idea...What kind of security features are you going to implement?

 

[Xilef]

I actually really like this idea...What kind of security features are you going to implement?

Do you mean packet-level security (headers, port masking, throttling) or security of the game network (Admins, moderators, chat muting, banning, kicking)?


EDIT: As a brief update, you can now build up chunks of commands to send to the server, so when you fill up one packet it will create a new packet, mark the previous one as a chunk piece and then put that command into the new packet, returning it's pointer so you can handle the new packet.

EDIT 2: Format is now complete, I can construct control packets, reliable packets, unreliable packets and reliable response packets and convert them into byte streams to send across the net, on the receiving end they can be converted into a generic packet and now I need to make some methods to read the contents.

 

[Injury]

Packet-level. I guess what type of encryption and client authentication can be implemented for this project? Just curious :P

 

[Xilef]

Client authentication is two step; control packets are sent with the clients game name and version, the server decides if it wants to accept this client based on these, at this point they can do an IP ban check, if they like the credentials it responds with an acceptance and a unique communication port, the client then has up to the server's timeout limit to make a full connection using the communication port they were given otherwise the port expires and they need to make a new connection request.

In game the idea is that the client can only send commands about itself and be totally blind to what is around it, the server would decide on the actions and give back state data to the client for displaying. Things like starting an item trade would be "I want to do a trade with anyone at x, y" and the server can check if there was someone there at the time of the request (I plan to have the option to store some data over time so historical checks can be made if a client is lagging).

The game packets as of yet have no protection and are pumping out in raw text, but I plan to include a compression technique on commands before they get stacked into the network, it's hardly encryption but it should be fine for packing more data if you want to write your own security layer (which is totally possible).

 

[Injury]

Sounds cool, I can't wait to see what happens with this! I'll be happy to test it if you ever get something up and running in terms of security. Practice makes better and it would benefit both of us I'm sure.

I hope there will be a security feature that will possibly look at a hashed value to ensure repeated commands can't be sent to the server, such as someone looking at a packet where XP or money was rewarded and sending it back to reward themselves after an actual battle or even took place.

Thanks for the insight into what your working on!

 

[Xilef]

I hope there will be a security feature that will possibly look at a hashed value to ensure repeated commands can't be sent to the server, such as someone looking at a packet where XP or money was rewarded and sending it back to reward themselves after an actual battle or even took place.

The packets already work with ordered IDs, so packets from the same user with the same sequence ID as another packet will be ignored, this is also used for sequencing commands, so if an unreliable packet of "start_move north" with sequence 674 was sent but then "start_move west" was received with 673, the server should be able to cancel that move north, run the westward movement and then apply the north movement, it could be difficult to set up though.

Stuff like "give_exp 40" would be server-side, but to inform the client it would be sent as a reliable packet or unreliably as "set_exp value" so no maths is done client-side, I still need to figure out all this stuff, especially the rolling-back of variables, I will have flags for variables and at the moment I have: Normal, Historic, Reversable - Normal has 1 copy of the variable and it gets changed only with reliable packets (Stats like gold, exp, party setup, current weapon), Historic contains a certain number of snapshots of the variable in time (For checking client:client interaction), Reversable can be changed with sequences that are lower than it's self (Reverts to a historic then re-applies the sequences up to the latest one, server-side correction).


EDIT: Had to rewrite some stuff, but now I have my packet format being sent across the net successfully and decoded successfully.

At the moment I have a client requesting connection and the server replying with an acceptance, but nothing more happens yet.

EDIT2: And I have a test program that does this flow:

Client -> Send Control to Port 6000
Server -> Get Control from Port 6000, Send Port 4000, Open Port 4000
Client -> Get Port 4000, Send "Message" to Port 4000
Server -> Get "Message" from Port 4000

EDIT3: While the engine will only report up to 255 users, you can theoretically connect up to 65534 + 1 for the server to operate on, you just need to allocate the space on your network for these users with the server, so when I create the server I need to say "Operate on port 1200, communicate with ports 27000-27032", which would allow 33 users to be on the server at any time.

If you want to, you can change the message system to include user ID and have all players operate through the same port, but this lowers the security, at least if the server has 320 ports to allocate and limits the users to 32, there's a not much chance of guessing a port is suitable for hacking on PLUS you'd need to know what port ranges the server operates with within a 65535 range.

 

[Xilef]

Update! And time for a re-name:
xi Network Engine will now be called...XI Networking Engine!

So XI Networking Engine, XNE for short, has been coming on nicely as part of Battle World RPG's core engine. Defeating latency has been a big goal with this thing, you can only do so much to a packet to reduce latency, so the actual packet latency is a quick and easy fix, the goal after that is defeat packet-loss and to implement latency compensation.

Both of these problems I will solve with a concept I have been experimenting with, which I am going to call smart-variables (I haven't found a name for them online yet, maybe I'm the first to implement the idea?).

So the plan is; you create an integer variable with the class xiInteger and flag it as CHRONO_PREDICTIVE, xiInteger will record snapshots of the integer's value over time and with CHRONO_PREDICTIVE it will, when it is updated, set itself to the next predicted value in it's trend over time, so if the server has sent me 44, 43, 42, 41 and goes quiet for 3 ticks, the variable would automatically set itself to 40, 39, 38 for those 3 ticks, of course if the last two values that the server sent were the same, say, 44, 43, 42, 42 then xiInteger will keep itself at 42.

The next smart variable is CHRONO_COMMAND, where the actions applied to variables is recorded over time, so say for a few ticks the client receives these commands to the number 6:
0001 : + 1 [number is 7]
0002 : + 2 [number is 9]
0003 : * 3 [27]
0005 : + 1 [28]
0006 : / 2 [14]
But then the client receives the missing 0004 snapshot:
0004 : - 8
This smart variable knows how far to roll-back:
0006 : * 2 [28]
0005 : - 1 [27]
0004 : - 8 [19]
0005 : + 1 [20]
0006 : / 2 [10]
And then the client is back on track with the correct value.
Of course, every now and then the server will send a fixed value in stead of a command where the number would be reset, this kind of stuff is for things like moving a player, so the server can say:
X += 4, Y += 0, Z += 1 and the XYZ co-ords will stay on track with the server every-now-and-then reminding the client where the co-ords exactly are, this would be it's own "set" command.
0007 : + 4 [14]
0008 : set 7 [7]
0010 : + 4 [11]
0009 : -1 [6]
0010 : + 4 [10]
The alternative to this method is to send the current value along with the instruction, which increases the packet size, the goal is to cram as much information as possible into a single packet, so small instructions are needed to achieve this, adding the 3rd parameter could be costly for large network games.

We then have the CHRONO_FINAL type, which is simply a variable that can only be set by the sender, this is what a normal variable would be, however snapshots are still taken of it for server-side rollbacks, the example used would be: Client wants to shoot someone they are aiming at during tick 0102, however the server is at tick 0130, the server can check back 28 snapshots and see where the client was and who they were aiming at and then re-wind time and apply that client's request, this is actually used in competitive shooters these days, Counter-Strike has the famous "He shot me despite me being round a corner", what actually happened here is the player did actually shoot you and he definitely hit you, however the server got that information late and had to rewind time and apply those changes, it seems unfair to the victim as they believe they got away, but in the grand-scheme of things this method is very fair as it accurately captures what actually happened, it's easier to think of it as "Server was unsure what happened so my client thought I survived" rather than "Enemy was lagging so he got an unfair shot".

The last value is the normal variable, this is for stuff like player names (Which should be updated reliably) and scores/pings (Which should be updated unreliably).

 

[Nachos]

In game the idea is that the client can only send commands about itself and be totally blind to what is around it, the server would decide on the actions and give back state data to the client for displaying.

It basically how an MMO system works, right? The client is only allowed to send input of they keyboard to the server, then it process everything and sends it to the clien for it to display it data...i guess?

 

[Xilef]

In game the idea is that the client can only send commands about itself and be totally blind to what is around it, the server would decide on the actions and give back state data to the client for displaying.

It basically how an MMO system works, right? The client is only allowed to send input of they keyboard to the server, then it process everything and sends it to the clien for it to display it data...i guess?

What you quoted me on was in regards to packetting, so the player can only request trades with whatever is in front of them, or shoot a bullet in a direction from a position, in every other sense of network game the client must know some information about the world to eliminate perceived latency. The pure client:server model does fail with games (Quake 1 showed why and hence Quake World exists), the client must simulate some of the world with the server correcting them with the actual world state, otherwise the game would just pop around like mad with delayed/lost packets.

With MMORPGs, they should be behaving the same as any other online game. I know that with most of the MMOs I've played when there is latency with the server all characters on the screen run forward for a bit, pop back into position and repeat until the connection is re-established, this is the client-side simulation where the client does know something about the game world, it knows the movement vector of game objects.