Jump to content

Understanding the ACE library and server model

Recommended Posts

Seems like you guys sorted out the differences between TCP and UDP, but I think this SO question covers just about everything.

I just want to highlight the fact that there is no noticeable latency difference between [TCP with nagle disabled] and UDP.

Also, the RC4 stream cipher used by WoW requires reliable and ordered transmission of packets, which are built into TCP. You may find a similarly compelling reason to use TCP in your projects; it just depends on what you are doing.

Mangos is Reactor-based, but I tried very hard to convert it to Proactor-based. ACE's support on platforms other than Windows just isn't good enough for MaNGOS. You can read about my attempt in this post, and look at the code in my repository.

Link to comment
Share on other sites

  • 42 years later...

Hi everyone.

I've recently gained interest in developing a game server and client from scratch. I don't plan anything big as of now, but I want the server to be scaeable to lots of clients. I have done some research and I see that ACE is a good networking library and that Mangos uses it. I've seen that there are a lot of ways of doing this but they have their strengths and weaknesses. Since mangos is written for an MMORPG and as such can handle lots of clients, I'm guessing the model which mangos uses to handle the connections and messages between the server and client is exactly what im looking for to learn. The thing is, there is too much code and the specific part I need to know is hidden there somewhere. So maybe if someone could explain, what model does mangos use? and how does it work?, i would appreciate it very much.

From what I've read, one can make a thread to listen the network and enqueue the messages so that other worker-threads read them when they have time to process them. I think this is called the half-synch/half-async model. I think this model uses a reactor in the listening thread and it blocks as soon as a message arrives, then enqueues the message and unblocks to start listening again. Another way of doing it was with a proactor instead, which is supposed to read and write asynchronously and as such, it won't block the thread so it can be implemented in the main thread, correct me if I'm wrong, I don't understand this method very well. Other models include: having a thread pool (which I don't understand either), having a listening thread for each connection (which scales poorly), or a single thread that iterates through all connections (which also scales poorly).

What I found in mangos code was it having a SOAP thread, a World thread, a database thread and a network thread if I'm correct. I also saw some reactors in the code and some enqueues and message queues, but i couldn't figure the model out entirely since lots of methods are implemented in other header or cpp files (and finding them can be complicated) and also because of the extensive use of defines which also is hard to find without an IDE. I didn't find a visual studio solution file (which is the IDE I'm used to) and I'm not sure what IDE are the files supposed to be compiled under windows.

I also read that for this type of applications (real-time with lots of clients, specially MMORPGs) UDP is better than TCP. I'm not sure if this has anything to do with the model I choose to implement, as in UDP can only be used with a proactor and TCP with a reactor or something like that. and if Mangos uses UDP or TCP.

I know that probably I should be looking for ACE tutorials, but i've done that already and there aren't as many, and the ones that do exist, are too generic. Since mangos is something very similar to what I want to do, and has already been developed, then I'm guessing having help from you guys would be very straight forward to my problem instead of generic tutorials which just confuse me even more.

So thanks in advance and sorry for my long post


PS. English is not my native language. Excuse me if my English is bad.

Link to comment
Share on other sites

You need more generic tutorials, not less. You don't seem to understand what UDP, TCP, proactors and reactors are. You should make sure you very clearly understand all the underlying concepts before trying to code anything.

Mangos has a main thread (the World thread) along with support for map threads that can handle certain packets, one or more networking threads, and a SOAP thread to handle SOAP requests.

A more optimal design (and probably what *lizzard uses) is to daemonize the maps into separate processes, along with any tasks that are not map specific into a separate process, and have a reverse proxy for the realm that users connect to which forwards packets to the various processes. Obviously it's far more involved, but it solves a variety of problems (load balancing, cluster support, cross-realm battlegrounds and dungeons, can support most players even if one node fails, etc.).

WoW only supports TCP, so mangos does as well. UDP is good for broadcasting packets to many clients at the same time, as well as high speed connections (sending lots of data) but for an MMO reliability is far more important, and the bandwidth of each connection never exceeds a few kilobytes per second, at most.

If you want to talk in more depth, send me a PM and I'll give you my IM address.

Link to comment
Share on other sites

Correct me if im wrong.

TCP. it is a protocol that ensures that the packets arrive and arrive in order if a package doesnt arrive it resends it and all. so the protocol ensures the comunication is reliable. and it has a connection established until closed.

UDP. it is a protocol that sends datagrams. the packets in these protocol can arrive, not arrive and arrive in different order that it should. this protocol will not resend lost packages and it is responsability of the application using this protocol to ensure the data is complete, reorder it and ask for a resend if data is lost.

Reactor. it is an event handler (well not really but sometihng like that) that waits for an event to happen (ask for a read or write in a socket for example) and when it happens it invokes the actual event handler to deal with the task. since it is waiting for the task to arrive, it inherently blocks the thread it is running on until it has done the task. if a reactor is going to be used it has to be used in a separate thread than the main thread so the main thread can continue. in this case to calculate the world state every "tick". and one model for this is the half-synch/half-asynch which uses an intermediate layer of a queue to comunicate the synchronous and asynchronous layers.

Proactor. it is like the reactor as in that it handles events. but instead of waiting for a I/O request, the synchronous thread initiates the I/O operation asynchronously and continues its work. when the asynchronous task is done, the proactor is trigger and invokes the real event handler that tells the main thread the task is done. i don't know of a model of how to use the proactor in this case but im guessing that since it is asynchronous inherently, you can just use it in the synchronous thread and let the OS deal with the asynchronous tasks. but i'm guessing i am wrong since it cant be that simple xD.

I'm somewhat shocked that mangos uses TCP but im guessing it is possible. i would like to understand certain concepts like what do you mean by packets, map threads, and what is SOAP. i mean i saw the thread but i don't know what it is. i also saw the cliRunnable thread that handles the console if i am right to stop and start the server from the computer itself. also i don't understand the concepts of daemonize and reverse proxy.

i find your post very useful and i also want to know if mangos takes care of everything, or if it lets the client do some of the decitions? i mean, the server should be the one that takes all the decition to avoid cheating like having infinite life or mana, etc. but I can't imagine a server analysing that much variables for every character at the same time. mainly due to the massive amount of processing involved and because all the results of the processing must be sent back to the client ASAP.

I read something about quake 3 sending delta states in UDP which supposedly only notify the client of changes and there were compressed to avoid sending all the variables that the client already knew. so my question is, how do you define a delta state? does mangos uses something like this? or those it resends all the information every "tick" of the game.

other thing that bothers me is how can one synchronize the server timer with the client timer? is there any lag compensation or anything server side? i know in this case the client code is closed so we can only assume if there is any kind of lag compensation or prediction client side. but server side we should be for sure.

Thanks in advance.

Link to comment
Share on other sites

I'm somewhat shocked that mangos uses TCP

Again, there isn't a choice to be made. WoW only supports TCP. As well the overhead of TCP is next to nothing on modern servers, so unless bandwidth is a giant issue (for WoW it isn't) it's perfectly fine.

Your definitions are mostly accurate. The details of reactors and proactors I'm not too sure of myself, everything I've read on proactors has been CS research paper type stuff, so I lose interest quickly.

Nothing the client sends can be trusted to be accurate at all. All data is assumed to be wrong and must be checked for accuracy (the first rule of client-server security). For WoW, client-side anti-cheat exists, but it is not perfect and it cannot be restored for 1.12 (which is what I work with). Unfortunately the original authors of mangos did not check every packet, and there are still crashes and exploits which exist because of that.

I would not be surprised if Quake 3 does a lot of clever things to improve performance. A delta state is just something that has changed. I haven't dug that far into WoW's protocol to know if it does (I think object updates only contain fields which have changed).

AFAIK there's no lag compensation, nor is there really a need to be. The client will continue to work happily even if it is completely disconnected from the server; you can still walk around and even cast spells (though they never complete and obviously the server's world isn't affected). It relies on the TCP connection to make sure everything is reliable. A small amount of lag when other people move or when you cast a spell isn't going to have a dramatic impact on the game.

Link to comment
Share on other sites

i see. well then, i just need to know what aproximation does it uses to handle the TCP connections. i doubt that it uses a thread-per-connection. and i am almost sure it uses a queue to synchronize or comunicate the asynch network tasks with the synch world tasks. but i have to be sure. thanks a lot patman

Link to comment
Share on other sites

This topic is now closed to further replies.
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue. Privacy Policy Terms of Use