[Question] MaNGOS's atomicity

[NitriX]

Hello everyone,

I'll be quite right to the point, this week some of my coworkers told me about your MMORPG engine and some of them even helped on the project and looked very enthusiast about it.

Well I'm running out of personal projects at home and I'm kinda curious so I would surely give a try.

I went on GitHub and download the tarball; I must say it's quite impressive.

There's indeed few things I don't get straight away as for any projects, but this one's driving me insane so I would like to know:

The incoming packets are probably not queued since I suppose sometimes one can take forever to be proceeded and slow everyone else down...

Like every connection must be a new thread... but then... what happen if two threads try to edit the same variable?

As for example, player1's getting hit by a monster and at the same time, player2 is healing... its HP will get messed up because operations arn't atomic and I saw no mutexes at all in the sources.

What's the magic behind?!

I would really like to help actively.

Thanks for your time,

~Alex

[Xees]

you have been a memeber since 2007 LOLZ

[NitriX]

Well thanks for your help but I already know man

A LOT of things happen in 3 (almost 4) years now.

I used to play with the binaries a while ago but never was interested by the sources by themselves.

My friends at work just remembered me that it could be nice.

In meantime I keep digging but I only saw ACE_Atomic_Op<> only twice or so, and for threading/sockets purpose.

Anyone please ?

Regards,

~Alex

[VladimirMangos]

As for example, player1's getting hit by a monster and at the same time, player2 is healing... its HP will get messed up because operations arn't atomic and I saw no mutexes at all in the sources.

No real parallel damage/heal. All world event proccessed in one world thread one by one in tick pass. All current events in queue procced in tick and thne new tick start.

With default tick delay: 50 msecs. So 20 ticks in sec. + Network lags and caching data at send by network you will se at client its as parallel.

As goal mangos team plan switch to per map thread event proccessing. But anyway all events at one map will be executed in queue.

Someone can say that exist already patches for like per map execution but i only comment: in diff level hackish way...

[NitriX]

Oh thanks! Sounds so promising!

Is that why MaNGOS is always running at 100% CPU, because of the ticking?

I think I would need a more detailed explanation here because in my mind adding events to the queue would still face the same atomicity problem...

...except if the world thread ALSO handle client packets one by one?

So there's no thread created every client or some sort of thread pool?

Feel free to answer me if you can,

I'll give another closer look, thanks again

~Alex

[VladimirMangos]

packets also proccesed in same thread with world events, in other cases you will need lot locks for access to diff structures

Is that why MaNGOS is always running at 100% CPU, because of the ticking?

This is not true in general, only with many player online...

when world thread do tick it go to sleep until tick delay (50msec) pass. So dependent from amount work need do at tick.

thread pool used for receive and queue packets and for DB write queries. Also most player login read queries make async and also prepared data from DB in own treads, except final login steps proccesing.

[faramir118]

Like every connection must be a new thread...

Not true!

In fact, this is a terrible way to write network IO handling for a server.

The extra threads will slow down your program with context changes and consume resources needlessly.

Most importantly, it can create excess latency. A particular client's network IO thread may be stuck in a sleep state for several hundred miliseconds, because the OS controls how long a time slice is and the order of thread execution.

Instead, you should use synchronous event demultiplexing.

This is provided by the select function. (see Windows or Unix documentation)

In short, select asks the OS to block until any of the sockets are ready to read, write, or are in an exception state, or until a certain amount of time has passed (timeout).

When select returns, the read socket list will contain only sockets that have data ready to be read. You can read from them, process the data, and then call select again.

If any of the sockets have data before select is called, select returns immediately.

There are two design patterns for synchronous event demultiplexing:

Reactor - data is queued, then processed synchronously.

Proactor - data is processed asynchronously.

MaNGOS uses the Reactor pattern.

More specificially, MaNGOS uses the ACE Reactor, which can receive events from multiple sources (network, console, etc).

Even more specifically, MaNGOS uses the ACE TP_Reactor, which can dispatch events to multiple threads. However, each time an event is dispatched, the Reactor is suspended - this is creates the same behavior as synchronous execution, so there's no need for thread-safety mechanisms. IMHO, using extra threads in this fashion is wasteful because of context switches.

[NitriX]

faramir118 -you- are my lover! o:

EDIT: Alright, TP_Reactor will dispatch events directly to threads handlers in a synchronously way. I get your point with the context switches, I admit now it doesn't make sense. Thanks again and here I go!

GitHub's waiting for me :]

[NitriX]

Faramir118, could you answer my very last question, here (http://tangentsoft.net/wskfaq/articles/io-strategies.html) it says:

Heuristic 2: Avoid select().

select() is the least efficient way to manage non-blocking I/O, because there is a lot of overhead associated with the function. Most of this overhead is a linear function of the number of connections: double the number of connections, and you double the processing time.

[...]

--> How does MaNGOS deal with it?

Even if it dispatch events, as you said the Reactor is suspended. That will cause the select() loop to slow down on heavier CPU events.

[freghar]

First, using select() isn't the same as "using one select() in the whole project". The most effective way I know about is to create a (possibly dynamic) pool of threads that do select() / poll() and queue data for processing. There are several reasons for that, including bonus features like packet payload defragmentation (since no read() is guaranteed to return a full packet from a network socket). I guess mangos does it similarly, at least that's what mangosd.conf tells.

###################################################################################################################
# NETWORK CONFIG
#
#    Network.Threads
#         Number of threads for network, recommend 1 thread per 1000 connections.
#         Default: 1
#
#    Network.OutKBuff
#         The size of the output kernel buffer used ( SO_SNDBUF socket option, tcp manual ).
#         Default: -1 (Use system default setting)
#
#    Network.OutUBuff
#         Userspace buffer for output. This is amount of memory reserved per each connection.
#         Default: 65536
#
#    Network.TcpNoDelay:
#         TCP Nagle algorithm setting
#         Default: 0 (enable Nagle algorithm, less traffic, more latency)
#                  1 (TCP_NO_DELAY, disable Nagle algorithm, more traffic but less latency)
#
#    Network.KickOnBadPacket
#         Kick player on bad packet format.
#         Default: 0 - do not kick
#                  1 - kick
#
###################################################################################################################

Moreover, I remember Derex implementing /dev/epoll interaction (kernel-level poll()), perhaps it's already within ACE.

[faramir118]

Perhaps he assumes that you want a non-blocking call, and is saying that select() will block for as long as it takes to iterate over the entire set of sockets...

Using the OS-provided select() is much, much more efficient than blocking on a read() call in a bunch of spawned threads. He should have at least said that.

And according to the article you linked, Unix doesn't support the 'better' ways (asynchronous sockets, overlapped IO) - I don't know if that's true, so I assume that is why we block on select().

I'll dig through the code a little more to find out the specifics. Who knows, maybe there's room for improvement in the current system?

edit: perhaps it would be prudent to look at other open-source projects that are scalable servers, and see how their network IO is handled.

[NitriX]

Well actually it's from far better than I expected.

I'm planning to take an advanced class next session on Networking/programming solutions, who knows what I can learn in mean time, but yes. Thanks everyone for the quick replies!

I'm glad to see the MaNGOS community's support is just as awesome as it used to be :]

[faramir118]

That article you linked was wrong about *nix systems.

There are several APIs that accommodate asynchronous IO, and ACE uses them for the Proactor.

I found an interesting paper that does a bit of benchmarking and scalability testing using different IO mechanisms on linux, you can read it here.

I also looked at the code for lighttpd - it uses asynchronous IO, and only falls back on poll()/select() if the OS doesn't support the more efficient mechanisms.