Multimangos

[scamp]

after changes in objectaccesor have problem to build mtmaps, anyone have working patch?)

There's must be a bit of work to get mtmaps working after theese changes.

So, I will try to update the branch a bit later.

Hope, they will be still working.

[higi]

There's must be a bit of work to get mtmaps working after theese changes.

So, I will try to update the branch a bit later.

Hope, they will be still working.

thx, i just reverted , have 8CPU, without this patch is used only one and "half":(

GL

[rilex]

can you tell me how much it supports per process?

If the process is 64bit (like Mangos can be), 8TB. More memory than you'll have installed for a little while.

[Alex]

First, short lines about AWE/PAE/3GB discussion.

a) PAE enables system to address more than 4Gb of physical memory using CPU VMM page table extensions. It does not enable single processes to address more than 4Gb of RAM easily because pointers stay 32-bit. But multiple processes can, because each process has 'its own' 4Gb address space.

b) 2GB ms limitation is because they split 4Gb process address space into 'process' 2Gb and 'system' 2Gb where kernel/drivers/hardware address spaces reside. /3GB key changes that split into 3Gb+1Gb, allowing more 'process' address space (3Gb) but shrinking system address space which in conjunction with ms resource bloatness can potentially cause a number of problems. Linux has 3Gb/1Gb split by default and more efficient system resources management.

c) AWE is like EMM for DOS. You define paging address space and then 'page' additional memory into that address space in your software. Good for large granular blocks of data, but totally useless with random allocation. MaNGOS uses second. The only real uses for AWE I see are multimedia applications operating with huge granular blocks and SQL servers operating with huge granular tables/indexes. Nothing else.

---

Then, about multithreading/multiprocessing. Multithreading is on the way (mtmaps), and multiprocessing was considered some times but stalled at the concept level. Personally, I opt-in for multiprocessing+multithreading. Let me elaborate:

a) There are NO real reasons not to combine. MTMaps are on the way and this IS multithreading. There is no reason to stop developing MTMaps because we can then utilize it even in multiprocess server.

b) Multiprocess server will offer more scalability and stability. As mentioned, when one map process dies, others keep working. We can even make core process to make users wait until died process restarts without disconnecting, only messaging them about the case and then teleport them to the DB-stored location when process restarts, just like they've entered world again.

c) I see multiprocessing at the map level. One supervisor process maintains all user connections, chats and group/guild interactions (and some more inter-process things), passing this connections to the final map servers along with some internal commands (like: give me state for the player X for storing; or: load this player data and create him on the map). All character DB interaction must be kept in this supervisor process to avoid conflicts. Spawn tables and some other map-dependent tables will have to be split to map-based tables as well to avoid process conflicts.

d) One map server can handle MORE than one map. This is where multithreading will have its way to go. On a single system, you can (as example) create three continent map servers, where first is multithreading az*roth with its instances, second multithread k*limdor with its instances and third multithread outl*nd/northr*nd and their instances. Or even single server multithreading all of that if you want the 'old' scheme. This is the combined multiprocessing/multithreading I mentioned.

e) IMO, with the aforementioned way, no standard solutions (I presume) like CORBA/TAO can currently be used in MaNGOS multiprocessing. They will require a completely different framework and rewrite of the entire system. I see multiprocessing at the network level, where player sockets end on the supervisor and then are proxied to map servers being extended with system commands. We will need to extend server protocol a bit, adding some system commands which exist only between supervisor and map servers.

f) The protocol way is (not complete but gives understanding of my point of view):

f1) Supervisor server receives i.e. guild entrance or chat command. Parses command data. Executes it. Answers all affected players with the result packets. There is nothing to be done at map server.

f2) Supervisor server receives i.e. movement command. Parses command header. Passes it completely to the map server of the player in action. There is nothing specific to be done at supervisor. Map server answers player with all needed data.

f3) Supervisor server receives i.e. group join command. Parses command data. Executes group operation. Sends player group data. Passes the command to the map server, because player locations (players see each other in group) must be sent too. Map server announces players with location data.

f4) Supervisor detects player join. It loads a player, answers him with the initial state and teleport data. Then it creates a map server connection and passes map server a special system command of loading player along with the data (map server does not interact with chardb). Map server creates player on the map and passes him a location data (creatures, objects etc.).

f5) Supervisor detects player leave. Performs group operations (possibly instructing map server to process some data too). Requests map server for a volatile save data for the player leaving. Saves into DB. Then commands map server to remove player from a map.

f6) Supervisor wants to save player. Requests volatile data from map server. Saves into DB.

f7) Map-to-map transition. Map server detects map-to-map transition and commands supervisor to transport character. Supervisor saves character, instructs source map server to remove player. Instructs target server to load player, the the target server sends all teleport/state commands needed.

... and so on.

g) I still don't know if we want to have multiple connections per-player to map server, multiple connections per-thread to map server or a single connection. Single connection is probably not an option, because it will not allow for multithreading. Connection-per-network-thread is better, but will require severe supervisor-to-map-server protocol changes. Connections per player will allow to use client protocol with some tiny extensions, and is easier to code. I would opt for a per-player connections between supervisor and map servers at first and change it later if per-player becomes a bottleneck on the socket level.

h) The drawback is clear: supervisor will have to be rock-solid stable, including all group/chat/guild/load-save/etc. operations, or else it will ruin all the connections. But these operations are not so complex in comparison with the grid/object/spell/aura management systems. We also can have a two-level supervisor. One is a mere proxy (connection supervisor) which simply stalls all the connections if the general supervisor is down and then passes all the connections to the newly-born general supervisor when it goes up. But with that scheme, the general supervisor will have to have some state-gather system to collect data from all the map servers in case it crashes.

[DasBlub]

i agree with alex, there is no reason to not combine both.

but just one thing to add: in your explanation you've said, that the 'master' (you called it 'supervisor') will allways recive the player packets and then redirect them to the map process/server. i think, it would be probably better to say, that there is a master map server which recives the player packet the first time and then redirects this to the map server where the player is. when then the map server detects, that the player is leaving his map, he gives back the map control for this player to the master map server.

but your idea is really good. hope we'll implement it on such a way

[Derex]

First, short lines about AWE/PAE/3GB discussion.

e) IMO, with the aforementioned way, no standard solutions (I presume) like CORBA/TAO can currently be used in MaNGOS multiprocessing. They will require a completely different framework and rewrite of the entire system. I see multiprocessing at the network level, where player sockets end on the supervisor and then are proxied to map servers being extended with system commands. We will need to extend server protocol a bit, adding some system commands which exist only between supervisor and map servers.

I tryed to do multiprocessing before several mounths, and I used raw sockets for this, and from the expiriance there I can clearly say its inpossible to do it without sacrificing a lot lot of maintainability in the code ( and it will be slow if you use all-in-one server ). The thing I did got totally not-maintainable, I ended in receiving a given client packet ,then modifing it in proxy server and resending it to world server and from there modifing it again and sending it to map server which was totally not-maintainable, everything I change required me to change code in 3 different servers. Moreover IF I used sockets then I COULD never bunch all the servers in one executable and take performance benefits of this , I WOULD ever use SOCKETS to connect, IN CORBA you can use collocated servants, I dont know if you know whats this if you dont better don't discuss wheather corba is good or not for IPC.

f) The protocol way is (not complete but gives understanding of my point of view):

f1) Supervisor server receives i.e. guild entrance or chat command. Parses command data. Executes it. Answers all affected players with the result packets. There is nothing to be done at map server.

f2) Supervisor server receives i.e. movement command. Parses command header. Passes it completely to the map server of the player in action. There is nothing specific to be done at supervisor. Map server answers player with all needed data.

f3) Supervisor server receives i.e. group join command. Parses command data. Executes group operation. Sends player group data. Passes the command to the map server, because player locations (players see each other in group) must be sent too. Map server announces players with location data.

f4) Supervisor detects player join. It loads a player, answers him with the initial state and teleport data. Then it creates a map server connection and passes map server a special system command of loading player along with the data (map server does not interact with chardb). Map server creates player on the map and passes him a location data (creatures, objects etc.).

f5) Supervisor detects player leave. Performs group operations (possibly instructing map server to process some data too). Requests map server for a volatile save data for the player leaving. Saves into DB. Then commands map server to remove player from a map.

f6) Supervisor wants to save player. Requests volatile data from map server. Saves into DB.

f7) Map-to-map transition. Map server detects map-to-map transition and commands supervisor to transport character. Supervisor saves character, instructs source map server to remove player. Instructs target server to load player, the the target server sends all teleport/state commands needed.

... and so on.

There shouldn't be any point of single failure. I supose you mean that this supervisor will be actually many supervisors called proxy servers in my terms.

In my idea for multiprocessing will be something like this:

PROXY servers - to these servers players will connect, one realm will consist of one or many proxy servers, and there will be load balancing of their load from realmd ( aka realmd will give different addresses to different players ). Proxy servers will do the communication with the other internal servers.

MAP servers - this will handle one or several maps, it will be SINGLETHREADED !!!!

WORLD server - This will handle stuff like channel chat, guilds, groups, friends and other stuff. If this server crashes, players will most likely stop receiving channel chat or guild stuff for a bit. Its possible to make this server reliable by adding backup server that will maintain the same state as the master WORLD server, but this is too much efford I think.

Now the good part of using CORBA for IPC with all this design is that you can tie up all the stuff in one executable and use colocated CORBA servants, this means that CORBA will do direct calls between the servers ( it wont use TCP, it wont use UDP , it wont use shared memory ... it will do it directly ). This gives a very very powerfull weapon and explains why I wont do multithreaded maps, several map servers tied in one executable will do the same work.

[DasBlub]

is it possible, that a small group develops a prototype for each of the evaluated versions? a prototype which has nothing to do with mangos, just console applications where you have a client and you tipe for ex. "1" and then he connects to server 1 and so on. then we could see, what efforts are needed. just a little benefit and cost analysis

[kamir86]

I think that you shoul use multiprocess to divide problems .. like realm, and mangosd mangosd can be splitted in many processes like mobs handler (mobs can be do using agent programming)..

But multithread is better in other cases for examble in players handler, in packet sending ecc...

[Alex]

IN CORBA you can use collocated servants. I dont know if you know whats this if you dont better don't discuss wheather corba is good

Yes, I will preferably back out from discussing another name for way of optimizing IPC from sockets to shm inside one application ('collocated servants', for short term). Coding paradigms need to be changed for CORBA, and rewriting all application from scratch seems unnecessary in MaNGOS case (we already have all processing we need implemented, and can split it without too many consequences).

We can mitigate single-server drawbacks using packet handler fucntions and/or some OS-specific IPC (ACE framework can probably help us there a bit) for single-server thread/daughter process communication, because packets pass 'proxy' unchanged. Only some system commands may be inserted into stream, nothing more.

I agree that there may be multiple 'supervisors'. But they can severely conflict on database level and will need broadcast communication and tight syncing implementation of can easily overcome all benefits. It's (IMO again) better to create fast packet distribution pipe that can handle great load (multithreading proxy is a way to go, too, we don't need to multiprocess proxy) which distributes packets to resource-hungry map servers that are now and will be the main bottleneck.

Explaining simply: network operation is not the primary bottleneck, we can live with only multithreading here. As far as I know (may be mistaken), bl*zz realm servers use one IP for client income, and multiple map servers thereafter. Why not do it the same way?

It's a pity I did not have time enough to work on the subject and still have no results to present for comparison.

[Insil]

erm its not a problem to have a multiprocessed server framework aka. a cluster where each process runs more threads n fibers...

People never gave this approach any thought in the past, since you only had 1 core per cpu available hehe... It was also back then where unix sucked with threadhandling and therefore multiprocessing was better...

Multithreading is the way to go honestly... Less IPC, less threadblocking, and some fewer headaches

[Alex]

There is one major issue though... multithreading cannot be split between physical servers

[rilex]

There is one major issue though... multithreading cannot be split between physical servers

No, but each executable can be multithreaded.

[rilex]

b) 2GB ms limitation is because they split 4Gb process address space into 'process' 2Gb and 'system' 2Gb where kernel/drivers/hardware address spaces reside. /3GB key changes that split into 3Gb+1Gb, allowing more 'process' address space (3Gb) but shrinking system address space which in conjunction with ms resource bloatness can potentially cause a number of problems. Linux has 3Gb/1Gb split by default and more efficient system resources management.

Please be aware that realmd and mangosd are, by default, have the LAA flag set, with no known issues as a result of doing so.

[Alex]

There are almost no issues with LAA apps, there can be some issues with kernel/hardware, though they are rare to be seen. One of them is VSS issue on W2K3 servers.

[Derex]

The multiprocessing idea would mean this model:

http://www.bigworldtech.com/technology/server.php (the second picture).

With TAO it will be possible to tie all the servers in single executable, which will result in multithreading version ( with the same performance ).

Login Server - realmd

Base Server - to this players will connect (call it proxy server ).

Cell Server - this handles one or several maps ( call it map server ).

World Server - handles chat, guilds, etc.

[thenecromancer]

It was not like one thread doing one map. There are some operations and classes that should belong under map, but they are not. IIRC for example object accessor is not one par map, but one per server, together with all other things this is resulting of need to lock things, and not really using all the power you can have.You probably want your maps to be done by single thread, and being sure that. ( Eg. one object would be accessed by only one thread at time ).

For example, a system with Q6600 quad-core Intel CPU can handle about 1000 of users with multithreading, and not more than 250 without multi, and it lags like hell.

This is true only if you are running something really intensive in your maps, like vmaps. Otherwise I would not go for multi-threading if I did not really know what I'm doing...

[higi]

It was not like one thread doing one map. There are some operations and classes that should belong under map, but they are not. IIRC for example object accessor is not one par map, but one per server, together with all other things this is resulting of need to lock things, and not really using all the power you can have.You probably want your maps to be done by single thread, and being sure that. ( Eg. one object would be accessed by only one thread at time ). This is true only if you are running something really intensive in your maps, like vmaps. Otherwise I would not go for multi-threading if I did not really know what I'm doing...

ye vmaps its really eating CPU , if not multithreaded :(/ with patch from Derex no problem at all, everything runs smothly, but now its not compatible(

Cpu0 : 2.0%us, 2.3%sy, 0.0%ni, 93.7%id, 0.3%wa, 0.3%hi, 1.3%si, 0.0%st

Cpu1 : 3.3%us, 0.7%sy, 0.0%ni, 93.7%id, 0.0%wa, 0.3%hi, 2.0%si, 0.0%st

Cpu2 : 6.0%us, 0.3%sy, 0.0%ni, 92.3%id, 0.0%wa, 0.0%hi, 1.3%si, 0.0%st

Cpu3 : 2.7%us, 2.3%sy, 0.0%ni, 94.0%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0%st

Cpu4 : 8.7%us, 1.0%sy, 0.0%ni, 88.0%id, 1.0%wa, 0.0%hi, 1.3%si, 0.0%st

Cpu5 : 95.4%us, 1.3%sy, 0.0%ni, 2.3%id, 0.0%wa, 0.3%hi, 0.7%si, 0.0%st without patch even with updated patch from hitman/scamp

Cpu6 : 2.7%us, 0.7%sy, 0.0%ni, 96.0%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st

Cpu7 : 3.3%us, 1.3%sy, 0.0%ni, 94.0%id, 0.0%wa, 0.0%hi, 1.3%si, 0.0%st

pls if anyone can fix mtmaps from Derex to latest rev, i will pay for it:)

[eox425]

Yep. I think that multiprocessing is much better, but multithreating is simpier to develop...

I worked with some OSKAR linux clusters which are based on CORBA.

I think that they are the best. (MPI based clusters are stable ... But they work perfectly only without network collisions) and guys from OSKAR community have developed many megaBytes of sources to provide synchronization of all this stuff. ph34r:

I know only one thing - if you start to develop multiprocessing it will be so unstable that users (noob admins and other lamers) will not be able to use it at first time.

And development process will be so hard...

I think that first of all you guys should develop multithreating 'cause it's much simpilier than multiprocessing.

And when you will end it you can develop multiprocessing... :rolleyes: It much better for my purposes.

And I've voted for multiprocessing. :lol:

My English is so bad 'cause I am ukrainian.

Sorry.

[scamp]

The problem is that atm I can code only in the weekends. If I have free time this weekend I may try to port it ... but I deffinitely won't have time to stress-test it.

Thanks in advance!

[Derex]

Thanks in advance!

Infact ... looking at the new code it offers the option to go step further and do Player::Update also in paralel. This will give more performance, but will potentially cause more crashes. After having paralel Player::Update, its possible to differ process some WorldSession opcodes, like movement.

So should I do this or do the old stuff ?

[Lord X-Press]

Infact ... looking at the new code it offers the option to go step further and do Player::Update also in paralel. This will give more performance, but will potentially cause more crashes. After having paralel Player::Update, its possible to differ process some WorldSession opcodes, like movement.

So should I do this or do the old stuff ?

Do the new, backtraces will bring the crash problems up to the surface (like in ACE) and we will have more perfomance *hopefully*

[Wyk3d]

Do the new, backtraces will bring the crash problems up to the surface (like in ACE) and we will have more perfomance *hopefully*

The difference is, fixing the player problems would require a lot more changes to other systems. It should be easy to make a config option to do the updates either in Map::Update or ObjectAccessor::Update though so as long as it's done in ObjectAccessor by default it would be fine.

[freghar]

Whether the multiprocessing or multithreading is slower, that depends on style of communication.

For example multiprocessed Apache web server - it uses many forks, each one for one connection, it handles it separately, without much communication with the parent server, so the performance it's pretty good.

However if the map server would need to exchange a lot of data with other processes, it would be pain. Passing structures to threads is simplier (and in this case faster) rather than sending them through UNIX sockets, TCP/IP conn, SIGPIPE, ...

... and multipocessing isn't hard to use even for noobs, if they have the right tutorial ..

[AandD]

hm new is good, but it would be great you will update mmaps, i dont think what in the core patch related part would be any changes in nearest time.

senk.

[higi]

The difference is, fixing the player problems would require a lot more changes to other systems. It should be easy to make a config option to do the updates either in Map::Update or ObjectAccessor::Update though so as long as it's done in ObjectAccessor by default it would be fine.

this could be really nice, .

Derex, big thx if u update mtmaps, really cant live without it:)( (i will happy for every ver, if its quick fix to update old mtmaps, it would be nice if we get old ones first(i had 2days uptime with them), then after some time with config options:)