High Performance Tips

[Auntie Mangos]

Hi , this post is intended to help some users of mangos to improve mangos performance Linux systems or at least know some basic concepts that will improve the performance of your server.

Together we can get some tips update and improve this post. (firt, sorry for my english )

The Tips are based in aprox 3000 players online and for 2 separate server, the most important part is.. if you want performance, Core Server and Database server are not the same, not need the same and is not good which are in the same server. Two server connected to GigaLan is a perfect solution =)

Tips for Core Server:

- Networking

Mangos bandwidth consumption: 3000K players aprox 70 mbps, you need a 100mb symetric hosting

TCP Stack: Linux systems have a automatic system to configure self TCP stack but when you have more of 1000k players if posible that you need tunning some TCP and kernel config for better system performance and responses. This is my sysctl.conf for example i use:

http://pastebin.com/JBX8cdbT

- System

RAM Memory: Have two factor of consumption: players online and vmaps, 3000k online and full vmaps = aprox 6GB. Not use swap memory if you have enough ram, use vm.swappiness = 0 in sysctl.conf.

File Descriptors: In some linux distros the limit on the number of file descriptors a process may have by default is 1000 and if you want to have more of 900 - 1000 player you need modify this limits in limits.conf adding:

root soft nofile 4096

root hard nofile 4096

(use root if your mangos users is root )

Processors: In mangos processors are one of the biggest problems, because currently does not support multithreading. Modifying some configuration in mangosd.conf (eg: vmaps, DetectPosCollision or Visibility.*) , you can reduce consumption but sacrificing playability. For now the better solution are ACE MTmaps patch (http://getmangos.eu/community/showthread.php?8896-mtmaps-(OpenMP)) for activate multithreading, but if you want to use it you need ACE 5.8.3, instalation for Debian/Ubuntu:

   > cd /root
   > wget [url]http://download.dre.vanderbilt.edu/previous_versions/ACE-5.8.3.tar.gz[/url]
   > tar xvfz ACE-5.8.3.tar.gz
   > cd ACE_wrappers
   > mkdir objdir
   > cd objdir
   > ../configure --disable-ace-tests --disable-ace-examples --prefix=/usr/local/  #The Game Master tip
   > make -j8
   > make install
   > cd /usr/lib
   > rm libACE*.so
   > ln -s /usr/local/lib/libACE.so .
   > ln -s /usr/local/lib/libACE-5.8.3.so .
   > cd /usr/include/
   > rm -rf ace
   > ln -s /usr/local/include/ace/ .

When you configure compilation of mangos use -DACE_USE_EXTERNAL=1, for example i use for maximize my core performance:

declare -x CFLAGS="-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O3 -m64 -pipe  -msse4.2 -DFD_SETSIZE=4096 -fno-delete-null-pointer-checks -fno-strength-reduce -fno-strict-aliasing -frename-registers" 
declare -x CXXFLAGS="-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O3 -m64 -pipe  -msse4.2 -DFD_SETSIZE=4096 -fno-delete-null-pointer-checks -fno-strength-reduce -fno-strict-aliasing -frename-registers"

cmake ..  -DPREFIX=/opt/mangos -DDEBUG=1 -DACE_USE_EXTERNAL=1

And for stability use MapUpdate.Threads = 1 (2 processors)

Continue in next thead...

[Auntie Mangos]

I think, for stable servers, dont put mtmaps like tip, becouse with the visibility and reloc patch from silver, is not necessary mtmaps.

With the correct configuration in my.cnf and mangosd.conf is possible allocate more than 3000.

I recommend use SAS 15K or SSD Extreme "-E", tune for high workload in mysql, remember that the SSD Disk, in some cases, change the latency from 0.5 to 0.1.

in limits, you can tune too MySQL.

Chears!

My status: 1 day ON with 3500+

In next thread we can see some tips about HDD IO importance too =)

I use Visibility and Reloc patch to but in my case is not enough because i have active all funcionalities:

vmap.enableLOS = 1, vmap.enableHeight = 1, vmap.ignoreMapIds = "", vmap.ignoreSpellIds = "7720, "DetectPosCollision = 1

And with Intel Bi Xeon Quad "Nehalem",

MapUpdate.Threads = 1

and

Visibility.Notify.Period.OnContinents = 1100

Visibility.Notify.Period.InInstances = 1100

Visibility.Notify.Period.InBGArenas = 1100

Sometimes one of cores jump to 90% on 3k ppl online

And maybe mtmaps is not soo much inestable, some days ago we have 27h uptime with this =), but normaly we have 1-4 crashes at day, every time the same (http://getmangos.eu/community/showthread.php?4235-Linux-crash-dumps&p=100427&viewfull=1#post100427)

Best regards.

[kero99]

Tips for Database Server:

- Networking

It is very important that your database is in the same network as your core server and connected to GigaLan, for a better latency and performance you can use same config in TCP Stack that in core tips

- System

RAM Memory: The only factor of consumption is your character db size, is easy... more characters.. you need more RAM or do more cleanning in character db, no more secret (and obviously Ram more faster is better ).

Processors: Is not de more important in DB Server, but not bad have a good processor.

IO: The secret and the most important thing you must know is that reached a point... how powerful it may be your processor or your ram is not important because you will have a bottleneck in your hard drives and all your system will wait the HDD response (yes... here is the game lag). This problem appear aprox with 1000+ ppl in SATA, 1800+ in SAS... RAID 5 or 0+1 can modify that statistics.

The solution if you want a 2000+ system without lag is a better HDD Performance, High Performance models of SSD disk is the better solution for now, and with RAID 1+0... your server can fly =)

- MYSQL Details

MySQL Version: If you want a very stable and powerful DB, XtraDB of Percona is a good solution and very easy to install (https://launchpad.net/percona-xtradb).

You can have a funtional MYSQL server in less that 10 min if you have a Ubuntu Server 10.10 Maverick:

- Edit your /etc/apt/sources.list

- Add two lines:

deb [url]http://repo.percona.com/apt[/url] maverick main
deb-src [url]http://repo.percona.com/apt[/url] maverick main

(this repository is high tested for percona devel and maximize compatibility with a clean Ubuntu Server)

- wget http://www.percona.com/downloads/RPM-GPG-KEY-percona

- apt-key add RPM-GPG-KEY-percona

- aptitude update

- aptitude upgrade

- aptitude install percona-server-server-5.1 percona-server-client-5.1

- Config your password with assistant and ready to use =)

MySQL Tunning: (thank to Homenixx for tips)

Config designed for 2000 to 3000 player online max:

max_allowed_packet = 128M # Allows inserts dumps with bulks up to 128 MB which allows backups / restore faster.

Recomended values:

max_allowed_packet = 1M # Default
max_allowed_packet = 32M # 256 MB RAM
max_allowed_packet = 64M # 1 GB RAM
max_allowed_packet = 128M # 2.5+ GB RAM

default-storage-engine = INNODB # For characters DB tables (dont forgot change your tables to INNODB)

skip-external-locking - Recommended to avoid table locks with MaNGOS running.

max_connections        = 20 -> MaNGOS opens 3 connections. One for mangosworld table, other for characters and one for scriptdev2. Realmd open the fourth connection for logon.
20 connections with more than enough to even have PHP scripts accessing the DB.

sort_buffer_size = 4M -> Recommended for MaNGOS.
join_buffer_size = 4M ->  Recommended for MaNGOS.

INNODB FEATURES

===============

innodb_additional_mem_pool_size = 50M # Values beetween 20M and 50M are enough for character tables and for every innodb database.
innodb_thread_concurrency = 8 # 1 To Processors of 1 or 2 cores. 2 for Quad CoRe.  4 for i7 / Xeon and 8 for Bi-XEON (2 processors).
innodb_flush_method = O_DSYNC # Depend of your HDD
innodb_flush_log_at_trx_commit = 0 # 0 to reduce the I / O of HD, but in case of a fall is to lose data from the last transaction. 1 recommended for losing nothing. (More I/O) 2 Mix beetween one and two.
innodb_buffer_pool_size = 7G # The size of the table of characters plus a margin. ¡¡¡¡ATTENTION THIS IS SPACE FOR MYSQL RAM!!!! which is reserved!
innodb_log_files_in_group = 2 # Recomended for reduce log file access.
innodb_log_file_size = 32M # Size of the log.    innodb_log_file_size x innodb_log_files_in_group.  
innodb_log_buffer_size = 16M # Buffer needed to process...  1/4 del log_size total.
innodb_file_per_table = 1 # Recomended for reduce file access.
innodb_table_locks = 0 # Disable table lock. We do not care that an application crash or MaNGOS crashes too!.

MAGIC FORMULA

=============

MySQL DEDICATED SERVER MEMORY = key_buffer_size + max_connections * (join_buffer + record_buffer + sort_buffer + thread_stack + tmp_table_size)

note: MySQL settings should correspond to the use which gives the server, there is no ideal configuration. Only the server admin, the person who monitors the server every day knows the exactly MySQL settings.

These are basic tips result of my experience. by Homenixx

[hentai01]

I thought that in linux magos had suport to multithreading =O is that right?

[nanounico]

I think, for stable servers, dont put mtmaps like tip, becouse with the visibility and reloc patch from silver, is not necessary mtmaps.

With the correct configuration in my.cnf and mangosd.conf is possible allocate more than 3000.

I recommend use SAS 15K or SSD Extreme "-E", tune for high workload in mysql, remember that the SSD Disk, in some cases, change the latency from 0.5 to 0.1.

in limits, you can tune too MySQL.

Chears!

My status: 1 day ON with 3500+

[Wowka321]

Very interesting. And what CPU on server?

[kero99]

I thought that in linux magos had suport to multithreading =O is that right?

No, mangos linux dont have MT =(

[kero99]

DB Tips updates =)

[freghar]

Just a note about disk I/O - RAID4/5/6 isn't about performance, it's about space. For read performance, RAID1 (or any combination 1+0 / 0+1) wins.

RAID1 can be theoretically (and is) even faster than RAID0 - the reads aren't limited to some stripes, driver can stripe the reading dynamically (ie. it can read last 1KB of some file from the same drive, without the need of seeking and reading it from the second one). It can even reach ~70% of RAID0 write performance - just by writing to both drives independently and syncing data+metadata during idle states.

None of this is currently in the Linux MD driver (it was, I dunno why they disabled it), it can't read from several RAID1 drives in parallel. On the other side, FreeBSD can - it has even several policies for that.

If you need a LOT of drives, you can go for RAID 0+1 with HDDs in couples of 2 drives (forming RAID1). So the setup would be simply RAID0 spanning over <insert value here> RAID1 arrays. When one of the drives fail, simply replace it (or use a spare one) and resync just the small piece.

And about network - if one Gbit connection isn't enough, you can always use two or more cards on each machine and do some form of load balancing (there are IIRC kernel patches for it as well). You, of course, need to use several DB connections to benefit from that - I don't know if mangos supports that.

[Ayane]

RAID 0 is designed for performance, RAID 1 for backups: RAID 1 don't give performances, it have the same of normal disk. RAID0 read the data simultaneous on all disks in the array, the seeking time is compensed by gain and you have only seeking if the file is fragmented, also on SSD disk you are no seeking time.

The default of RAID 0 is if one disk break, all data is loss.

RAID 5 combine the advantage of RAID 0 and RAID 1: the data is stripped and mirrored, total you lose the space of one disk of the array; it is good if you have several disks (minimum 3). RAID 6 is RAID 5 in more secure.

With four disks you have RAID 1+0 or 0+1 (RAID 10 is most).

The default of RAID 5/6 is the parity calculation, so a good RAID hardware is good for RAID performance (better than software).

With six disks or more, RAID 50 is good.

http://en.wikipedia.org/wiki/RAID

[xeross155]

Just a note about disk I/O - RAID4/5/6 isn't about performance, it's about space. For read performance, RAID1 (or any combination 1+0 / 0+1) wins.

RAID1 can be theoretically (and is) even faster than RAID0 - the reads aren't limited to some stripes, driver can stripe the reading dynamically (ie. it can read last 1KB of some file from the same drive, without the need of seeking and reading it from the second one). It can even reach ~70% of RAID0 write performance - just by writing to both drives independently and syncing data+metadata during idle states.

None of this is currently in the Linux MD driver (it was, I dunno why they disabled it), it can't read from several RAID1 drives in parallel. On the other side, FreeBSD can - it has even several policies for that.

If you need a LOT of drives, you can go for RAID 0+1 with HDDs in couples of 2 drives (forming RAID1). So the setup would be simply RAID0 spanning over <insert value here> RAID1 arrays. When one of the drives fail, simply replace it (or use a spare one) and resync just the small piece.

And about network - if one Gbit connection isn't enough, you can always use two or more cards on each machine and do some form of load balancing (there are IIRC kernel patches for it as well). You, of course, need to use several DB connections to benefit from that - I don't know if mangos supports that.

Every database has a seperate connection afaik so it does support multiple, just don't think that's really gonna improve the performance a lot as the load is on the char db.

And with raid a raid10 combination with SSDs seems best, both failsafe and fast.

[The_Game_Master]

A little adition to your ACE configuration. Use "../configure --disable-ace-tests --disable-ace-examples --prefix=/usr/local/" for more performance.

P.S. 5.7.6 is out.

[freghar]

RAID 0 is designed for performance, RAID 1 for backups: RAID 1 don't give performances, it have the same of normal disk. RAID0 read the data simultaneous on all disks in the array, the seeking time is compensed by gain and you have only seeking if the file is fragmented, also on SSD disk you are no seeking time.

The default of RAID 0 is if one disk break, all data is loss.

RAID 5 combine the advantage of RAID 0 and RAID 1: the data is stripped and mirrored, total you lose the space of one disk of the array; it is good if you have several disks (minimum 3). RAID 6 is RAID 5 in more secure.

With four disks you have RAID 1+0 or 0+1 (RAID 10 is most).

The default of RAID 5/6 is the parity calculation, so a good RAID hardware is good for RAID performance (better than software).

With six disks or more, RAID 50 is good.

http://en.wikipedia.org/wiki/RAID

I knew there will be always trolls quoting wikipedia. Maaan, think a bit! World isn't that easy as RAID1 = slow and RAID0 = double performance.

see http://www.tomshardware.co.uk/forum/250390-14-does-raid-increase-read-speed - there's also a guy quoting wikipedia + replies to his post

I personally own a FreeBSD storage server (well, not "I", but the company I work for) and can confirm the "RAID1 is faster for read than RAID0".

edit: And about RAID 4/5/6 - it's true the read is stripped, so it might achieve read performance similar to RAID0, but write performance is a lot slower - thanks to parity recalculations.

[Ayane]

Because you use a controller that enhance RAID 1 for read data like RAID0, it's not the standard of RAID1 and all controllers does'nt have this enhance. I'm not a troll, I referred to standard, your choice is good only for somes RAID controllers, it's not really RAID1. I quote Wikipedia only for more informations and explaination about RAID.

[freghar]

Because you use a controller that enhance RAID 1 for read data like RAID0, it's not the standard of RAID1 and all controllers does'nt have this enhance. I'm not a troll, I referred to standard, your choice is good only for somes RAID controllers, it's not really RAID1. I quote Wikipedia only for more informations and explaination about RAID.

Well the original post was based more on functionality than theory. It's true that "standard" RAID1 is just dumb mirror, but that's only theory, nothing more. You can connect two drives to some very simple hub and duplicate every single I/O voltage change the motherboard sends (and vice versa) - in which case you're gonna have even slower access.

My "version" of RAID1 is really RAID1 - the data are stored as RAID1 specifies, there are no stripes (unlike RAID10). If you mount the mirrored partition via mount(8), it'll behave as you would expect it to.

Wikipedia says

RAID 1 mirrors the contents of the disks, making a form of 1:1 ratio realtime backup. The contents of each disk in the array are identical to that of every other disk in the array. A RAID 1 array requires a minimum of two drives. Although RAID 1's writing process copies the data identically to all drives, a RAID 1 mirror would not be suitable as a permanent backup solution since RAID architecture by design allows for certain failures to take place (e.g., vandalism or accidental file deletion).

-- which doesn't conflict with anything the FreeBSD driver does. It simply uses smart I/O scheduling to read from more drives simultaneously.

Furthermore I can't find any RFC on RAID / RAID1, is there any? Where do you take this "standard" from? Even http://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_1_performance actually DOES mention the performance boost.

[raps]

"RAID1 is faster for read than RAID0".

From what i heard, the reading speeds of RAID 0 and RAID 1 systems are about the same, but yeah, you may be right of course. It highly depends on the application, on the number of parallel reads etc. But in this case i think that the writing speed is rather the problem, and writing is where RAID 0 is faster

RAID 10 may also be a good choice.

[kero99]

RAID 0 = RAID 1 more or less in READ and RAID 0 > RAID 1 in WRITE but RAID 0 is dangerous for all systems (desktop included ) and RAID 1 is not enoght for INNODB write requirements... But... maybe this post is not for RAID discussions ^^

[nanounico]

What about mangos conf...with this 3 tune options anybody can allocate a lot of people!

two things, i dont know why, but for me, larger buffer in mysql, dont be the best configuration, and right now, i'm using query cache disable.

[higi]

hi, i have 64gb RAM + 2xXeon 64bit debian

i have whole mysql on RAMDISK, and i am not sure, how should i setup my.cnf , cause buffers etc

[nanounico]

hi, i have 64gb ram + 2xxeon 64bit debian

i have whole mysql on ramdisk, and i am not sure, how should i setup my.cnf , cause buffers etc

64gb ram?? Wtf!!!

[kero99]

O_O 64gb!!! wow!!, we db server have 24 gb and mysql dont reserve more of 12 - 13gb... you cant see your real mysql ram consumption with htop (not use top, use htop -> apt-get install htop), my real use is 5-6gb. innodb_buffer_pool_size parameter is simple... is (little more) the size of your character DB:

More Character DB size = More innodb_buffer_pool_size

More innodb_buffer_pool_size = Less Performance

else

Less Character DB size AND Less innodb_buffer_pool_size = Better Performance

Best Regards

P.D: Yes i know that my innodb_buffer_pool_size is huge... but not lag for now... i need know my limit ^^, and then.. cleaning characters from db...

[nanounico]

Kero, this option is new for me..

#kernel.core_pattern = core.%p kernel.sem = 512 32000 100 512

what exactly does?

[kero99]

Kero, this option is new for me..

#kernel.core_pattern = core.%p kernel.sem = 512 32000 100 512

what exactly does?

I use this two config by default in all my linux systems :

kernel.core_pattern = core.%p -> Is use to set the default format of coredump: core:%p = core.pid, or core.%e = core.filename

kernel.sem = 512 32000 100 512 -> Is for set a custom limit on kernel semaphores ("Semaphores can best be described as counters which are used to provide synchronization between processes or between threads within a process for shared resources like shared memories"), i never sure if a application need more limits in semaphores or not (ORACLE OR SAP for example need custom config too)... and i always up the default parameters for avoid system crashes.

max number of arrays = 512

max semaphores per array = 32000

max semaphores system wide = 100

max ops per semop call = 512

Best Regards

[freghar]

I use this two config by default in all my linux systems :

kernel.core_pattern = core.%p -> Is use to set the default format of coredump: core:%p = core.pid, or core.%e = core.filename

kernel.sem = 512 32000 100 512 -> Is for set a custom limit on kernel semaphores ("Semaphores can best be described as counters which are used to provide synchronization between processes or between threads within a process for shared resources like shared memories"), i never sure if a application need more limits in semaphores or not (ORACLE OR SAP for example need custom config too)... and i always up the default parameters for avoid system crashes.

max number of arrays = 512

max semaphores per array = 32000

max semaphores system wide = 100

max ops per semop call = 512

Best Regards

Careful about defaults - mysql doesn't need this specific line to "use the defaults" - it can even cause slowdowns/crashes whenever the defaults change (and you forget to update the line).

[dsrgre]

Please explain:

sort_buffer_size = 4M -> Recommended for MaNGOS.

join_buffer_size = 4M -> Recommended for MaNGOS.