Mangos 99% sure have memory leak(LINUX)

[ilija]

Hey , i think mangos has memory leak because the memory keeps increasing over time and not lowering.

Example : Memory usage : 30% , Players : 200 , Server Uptime = 5 hours.

Memory usage = 45% , Players:80 , Uptime = 10 hours.

I have this issue for a long time ! (ON debian , ubuntu)

Some one also have this problem?

Im using newest mangos 9870, Newest ScriptDev2, YTDB.

Any ideas how to fix?

EDIT: one thing can be that fireworks dont disappear if some one fire it!

[rilex]

Nothing you've posted indicates a memory leak.

[caeruleaus]

yeah.... if you posted your conf then maybe we can help you out. If you have gridunload = 0 then that'll do it. We can't really tell whether you have a mem leak just from what you're posting.

[ilija]

Here is my config , ye i had grinunload =0 , so thats reason , sorry for the conclusions i made ;( , but still fireworks dont disappear.

#####################################
# MaNGOS Configuration file         #
#####################################
ConfVersion=2010040601

###################################################################################################################
# CONNECTIONS AND DIRECTORIES
#
#    RealmID
#        RealmID must match the realmlist inside the realmd database
#
#    DataDir
#        Data directory setting.
#        Important: DataDir needs to be quoted, as it is a string which may contain space characters.
#        Example: "/home/luminawow/district9fun/share/mangos"
#
#    LogsDir
#        Logs directory setting.
#        Important: Logs dir must exists, or all logs need to be disabled
#        Default: "" - no log directory prefix, if used log names isn't absolute path
#        then logs will be stored in current directory for run program.
#
#
#    LoginDatabaseInfo
#    WorldDatabaseInfo
#    CharacterDatabaseInfo
#        Database connection settings for the world server.
#        Default:
#                ---MYSQL---
#                    hostname;port;username;password;database
#                    .;somenumber;username;password;database - use named pipes at Windows
#                        Named pipes: mySQL required adding "enable-named-pipe" to [mysqld] section my.ini
#                    .;/path/to/unix_socket;username;password;database - use Unix sockets at Unix/Linux
#                ---PGSQL---
#                    hostname;port;username;password;database
#                    .;/path/to/unix_socket/DIRECTORY or . for default path;username;password;database - use Unix sockets at Unix/Linux
#
#    MaxPingTime
#        Settings for maximum database-ping interval (minutes between pings)
#
#    WorldServerPort
#        Default WorldServerPort
#
#    BindIP
#        Bind World Server to IP/hostname
#
###################################################################################################################

RealmID = 1
DataDir = "/home/luminawow/district9fun/bin"
LogsDir = "/home/luminawow/district9fun/logs"
LoginDatabaseInfo     = "localhost;3306;xxx;xxxx;realmd"
WorldDatabaseInfo     = "localhost;3306;xxxx;xxxx;districtfun"
CharacterDatabaseInfo = "localhost;3306;xxxx;xxxx;chardistrict"
MaxPingTime = 30
WorldServerPort = 8085
BindIP = "0.0.0.0"

###################################################################################################################
# PERFORMANCE SETINGS
#
#    UseProcessors
#        Used processors mask for multi-processors system (Used only at Windows)
#        Default: 0 (selected by OS)
#                 number (bitmask value of selected processors)
#
#    ProcessPriority
#        Process priority setting (Used only at Windows)
#        Default: 1 (HIGH)
#                 0 (Normal)
#
#    Compression
#        Compression level for update packages sent to client (1..9)
#        Default: 1 (speed)
#                 9 (best compression)
#
#    PlayerLimit
#        Maximum number of players in the world. Excluding Mods, GM's and Admins
#        Default: 100
#                 0 (for infinite players)
#                -1 (for Mods, GM's and Admins only)
#                -2 (for GM's and Admins only)
#                -3 (for Admins only)
#
#    SaveRespawnTimeImmediately
#        Save respawn time for creatures at death and for gameobjects at use/open
#        Default: 1 (save creature/gameobject respawn time without waiting grid unload)
#                 0 (save creature/gameobject respawn time at grid unload)
#
#    MaxOverspeedPings
#        Maximum overspeed ping count before player kick (minimum is 2, 0 used for disable check)
#        Default: 2
#
#    GridUnload
#        Unload grids (if you have lot memory you can disable it to speed up player move to new grids second time)
#        Default: 1 (unload grids)
#                 0 (do not unload grids)
#
#    SocketSelectTime
#        Socket select time (in milliseconds)
#        Default: 10000 (10 secs)
#
#    GridCleanUpDelay
#        Grid clean up delay (in milliseconds)
#        Default: 300000 (5 min)
#
#    MapUpdateInterval
#        Map update interval (in milliseconds)
#        Default: 100
#
#    ChangeWeatherInterval
#        Weather update interval (in milliseconds)
#        Default: 600000 (10 min)
#
#    PlayerSave.Interval
#        Player save interval (in milliseconds)
#        Default: 900000 (15 min)
#
#    PlayerSave.Stats.MinLevel
#        Minimum level for saving character stats for external usage in database
#        Default: 0  (do not save character stats)
#                 1+ (save stats for characters with level 1+)
#
#    PlayerSave.Stats.SaveOnlyOnLogout
#        Enable/Disable saving of character stats only on logout
#        Default: 1 (only save on logout)
#                 0 (save on every player save)
#
#    vmap.enableLOS
#    vmap.enableHeight
#        Enable/Disable VMmap support for line of sight and height calculation
#        Default: 0 (disable)
#                 1 (enable)
#
#    vmap.ignoreMapIds
#        Map id that will be ignored by VMaps
#        List of ids with delimiter ','
#        If more then one id is defined and spaces are included, the string has to be enclosed by "
#        Example: "369,0,1,530"
#
#    vmap.ignoreSpellIds
#        These spells are ignored for LoS calculation
#        List of ids with delimiter ','
#
#    DetectPosCollision
#        Check final move position, summon position, etc for visible collision with other objects or
#        wall (wall only if vmaps are enabled)
#        Default: 1 (enable, required more CPU power usage)
#                 0 (disable, less nice position selection but will less CPU power usage)
#
#    TargetPosRecalculateRange
#        Max distance from movement target point (+moving unit size) and targeted object (+size)
#        after that new target movmeent point calculated. Max: melee attack range (5), min: contact range (0.5)
#        More distance let have better performence, less distance let have more sensitive reaction at target move.
#        Default: 1.5
#
#    UpdateUptimeInterval
#        Update realm uptime period in minutes (for save data in 'uptime' table). Must be > 0
#        Default: 10 (minutes)
#
#    MaxCoreStuckTime
#        Periodically check if the process got freezed, if this is the case force crash after the specified
#        amount of seconds. Must be > 0. Recommended > 10 secs if you use this.
#        Default: 0 (Disabled)
#
#    AddonChannel
#        Permit/disable the use of the addon channel through the server
#        (some client side addons can stop work correctly with disabled addon channel)
#        Default: 1 (permit addon channel)
#                 0 (do not permit addon channel)
#
#
#    CleanCharacterDB
#        Perform character db clean ups on start up
#        Default: 1 (Enable)
#                 0 (Disabled)
#
###################################################################################################################


UseProcessors = 0
ProcessPriority = 1
Compression = 4
PlayerLimit = 500
SaveRespawnTimeImmediately = 1
MaxOverspeedPings = 2
GridUnload = 1
SocketSelectTime = 10000
GridCleanUpDelay = 300000
MapUpdateInterval = 100
ChangeWeatherInterval = 600000
PlayerSave.Interval = 720000
PlayerSave.Stats.MinLevel = 0
PlayerSave.Stats.SaveOnlyOnLogout = 1
vmap.enableLOS = 1
vmap.enableHeight = 1
vmap.ignoreMapIds = "530, 369, 1, 571"
vmap.ignoreSpellIds = "7720"
DetectPosCollision = 0
TargetPosRecalculateRange = 2
UpdateUptimeInterval = 10
MaxCoreStuckTime = 120
AddonChannel = 1
CleanCharacterDB = 0

[Endon]

I'm having issues with memory too. I'm Using Ubuntu 10.04LTS and the issues started since 3.3.3 (before I was using 8.04)

Server Hardware:

Processor: AMD ATHLON 64 X2 DUAL CORE 5600+ 2,9GHz 1MB Cache

Mainboard: Supermicro AM2 H8SSL-i2 (link)

Memory: Kingston 8GB DDR2 800mhz (4x 2GB PC2-6400)

Hard Drive: 2x250GB WesternDigital RE2 7200rpm SATA RAID1 3ware 8006-2LP

Load Average get above 5.0 sometimes and mysqld often requires more cpu-power than mangos-world,

USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

mysql 15 0 247m 52m 7628 S 34 1.7 52:27.14 mysqld

root 17 0 725m 564m 10m S 22 18.4 2:22.00 mangos-worldd

The memory gets filled up till 100%. Only thing that helps getting it low again is restarting worldd, realmd en mysql.

As result, there is much lagg in-game (with bursts) and logging in takes up to 5 minutes... It's only running stable with less than 5 players online (the moments just after a restart )

[Diablox]

I'm using Ubuntu 10.04LTS and 3.3.3 mangos using some old celeron and 3gb of ram no problems with 70 "testers" and some player bots online. Try finding out what is forcing mysql to work so hard.

@ilija

You posted your server name mysql username and password.

[Endon]

@above

Do you know anyway to log every query executed by mangos / performed by mangosd? Or do you know an other way to see what's happening with mysql?

Do you also have any functions disabled, like CharDbCleanup etc..?

[Diablox]

Start with something simple. Start with mysqlreport -u username -pass (replace username with your mysq user). Check how much slow queries you have and how big your InnoDB Buffer Pool Read hit. Did you changed your mysql config?

I do not have anything disabled. Vmaps turned on on all maps and compression is on 6

[Unkle Nuke]

There's been enough problems with Ubuntu, since 9.4, that I have to wonder if some of the trouble doesn't lie there. Just look around some of the more popular Linux blogs, like Fewt's.

Aside from possible OS issues, check your configs for MaNGOS and MySQL to make sure you haven't skipped over something like grid unload or set values too high. If you don't know what you're doing when configuring MySQL, it can seriously impact server performance. Bigger numbers don't translate into better performance. With a typical desktop setup used to run a MaNGOS server, buffers that are too large, too many threads per read/write, and so forth can gobble up memory and CPU cycles.

[Endon]

We switched for ubuntu to debian, but we still got this problem..

I ran the mysqlreport script and logged the slow-mysql: result

As you can see, the InnoDB Buffer Pool is constantly 100% full and I haven't found out how to increase this..

Disabling grid_unload has effect for a moment, but eventually the load-average rises again.

[Diablox]

First of all go to /var/lib/mysql i check how big is ibdata1 to see how much memory pool you need. Then open your /etc/mysql/my.cnf and add these lines after [mysqld]

innodb_buffer_pool_size = 256M
innodb_additional_mem_pool_size = 32M

This is just example. Set the first value to be as bigger as you can afford (the best value is the one bigger then ibdata1). Keep in mind that you must avoid to run out of memory so be careful. Second value is recommended to bee arround 25% of innodb_buffer_pool_size.

[Endon]

Thanks, I increased the innodb_buffer_pool_size to 128M (ibdata1 = 106mb)

but still got a issue with mysql. Mysql still requirer alot of CPU power.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

15396 mysql 15 0 655m 139m 6092 S 51 4.5 2:06.13 mysqld

15624 root 15 0 634m 421m 9936 S 20 13.7 1:38.84 mangos-worldd

However, the average load average, it increases when someone logs in, but it slowly goes down again after a while.. Logging in still takes some time (mainly connecting and retrieving char. list)

result mysql-slow.log

Do you also have any suggestions for the normal mysql cache? query_cache_limit, query_cache_size, key_buffer, max_allowed_packet, thread_stack etc?

[ilija]

Wtf , my ibdate1 = 1.3 gb big, how could that come? And should i also change innodb_buffer_pool_size = 256M to bigger number?

My mysql conf:

#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# [url]http://dev.mysql.com/doc/mysql/en/server-system-variables.html[/url]

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port        = 3306
socket        = /var/run/mysqld/mysqld.sock

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket        = /var/run/mysqld/mysqld.sock
nice        = 0

[mysqld]
#
# * Basic Settings
#
user        = mysql
pid-file    = /var/run/mysqld/mysqld.pid
socket        = /var/run/mysqld/mysqld.sock
port        = 3306
basedir        = /usr
datadir        = /var/lib/mysql
tmpdir        = /tmp
language    = /usr/share/mysql/english
skip-external-locking
#
# For compatibility to other Debian packages that still use
# libmysqlclient10 and libmysqlclient12.
old_passwords    = 0
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address        = 127.0.0.1
#
# * Fine Tuning
#
key_buffer        = 100M
max_allowed_packet    = 128M
thread_stack        = 50M
thread_cache_size       = 8
default-storage-engine = INNODB
sort_buffer_size = 4M
join_buffer_size = 4M 
innodb_additional_mem_pool_size = 50M
innodb_thread_concurrency = 1
innodb_log_files_in_group = 2
innodb_table_locks = 0
innodb_file_per_table = 1
#
innodb_log_files_in_group = 2
#innodb_log_file_size = 32M
innodb_log_buffer_size = 16M
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover         = BACKUP
#max_connections        = 100
#table_cache            = 64
#thread_concurrency     = 10
#
# * Query Cache Configuration
#
query_cache_limit    = 50M
query_cache_size        = 512M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the  at runtime!
#log_type           = FILE
#general_log        = /var/log/mysql/mysql.log
#
# Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf.
#
# Here you can see queries with especially long duration
#log_slow_queries    = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
#server-id        = 1
#log_bin            = /var/log/mysql/mysql-bin.log
expire_logs_days    = 10
max_binlog_size         = 100M
#binlog_do_db        = include_database_name
#binlog_ignore_db    = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem



[mysqldump]
quick
quote-names
max_allowed_packet    = 16M

[mysql]
#no-auto-rehash    # faster start of mysql but no tab completition

[isamchk]
key_buffer        = 16M

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

[rilex]

Your INNODB buffer looks fine (read hit at 99%).

[Unkle Nuke]

Another issue could be Auction House Bot. If you have this added to your server, it will cause lag spikes at login and cause high CPU usage with cache thrashing if you have configured AHBot to have an excessively high number of items stocked in the auction houses. Depending on your system hardware and database configuration, try lowering the item count by increments of 1,000 until you notice more responsiveness from your server.

Also...make sure you set minitems and maxitems to the same value or very near to each other. If you have a minitems of 6,000 and maxitems of 10,000 for each auctionhouse, this means your server will have to delete old items and then add new items with each auction cycle for as many as 12,000 items!!

[Diablox]

@Endon

Well your config is good for the size of your DB and number of queries per second. I once had a problem with mysql using much cpu but in my case mangos could not connect to mysql so i do not think this could help you much. Only solution was MySQL reinstall (i remove it with purge). Beside that idk what could be the problem.

@ilija

ibdata1 is file where innodb keeps all data for innodb tables or in mangos case that is the size of characters db (if you dont have any other tables that are using innodb). There is no reason to panic if it is that big. Run mysqlreport and you will se what you need to change in your config.

[ilija]

I have runned, but i dont understand that much from mysql config, i will post my mysqlreport, sec:

__ Key _________________________________________________________________
Buffer used     4.54M of 100.00M  %Used:   4.54
 Current      11.96M            %Usage:  11.96
Write hit      76.11%
Read hit       99.75%

__ Questions ___________________________________________________________
Total          11.04M    25.7/s
 DMS           9.73M    22.6/s  %Total:  88.18
 Com_        959.74k     2.2/s            8.70
 QC Hits     312.03k     0.7/s            2.83
 COM_QUIT     32.55k     0.1/s            0.29
 -Unknown        123     0.0/s            0.00
Slow 10 s          38     0.0/s            0.00  %DMS:   0.00  Log: OFF
DMS             9.73M    22.6/s           88.18
 DELETE        2.91M     6.8/s           26.35         29.88
 INSERT        2.75M     6.4/s           24.91         28.24
 UPDATE        2.36M     5.5/s           21.35         24.21
 SELECT        1.40M     3.3/s           12.72         14.43
 REPLACE     314.70k     0.7/s            2.85          3.23
Com_          959.74k     2.2/s            8.70
 commit      397.14k     0.9/s            3.60
 begin       397.10k     0.9/s            3.60
 set_option   78.02k     0.2/s            0.71

__ SELECT and Sort _____________________________________________________
Scan          102.85k     0.2/s %SELECT:   7.32
Range          19.46k     0.0/s            1.39
Full join          20     0.0/s            0.00
Range check         1     0.0/s            0.00
Full rng join       0       0/s            0.00
Sort scan       4.54k     0.0/s
Sort range     97.48k     0.2/s
Sort mrg pass       0       0/s

__ Query Cache _________________________________________________________
Memory usage  201.12M of 512.00M  %Used:  39.28
Block Fragmnt  15.31%
Hits          312.03k     0.7/s
Inserts         1.33M     3.1/s
Insrt:Prune   1.33M:1     3.1/s
Hit:Insert     0.23:1

__ Table Locks _________________________________________________________
Waited             95     0.0/s  %Total:   0.00
Immediate      10.24M    23.8/s

__ Tables ______________________________________________________________
Open               64 of   64    %Cache: 100.00
Opened        106.67k     0.2/s

__ Connections _________________________________________________________
Max used           25 of  151      %Max:  16.56
Total          32.61k     0.1/s

__ Created Temp ________________________________________________________
Disk table     14.47k     0.0/s
Table          40.10k     0.1/s    Size:  16.0M
File                5     0.0/s

__ Threads _____________________________________________________________
Running             1 of   10
Cached              4 of    8      %Hit:  99.84
Created            51     0.0/s
Slow                0       0/s

__ Aborted _____________________________________________________________
Clients           126     0.0/s
Connects           50     0.0/s

__ Bytes _______________________________________________________________
Sent            3.59G    8.3k/s
Received        1.83G    4.3k/s

__ InnoDB Buffer Pool __________________________________________________
Usage         127.98M of 127.98M  %Used: 100.00
Read hit       99.38%
Pages
 Free              0            %Total:   0.00
 Data          8.00k                     97.66 %Drty:   0.00
 Misc            192                      2.34
 Latched           0                      0.00
Reads         173.72M   404.3/s
 From file     1.07M     2.5/s            0.62
 Ahead Rnd                 0/s
 Ahead Sql                 0/s
Writes         46.42M   108.0/s
Flushes         3.86M     9.0/s
Wait Free           0       0/s

__ InnoDB Lock _________________________________________________________
Waits               0       0/s
Current             0
Time acquiring
 Total             0 ms
 Average           0 ms
 Max               0 ms

__ InnoDB Data, Pages, Rows ____________________________________________
Data
 Reads         1.14M     2.6/s
 Writes        4.26M     9.9/s
 fsync       959.52k     2.2/s
 Pending
   Reads           0
   Writes          0
   fsync           0

Pages
 Created      26.89k     0.1/s
 Read          1.14M     2.7/s
 Written       3.86M     9.0/s

Rows
 Deleted       6.27M    14.6/s
 Inserted      7.49M    17.4/s
 Read        101.23M   235.6/s
 Updated       1.90M     4.4/s

THANKS!

[Endon]

@Diablox

Thanks for your help, I will try that the next thing in the morning and keep you updated

[Diablox]

@Diablox

Thanks for your help, I will try that the next thing in the morning and keep you updated

Just do not forget to do a backup

[Endon]

I've reïnstalled mysql and optimized the settings, but mysql is still using much cpu-power. We think that the hardware is the bottleneck, our raidcard is only capable till 5 mb/s.

[rilex]

Are you running SCSI-1 on that box?

[Endon]

Nope

We just got ourselves some new hardware and the lagg is gone...

OS: Debian 5.0 (stable) (64bit)

Processor: Core i7 920

Memory: 8GB DDR3

[ilija]

Guys , how can i compile 64 bit on linux? need something to add in ./configure line?

[Wowka321]

Something, like this

../configure  CFLAGS="-O3 -pipe -msse4.2 -DFD_SETSIZE=4096 -march=core2 -fomit-frame-pointer -ffast-math -fno-strength-reduce -fno-strict-aliasing -frename-registers" CHOST="x86_64-pc-linux-gnu" CXXFLAGS="-O3 -pipe -msse4.2 -DFD_SETSIZE=4096 -march=core2 -fomit-frame-pointer -ffast-math -fno-strength-reduce -fno-strict-aliasing -frename-registers" --prefix=/home/serv --sysconfdir=/home/serv/etc --datadir=/home/serv --with-debug-info --disable-builtin-ace

[ilija]

Could be a little bit exactlier? like -march=core2 , if i have 3 cores i should use core3?

EDIT: Still getting ingame : Linux_x32 , any ideas?