[HOWTO] GDB Debugging

[Auntie Mangos]

I tryed to make it as simple as possible, because there are a lot of noob admins that dont know what is /etc/security/limits.conf or how to change ulimit -c ( or any other ulimit ). Also a lot of ppl compile mangos without knowing what is CXXFLAGS .

Yes, there are a lot of ways to get crash information, here are some.

1) Start mangos from gdb.

2) Use ulimit -c ... I think I made some script that can process all cores in given directory, infy uses it to give crash reports, for example.

3) Install SIGSEGV signal handler in mangos and make it call gdb when it crashes. This requires modification in mangos code, common example of apps that do this are gnome and KDE desktop apps.

I put here the easyest way I see.

The most important for crash report is not how you generated it, but what information it contains. A lot of reports for example dont contain the signal that caused the abnormal termination or dont provide backtrace of all threads, or dont provide other important information.

[Auntie Mangos]

Maybe these gdb commands may be usefull:

set logging file /some/file
set logging on

Together with gdb arg '-ex'

I tried the "-ex <filename>" setting and it slowdown the speed a lot, what I did wrong?

For now I tried to take at least SS ( http://getmangos.eu/community/showthread.php?p=50278#post50278 ) but a log file would be better

[Derex]

GDB: The GNU Project Debugger

Hi, this is a simple/dirty tutorial of what is GDB and how to use it.

This tutorial asumes that you can start your mangos-worldd by typing:

mangos-worldd

in the console.

1. Before you start debugging mangos you need to have ensured that it is compiled with debug information. To build mangos with debug info just add --with-debug-info to your options when running ../configure. Then compile and install mangos as usual.

2. Now you can start mangos with gdb by typing this.

gdb mangos-worldd

Then you will most likely see something like this:

derex@*:~/workspace/mangos/build$ gdb mangos-worldd
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
(gdb)

(gdb) - is the GDB command prompt , here you can type some commands, almost like in normal shell.

Now after you have gdb running, you may instruct it to start mangos ( by having gdb start mangos you can debug it ), here is the command:

 run

Just type it and if you are lucky you will have mangos loading.

Ok ... you made it, now mangos runs. Take a break until it crash ,then you can come back

3. When mangos crashes you will most likely see this message, or any similar.

Program received signal SIGSEGV, Segmentation fault.
[switching to Thread 0x42c5c950 (LWP 9283)]
Player (this=0x42c598e0, session=0x0) at ../../../src/game/Player.cpp:265
265    ../../../src/game/Player.cpp: No such file or directory.
   in ../../../src/game/Player.cpp
(gdb)

Now you can type some comands to get information about the crash, and possibly give it to some dev to fix the problem.

Here are the commands that are best to be typed ( or at least I find the most usefull for crash report )

shell echo -e "\\nCRASH ON" `date`
info program
shell echo -e "\\nBACKTRACE\\n"
bt
shell echo -e "\\nBACKTRACE FULL\\n"
bt full
shell echo -e "\\nTHREADS\\n"
info threads
shell echo -e "\\nTHREADS BACKTRACE\\n"
thread apply all bt full

Just type them one after another and give the output in your bug report ...

OK. Thats it, now we can think of some way to automate all this process. GDB has 2 very good switches:

--batch            Exit after processing options.
--command=FILE, -x Execute GDB commands from FILE.

4. So you can put all the commands in one file and have GDB execute it, and when it finishes to exit. Lets say we put this in one file called gdb-commands.

run
shell echo -e "\\nCRASH ON" `date`
info program
shell echo -e "\\nBACKTRACE\\n"
bt
shell echo -e "\\nBACKTRACE FULL\\n"
bt full
shell echo -e "\\nTHREADS\\n"
info threads
thread apply all bt full

Now you can start the whole monster with:

gdb mangos-worldd --batch -x /path/to/gdb-commands

TIPS

1.You can also redirect stdout to some log file. I mean this:

gdb mangos-worldd --batch -x /path/to/gdb-commands > /some/log/file

2.You can add a tail command to gdb-commands file to get the latest lines from your server.log file:

shell echo -e "\\nSERVER.LOG\\n"
shell tail -n 50 /path/to/your/server.log

50 means how much lanes to take from server.log

3. You can add -ggdb3 -g3 flags to your CXXFLAGS in order to get more debug output. If you add them there is no need to add --with-debug-info switch in order to get meaningfull backtrace.

4. You can pass arguments to mangos-worldd by passing them to the run gdb command ( run -c /path/to/mangosd.conf ).

5. With recent mangos versions you can also add this line to your gdb-commands

shell echo -e "\\nMANGOS VERSION"
shell mangos-worldd --version

To get correct version information about mangos ( so no need to specify it in the bug report ... its already there and its correct, and you can't forget it ).

[leak]

Thanks Derex for bringing this guide back to the forums.

6 Before you start debugging mangos you need to have ensured that it is compiled with debug information. To build mangos with debug info just add --with-debug-info to your options when running ../configure. Then compile and install mangos as usual.

--with-debug-info enables -g and sets MANGOS_DEBUG which ain't helping you that much unless your loglevel is "debug".

I'd rather go with "-g -ggdb", if you wanna go if you want no unused code to be compiled in (it's mostly a few sLog.outDebug but hey, aren't we all hunting for performance?)

Speaking of performance:

I switched to a slightly different debugging method.

ulimit -c unlimited

Enables a coredump to be created on signals like SEGVAULT

You'll find a "core.PID" in your mangos bin dir,

gdb mangos-worldd --core core.`cat worldd.pid` --batch --quiet -ex "info threads" -ex "thread apply all bt full" -ex "quit"

Creates the debug output from that coredump file.

This method should be less performance draining than running mangos with an attached debugger and according to gentoo guys (http://www.gentoo.org/proj/en/qa/backtraces.xml) a little less error-prone.

I couldn't find any disadvantages of that method so far, maybe the debug output quality is less detailed, i don't know. If someone has more information about that - feel free to share it.

Ah yeah, before i forget: Depending on the size of your server (playercount-> ram usage) you might wanna remove that coredump after you got the debug log out of it. Those files can get pretty big and will rapidly fill your harddrive.

[crashuncle]

The tutorial helped me a lot and works perfectly on Ubuntu, thanks

I have one problem left, I want to log the debug information's to a file to be able to read it easier and post it here in the debug section. I use a console only Linux, so I tried to use this line...

gdb mangos-worldd --batch -x /path/to/gdb-commands > /some/log/file

The problem is that it cant create the file due to permission reasons so I logged it to /home/<username/gdb.log and that worked, but the console dont display all informations anymore?

Is it possible to log ONLY the debug part when the Server crash and let it show still all current informations in the server console before it crash?

Thanks in advance,

Crashuncle

[leak]

Well if mangos differs between stdout and stderr you might try

2> instead of >

[Derex]

Maybe these gdb commands may be usefull:

set logging file /some/file
set logging on

Together with gdb arg '-ex'

[Derex]

man, the -ex is not for filename

it specifies command to execute

--eval-command=COMMAND, -ex

Execute a single GDB command.

May be used multiple times and in conjunction

I will try to make a script that can be used to directly debug mangos, if I have free time this weekend.

[Endlesskiss]

I already started mangos-worldd, and I want to attach GDB to the current running process.

I tried attach 25620 [which is the PID of mangos-worldd], and I received an error:

ptrace: Operation not permitted.

Does anybody know why it is happen?

Thanks!

[LickedLurk]

Is it possible to bind an start.sh restarterscript to the gdb process that restarts mangos-worldd aswell when the server crashes ?

Cause the gdb process i not really ended when the server crashes, are there some tutorials or something or you derex can explain ?

Or i have to put the gdb code into the restarter-start.sh file with iam start the mangos-worldd ?

Regards

LickedLurk

[Morphau]

Is it possible to bind an start.sh restarterscript to the gdb process that restarts mangos-worldd aswell when the server crashes ?

Cause the gdb process i not really ended when the server crashes, are there some tutorials or something or you derex can explain ?

Or i have to put the gdb code into the restarter-start.sh file with iam start the mangos-worldd ?

Regards

LickedLurk

Up ! it's a good idea

[waza123]

.: Script ::

Auto restart mangos-worldd on crash & generate crash report into crash_log_(DATE_TIME).log

http://rapidshare.de/files/47316133/auto_restarter.sh.html

Enjoy !

[Morphau]

Thanks !

And, can you include the restart of realmd please in the script ?

[waza123]

Thanks !

And, can you include the restart of realmd please in the script ?

in my practice , there was no crashes in realmd

[Morphau]

Maybe ...

But can i have the solution for have the launch of realmd too please ...

Thanks and sorry for my english but i am french

[waza123]

Maybe ...

But can i have the solution for have the launch of realmd too please ...

Thanks and sorry for my english but i am french

look at file sources #config section, put there mangos-realmd

[Morphau]

Hello, thanks but i am a last question, how do you launch the script with screen mod ?

Beacause then i do this, if the server crash, the screen should be off and the server doesn't restart ...

Thnaks if you have a solution for me

Sorry for my english but i am french ...

[Darky88]

HI

:huh: How to?.... If i need disable gdb files core.*?

Thx Darky

[Derex]

Look at following manuals

man limits.conf

man ulimit

You would like to change the core limit to 0, so the kernel will not generate core dumps for crashed processes.

[digitalni]

.: Script ::

Auto restart mangos-worldd on crash & generate crash report into crash_log_(DATE_TIME).log

http://rapidshare.de/files/47316133/auto_restarter.sh.html

Enjoy !

Thanks a lot for this script!

I do have a little problem with it. My crash logs that get created are empty.

I am, however, running the script via screen, could this be the problem?

Also, server logging parameter in config is set to full.

Any idea what might be wrong?

[EMTEC_GM]

A small change in wazzy bash, have combined with my loop because it is not always wanted to act.

#!/bin/sh

###############
# About: Auto restart mangos-worldd on crash & generate crash report into crash_log_(DATE_TIME).log
###############
# 1. Compile MaNGOS with parameter: --with-debug-info
# 2. Put auto_restarter.sh into compiled mangos directory (where folders: bin/, lib/, etc/)
# 3. (Only once): chmod +x auto_restarter.sh
# 4. Usage: ./auto_restarter.sh -c etc/mangosd.conf
# p.s. Make sure you have "gdb" installed.
###############

# config:
# path to mangos-worldd binary
daemon=./bin/bin/mangos-wotlk
# system
export LD_LIBRARY_PATH=.:lib:$LD_LIBRARY_PATH

if [ "`ulimit -c`" -eq 0 ]; then
   ulimit -c unlimited
fi


while true
   do 
   MANGOS=`ps -el | grep mangos-wotlk`
   $daemon $*
   if [ -z "$MANGOS" ]; then
       dte=`date +%F_%H-%M-%S`
       gdb $daemon core.* --batch --eval-command="bt ful" > crash_log_$dte.log
       mv crash_log_$dte.log gdb/crash_log_$dte.log
       mv log/Server.log dc/Server-$dte.log
       gzip dc/*.log --best
       rm core.*
   fi
   sleep 5
done

[Mordaunt]

A small change in wazzy bash, have combined with my loop because it is not always wanted to act.

#!/bin/sh

###############
# About: Auto restart mangos-worldd on crash & generate crash report into crash_log_(DATE_TIME).log
###############
# 1. Compile MaNGOS with parameter: --with-debug-info
# 2. Put auto_restarter.sh into compiled mangos directory (where folders: bin/, lib/, etc/)
# 3. (Only once): chmod +x auto_restarter.sh
# 4. Usage: ./auto_restarter.sh -c etc/mangosd.conf
# p.s. Make sure you have "gdb" installed.
###############

I have no experience with GDB debugging and scripts, I am glad if someone can help me out.

The problem is that the crashlog is empty with every crash. I Use this script above and did all the steps which are commented in the script. I configured with the --with-debug-info flag and the loglevel in mangosd.conf is 3(full debug). I've installed gdb and placed the restarter in the directory where the bin/lib/etc directories are.

What am I missing? What am I doing wrong...

Thanx in advance for any help!

edit1: I am using Ubuntu server. No GUI.

edit2: Is it maybe I have to do some changes for ulimit?

[Micks]

any update for this script ?

[ilija]

On debian the script from emtec_gm works but with ubuntu not , it also dont delete core and it makes empty crashlogs, someone know how to fix this on ubuntu?

[ifilgud]

Hi! I've made a small modification of emtec_gm script to match my preferences

#!/bin/bash

###############
# About: Auto restart mangos-worldd on crash & generate crash log
###############
# 1. Compile MaNGOS with parameter: --with-debug-info
# 2. Put auto_restarter.sh into compiled mangos directory (where folders: bin/, lib/, etc/)
# 3. (Only once): chmod +x auto_restarter.sh
# 4. Usage: ./auto_restarter.sh -c etc/mangosd.conf
# p.s. Make sure you have "gdb" installed.
###############

# config:
# path to mangos-worldd binary
daemon=./bin/mangos-worldd
# system
export LD_LIBRARY_PATH=.:lib:$LD_LIBRARY_PATH

if [ "`ulimit -c`" -eq 0 ]; then
  ulimit -c unlimited
fi

while true
do
  MANGOS=`ps -el | grep mangos-worldd`
  $daemon $*
  if [ -z "$MANGOS" ]; then
     gdb $daemon core.* --batch --eval-command="bt ful" > crash.log
     dte=`date +%F_%H-%M-%S`
     mkdir crash_$dte
     mv crash.log crash_$dte/
     mv log/Server.log crash_$dte/
     mv core.* crash_$dte/
  fi
  sleep 2
done

You will find the logs and dumps inside crash_$date folder. Add the gzip line if you wish