Semaphore created by ACE Posix/Mutex are not disappearing

[luciolis]

Hello,

In rev 10456, I had a new error in the console :

ACE_SV_Semaphore_Complex : No space left on Device. In fact, when I put ipcs on linux, I see loads of semaphores. With a earlier version in another machine, there is no lasting semaphore at all.

I try to delete them, when the server starts, it comes again.

Any idea ?

[fdb_]

Try this:

echo 134217728 >/proc/sys/kernel/shmall
echo 134217728 >/proc/sys/kernel/shmmax

Or put in your sysctl config file (to have the settings kept after a reboot):

kernel.shmall = 134217728
kernel.shmmax = 134217728

There are the descriptions of the semaphore kernel params:

SEMMSL: maximum number of semaphores per id

SEMMNS: maximum number of semaphores, system wide.

SEMOPM: maximum number of operations in one semop call.

SEMMNI: number of semaphore identifiers (or arrays), system wide.

You can also tune the semaphore max numbers to avoid these messages. For example Oracle needs value larger than 100 in semaphore kernel params.

To list all the system params:

sysctl -a

(use grep to find the one you want)

Also try to use google to find system kernel config for Oracle, perhaps it will help.

Good luck and keep us updated.

[luciolis]

Thank you, but the thing is that with another mangos rev in another server, there is no semaphore att all ! (and I use MySQL, that's weird)

EDIT : done this, but still the same error. I'll kill myself

[fdb_]

Thank you, but the thing is that with another mangos rev in another server, there is no semaphore att all ! (and I use MySQL, that's weird)

EDIT : done this, but still the same error. I'll kill myself

I also have those error on a test server but I did not think it was a problem.

I will try to find a workaround but I cannot promise to do it quick.

Meanwhile you can try to put these lines in your sysctl config file (in fact, that are the tests I will do on my test server):

kernel.semmsl = original value * 4 (or * 8)
kernel.semmns = original value * 4 (or * 8)
kernel.semopm = original value * 4 (or * 8)
kernel.semmni = original value * 4 (or * 8)

Of course, still use

sysctl -a

to find the current system values.

NB: The sysctl config file need to be reloaded to apply the changes (or reboot your server).

Also, I have heard from a Veritas engineer that some kind of semaphores can only be cleaned with a system reboot (I doubt that this is the only solution but it worth a try to have a clean system before reproducing the error).

You can try to list the semaphores with

ipcs -a

And to delete semaphore

ipcrm -s [i]id[/i]

Be gentle because some semaphores does not belong to mangos but to other system processes.

Good luck.

[luciolis]

Mh. The semaphores are created when somebody connects to the world server. It's not related with MySQL at all.

Those errors seem to cause various lags in game, and I've the error almost 7 times for 1 sec.

And even with 130+ mio (= infinity ) semaphores allowed in sysctl.conf, this error is still occuring.

Do you think it's mangos or system related ?

[fdb_]

I think this is related to mangos and not to system.

When I spoke about Oracle I did not mean that it came from the SQL server. But I was refering to some Linux system requirements for a medium loaded Oracle on Linux (some system tuning - including semaphores - have to be set).

In some case mangos can be very greedy. So some system tuning can be mandatory.

BTW, I think you set the number of allowable semaphores far too high, something like 1K seems already very high to me and should be fine for most of the mangos servers.

[luciolis]

BTW, I think you set the number of allowable semaphores far too high, something like 1K seems already very high to me and should be fine for most of the mangos servers.

That's what you advised

In some case mangos can be very greedy. So some system tuning can be mandatory.

I know this, but why there is no permanent semaphore at all in my other system ?

In sysctl -p

error: "kernel.semmsl" is an unknown key

error: "kernel.semmns" is an unknown key

error: "kernel.semopm" is an unknown key

error: "kernel.semni" is an unknown key

[fdb_]

That's what you advised

I know this, but why there is no permanent semaphore at all in my other system ?

Seems that some semaphore can become residual almost like zombie processes (I cannot confirm btw).

In sysctl -p

error: "kernel.semmsl" is an unknown key

error: "kernel.semmns" is an unknown key

error: "kernel.semopm" is an unknown key

error: "kernel.semni" is an unknown key

Yeah you are right I write this and it is totally wrong

In fact all these params are in one kernel config named "kernel.sem".

List current semaphore values:

# ipcs -ls

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

You can also use the following command:

# cat /proc/sys/kernel/sem
250     32000   32      128

Here are the information I found from the Oracle recommendations (should work for any other hungry systems):

SEMMSL

The SEMMSL kernel parameter is used to control the maximum number of semaphores per semaphore set.

Oracle recommends setting SEMMSL to the largest PROCESS instance parameter setting in the init.ora file for all databases hosted on the Linux system plus 10. Also, Oracle recommends setting the SEMMSL to a value of no less than 100.

SEMMNI

The SEMMNI kernel parameter is used to control the maximum number of semaphore sets on the entire Linux system.

Oracle recommends setting the SEMMNI to a value of no less than 100.

SEMMNS

The SEMMNS kernel parameter is used to control the maximum number of semaphores (not semaphore sets) on the entire Linux system.

Oracle recommends setting the SEMMNS to the sum of the PROCESSES instance parameter setting for each database on the system, adding the largest PROCESSES twice, and then finally adding 10 for each Oracle database on the system. To summarize:

SEMMNS = sum of PROCESSES setting for each database on the system

+ ( 2 * [largest PROCESSES setting])

+ (10 * [number of databases on system]

To determine the maximum number of semaphores that can be allocated on a Linux system, use the following calculation. It will be the lesser of:

SEMMNS -or- (SEMMSL * SEMMNI)

SEMOPM

The SEMOPM kernel parameter is used to control the number of semaphore operations that can be performed per semop system call.

The semop system call (function) provides the ability to do operations for multiple semaphores with one semop system call. A semaphore set can have the maximum number of SEMMSL semaphores per semaphore set and is therefore recommended to set SEMOPM equal to SEMMSL.

Oracle recommends setting the SEMOPM to a value of no less than 100.

Dynamic setting:

# echo "250 32000 100 128" > /proc/sys/kernel/sem

Persistant settings:

# echo "kernel.sem=250 32000 100 128" >> /etc/sysctl.conf

I hope this one will work