[Mangos Zero] Some Develop21 Install Issues and a Bash Install Script for Fun

[mjrpes]

Hello, I built server and database today off of develop21 branch onto a Ubuntu VirtualBox. MaNGOS is AMAZING!!

I ran into database sql issue. Git command:

git clone https://github.com/mangoszero/server -b develop21 --recursive
git clone https://github.com/mangoszero/database -b develop21 --recursive

1. World database insert fails because it complains that 'db_scripts' and 'warden' tables do not exist. I couldn't find in source anywhere where these two tables attempt to get created. I was able to work around this by generating the tables myself, although I am unsure if my datatypes match yours:

DROP TABLE IF EXISTS db_scripts;
CREATE TABLE db_scripts (
 script_guid int(11) NOT NULL,
 script_type int(11) DEFAULT NULL,
 id int(11) DEFAULT NULL,
 delay int(11) DEFAULT NULL,
 command int(11) DEFAULT NULL,
 datalong int(11) DEFAULT NULL,
 datalong2 int(11) DEFAULT NULL,
 buddy_entry int(11) DEFAULT NULL,
 search_radius int(11) DEFAULT NULL,
 data_flags int(11) DEFAULT NULL,
 dataint int(11) DEFAULT NULL,
 dataint2 int(11) DEFAULT NULL,
 dataint3 int(11) DEFAULT NULL,
 dataint4 int(11) DEFAULT NULL,
 x double DEFAULT NULL,
 y double DEFAULT NULL,
 z double DEFAULT NULL,
 o double DEFAULT NULL,
 comments varchar(255) DEFAULT NULL,
 PRIMARY KEY (script_guid)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

DROP TABLE IF EXISTS warden;
CREATE TABLE warden (
 id int(11) NOT NULL,
 groupid int(11) DEFAULT NULL,
 build int(11) DEFAULT NULL,
 type int(11) DEFAULT NULL,
 data varchar(255) DEFAULT NULL,
 str varchar(255) DEFAULT NULL,
 address int(11) DEFAULT NULL,
 length int(11) DEFAULT NULL,
 result varchar(255) DEFAULT NULL,
 comment varchar(255) DEFAULT NULL,
 PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

[Olion]

Hello!

The present world DB dump is of 21.10.1 version (due to the db_version.sql content found in World/Setup/FullDB). It means that after the full world DB installation, you need to apply only later (newer than 21.10.1) updates from World/Updates/Rel21/ folder. You might as well ignore them for now, since they have structure=10 and the core will start without them.

On the changes. The world DB structure was changed recently via merging/modifying tables:

older: `dbscripts_on_*` => current `db_scripts`
older: `warden_checks`, `warden_build_specific` => current `warden`

If you want to find the table formats, look in the DB repo at the specific point of history, like this one for the older dbscripts*.

If you ask my opinion, the dbscript change is good regardless of that it introduces a misguidedly named table field `script_guid` and breaks the index system with it; the warden change is of a wrong nature.

[antz]

[MENTION=8245]mjrpes[/MENTION] - i'd like to identify the cause of your install issues, rather than skip around the issue as has been mentioned.

The Install should install as it is since the last rollback commit, if I doesn't - we need to get it fixed.

[MENTION=5581]Olion[/MENTION] - Rather than griping about something you don't like, fix and commit it. The warden changes HAD to be made to fix issues with 6141. They were subsequently changed to fix your criticism of it.

[mjrpes]

[MENTION=8245]mjrpes[/MENTION] - i'd like to identify the cause of your install issues, rather than skip around the issue as has been mentioned.

The Install should install as it is since the last rollback commit, if I doesn't - we need to get it fixed.

[MENTION=5581]Olion[/MENTION] - Rather than griping about something you don't like, fix and commit it. The warden changes HAD to be made to fix issues with 6141. They were subsequently changed to fix your criticism of it.

Hello Olion and Antz. Below is a bash script that follows my installation. It wasn't until after I installed everything that I discovered there were already some install.sh scripts in the linux directory.... I had plans to clean it up but you guys probably have a more robust script. I installed things yesterday.

When I attempt to run the full_db.sql on World/Setup/FullDB data (running the concat feature), is when it complains about not finding db_scripts and warden tables. It seems like World/Setup/mangosdLoadDB.sql is the proper file to have these tables created, but they are not present there.

Also, World/Setup/mangosdCreateDB.sql I think has typo. It attempts to create 'character0' database, but the Character/Setup/characterCreateDB.sql uses 'characters0'.

Maybe one more think you could help me on. I compiled with playerbot support, and got bots spawning in game and I could interact using ~follow etc. Documentation elsewhere says to use '.bot add botname' to spawn a new bot, but this does not seem to do anything. I tried variations like 'bot add' and '~.bot add' '~bot add' etc. Maybe this is not supported on mangoszero. Thanks.

#!/bin/sh

####################################################################################
## notes
####################################################################################

# THIS GUIDE IS FOR MANGOSZERO!!

# install guide:
# http://together.chrisfaulkner.org/index.php/MangosZero_Install_-_Debian

###############################
## assumptions
###############################

# this script now is making assumptions with environment need to be replaced as needed:
# 1. Ubuntu 14.04 virtual box with bridged ethernet, static ip = 192.168.3.25
# 2. Ubuntu user mangos, password mangos
# 3. Mysql root password mangos
# 4. Mysql user mangos, password mangos
# 5. Using develop21 branch for database/server (db_script and warden table fixes)
# 6. mysql world, character, realm database => mangos0, characters0, realmd
# 7. will extract map table on virtualbox (it is 5x slower than native)
# 8. run server from /opt/mangos
# 9. wow client folder is a virtualbox share folder called 'wow' (shows up in /media/sf_wow)

###############################
## how to connect after running this script nad server is running!
###############################

# 1. go to WoW client directory and open up realmlist.wtf
# 2. change to 'set realmlist 192.168.3.25'
# 3. create user. in the mangosd running process, 
#    type: account create "playername" "mangos" (replace playername with name you want)
# 4. you should be able to log in

####################################################################################
## script prep; run this before running script
####################################################################################

###############################
## prep ubuntu
###############################

# create a virtualbox with ubuntu AMD64 and login user 'mangos', password 'mangos'

# when you first start ubuntu in virtualbox, it will have 640x480 screen. install
# apt-get guest additions to allow you to resize ubuntu desktop size and enable shared folder
# access that you will need to copy over WoW client data below:
# 1. click on first icon in ubuntu desktop ("search your computer...")
# 2. search for 'terminal'
# 3. drag terminal icon to icon bar
# 4. open terminal and run: sudo apt-get install virtualbox-guest-additions-iso

###############################
## prep mangos user with sudo privileges
###############################

# Open terminal.
# `su -` (this logs you in as root, the hyphen starts a fresh environment)
# `nano /etc/sudoers`
# Add this line in 'User Privilege Section' underneath root: 'mangos    ALL=(ALL:ALL) ALL'
# Logout as mangos and re-login

# NOT NEEDED: `adduser mangos`. Next time we set up VirtualBox we use mangos as default user.
#   This is better because we will use screen to run mangosd in background, which requires
#   we be logged in directly as mangos anyway (not logged in via `su`).

###############################
## prep CPUs
###############################

# Mangos requires that maps extraction happens on the same release as the server/database.
# However, map extraction takes a long time. By enabling multiple CPUs in VirtualBox,
# extracting can be 4x faster on a 4 core CPU. It is possible to extract on native
# Windows box, and this will be another 5x times faster, but this requires compiling 
# the develop release using cmake on Windows, so more software and steps to get it working.
# I had tried running server/database on 'release' branch, not 'develop21', but I had server/database
# mismatch here, and develop21 is supposedly better anyway.
#   1. Do this late at night because your computer will be hard to use with all CPUs running.
#   2. Set power settings on host/native to not go to sleep
#   3. VirtualBox -> Machine -> Settings -> System -> Processor 
#   4. Set CPU to 4
#   5. Set execution cap (optional). Extracting will use all CPUs and your
#      computer will not be responsive. Set to lower cap to avoid this. I tried 95% and
#      computer was still pretty unresponsive.

###############################
## prep static ip
###############################

# FYI, following ports are used by mangos, if you go NAT route: mangosd (tcp 8085), realmd (tcp 3724)
# To set up static ip, I did not edit interfaces file directly because it doesn't seem the place 
# where Ubuntu stores interface settings.
# STEP 1: Enable Bridge Adapter:
#    VirtualBox -> Machine -> Settings -> Network -> Set Attached to Bridged Adapter
# STEP 2: In Ubuntu UI, go to Settings -> Network -> Options for Wired Connection -> Add Static IP Info 
# like below, depending on your network settings:
#   address: 192.168.3.25
#   netmake: 255.255.255.0
#   gateway: 192.168.3.20
#   dns    : 8.8.8.8

###############################
## prep WoW client data as share
###############################

# getting wow client data accessible to virtualbox was a pain. I tried a couple of methods.
# SMB was SLOW and unusable. Could not browse to host computer in SMB anyway. Tried
# drag and drop, but that messed up on paste. Tried copy and paste, but files were
# not transferable (grayed out "paste" option in ubuntu). The solution was to set up
# folder share in virtualbox:
#   1. VirtualBox -> Devices -> Shared Folders -> Shared Folder Settings...
#   2. Add new machine folder
#   3. Source Directory => The main WoW folder (folder that contains WoW.exe)
#   4. Target => wow
#   5. In Ubuntu, need to give permission to access this folder. Run the following:
#   6. `sudo adduser mangos vboxsf`
#   7. Reboot Ubuntu server to get shared folder to show up
#   8. Check that folder /media/sf_wow is present and you have permission to open it
#   9. This script assumes this folder is where client WoW data is

####################################################################################
## sanity check
####################################################################################

# test mangos user is running script

if [ "$USER" != "mangos" ]; then
 echo "You need to run this script as mangos user. See instructions in this script"
 exit 1
fi

# test mango user has sudo access

CAN_I_RUN_SUDO=$(sudo -n uptime 2>&1|grep "load"|wc -l)
if [ ${CAN_I_RUN_SUDO} -eq 0 ]; then
 echo "Mangos user needs sudo permissions to run this script. See instructions in script."
 exit 1
fi

# test wow client share files are present

if ! [ test -e "/media/sf_wow" ]; then
 echo "Wow share not present. Create share of WoW client folder and give is 'wow' for name."
 exit 1
fi

if ! [test -e "/media/sf_wow/WoW.exe" ]; then
 echo "Wow data not present. Expecting Wow.exe. Check that you shared the right folder."
 exit 1
fi

# only run if database does not exist

if mysql -umangos -pmangos -e 'use mangos0'; then
 echo "Databases already exists. Aborting to avoid overwriting data! Delete databases manually first."
 exit 1
fi

####################################################################################
## prompt to begin
####################################################################################

echo -n "Press any key to begin mangos WoW server install! Or Ctrl-c to exit."
read ANSWER

####################################################################################
## setup
####################################################################################

# setup build environment
sudo apt-get install -y autoconf automake clang cmake gcc g++ libtool make patch
sudo apt-get install -y cmake-curses-gui git-core

# install development headers
sudo apt-get install apache2 cmake cmake-qt-gui git g++ libace-ssl-dev libace-dev \
libapache2-mod-php5 libbz2-dev libmysql++-dev libmysqlclient-dev libssl-dev \
libtbb-dev make php5-mysql libtbb-dev libtbb2 zlib1g-dev vim libtbb-dev

# setup mysql; prevent prompt for password when install mysql; we are using mangos for password
sudo debconf-set-selections <<< 'mysql-server mysql-server/root_password password mangos'
sudo debconf-set-selections <<< 'mysql-server mysql-server/root_password_again password mangos'
sudo apt-get install -y mysql-server mysql-common mysql-client

####################################################################################
## set up runtime directories and permissions
####################################################################################

# TODO: edit config file after compile to point to proper log directories

# WORKING DIRECTORY
cd /opt

# make run directories
sudo mkdir -p mangos
sudo chown -R mangos:mangos mangos

####################################################################################
## get, compile, and install source
####################################################################################

# WORKING DIRECTORY
cd ~

# get source
mkdir mangos-source
cd mangos-source
git clone https://github.com/mangoszero/server -b develop21 --recursive

# compile
cd server
mkdir build
cd build

# use this command instead to enable bots
#cmake -DCMAKE_INSTALL_PREFIX=/opt/mangos -DDEBUG=1 -DBUILD_TOOLS=1 -DACE_USE_EXTERNAL=1 -DPLAYERBOTS=1 ..

cmake ../ -DCMAKE_INSTALL_PREFIX=/opt/mangos -DDEBUG=1 -DBUILD_TOOLS=1 -DACE_USE_EXTERNAL=1

make && make install

####################################################################################
## set up config files
####################################################################################

# WORKING DIRECTORY
cd /opt/mangos/etc

cp ahbot.conf.dist ahbot.conf
cp mangosd.conf.dist mangosd.conf
cp realmd.conf.dist realmd.conf

# update realmd.conf
sed -rin 's/^(LoginDatabaseInfo[ \t]*=[ \t]*)"(.*)"$/\1"127.0.0.1;3306;mangos;mangos;realmd"/' realmd.conf

# update mangosd.conf
sed -rin 's/^(DataDir[ \t]*=[ \t]*)"(.*)"$/\1"\/opt\/mangos\/data"/' mangosd.conf
sed -rin 's/^(LoginDatabaseInfo[ \t]*=[ \t]*)"(.*)"$/\1"127.0.0.1;3306;mangos;mangos;realmd"/' mangosd.conf
sed -rin 's/^(WorldDatabaseInfo[ \t]*=[ \t]*)"(.*)"$/\1"127.0.0.1;3306;mangos;mangos;mangos0"/' mangosd.conf
sed -rin 's/^(CharacterDatabaseInfo[ \t]*=[ \t]*)"(.*)"$/\1"127.0.0.1;3306;mangos;mangos;characters0"/' mangosd.conf

####################################################################################
## extract game assets
####################################################################################

# NOTE: use this if you extracted assets on Windows and are copying over 
#mkdir /opt/mangos/data
#cd /media/sf_wow
#cp -r maps mmaps vmaps dbc /opt/mangos/data/

# -- WORKING DIRECTORY
mkdir ~/wow-client
cd ~/wow-client

# -- copy over wow client files from shared folder
cp -R /media/sf_wow/Data Data

# -- copy over utilities
cp /opt/mangos/bin/tools/* .

echo "==========================================================================="
echo "now extracting map data. this is a LONG process. be sure to read"
echo "instructions at top of script to enable all cores in VirtualBox. overall,"
echo "process is about 5x times slower than if compiling natively."
echo "on a i5-2500 system: 6 hours virtual, 1.2 hours native."
echo "==========================================================================="
echo ""

chmod +x ExtractResources.sh
./ExtractResources.sh

# -- move extracted data over to running directory
cp -R maps mmaps vmaps dbc /opt/mangos/data/

####################################################################################
## set up database
####################################################################################

# NOTE #1: not using make_full_WorldDB.sh because excludes the beginning of each sql file, which in 
#   some cases deletes the truncate statements. we want all truncate statements to run because
#   we may be re-running script. Removed Ref: `chmod +x make_full_WorldDB.sh && ./make_full_WorldDB.sh`
# NOTE #2: took out "CREATE USER 'mangos'@'localhost' IDENTIFIED BY 'mangos';" because it causes
#   error when re-running script. the grant command will create user if they don't already exist.

######################################
## get data
######################################

# get database info
cd ~/mangos-source
git clone https://github.com/mangoszero/database -b develop21 --recursive

# WORKING DIRECTORY
cd ~/mangos-source/database

######################################
## create/fix missing/broken sql to get dev20 to work
######################################

## these are db sql files that are supposed to be run but have typos or will cause error with 
## add user command if user already exists. also, one file users 'character0' db, another 'characters0'.
#mysql -uroot -pmangos realmd < Realm/Setup/realmdCreateDB.sql
#mysql -uroot -pmangos characters < Character/Setup/characterCreateDB.sql
#mysql -uroot -pmangos mangos < World/Setup/mangosdCreateDB.sql

# create sql statement to create database/users; note that grant will create mangos user if not exists
cat << EOF > install_create_databases.sql
CREATE DATABASE characters0 DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
GRANT ALL PRIVILEGES ON characters0.* TO 'mangos'@'localhost';

CREATE DATABASE realmd DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
GRANT ALL PRIVILEGES ON realmd.* TO 'mangos'@'localhost';

CREATE DATABASE mangos0 DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
GRANT ALL PRIVILEGES ON mangos0.* TO 'mangos'@'localhost';

GRANT USAGE ON *.* TO 'mangos'@'localhost' IDENTIFIED BY 'mangos' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0;
FLUSH PRIVILEGES;
EXIT
EOF

# creating missing tables (develop21)

cat << EOF > install_create_missing_tables_dev21.sql
DROP TABLE IF EXISTS db_scripts;
CREATE TABLE db_scripts (
 script_guid int(11) NOT NULL,
 script_type int(11) DEFAULT NULL,
 id int(11) DEFAULT NULL,
 delay int(11) DEFAULT NULL,
 command int(11) DEFAULT NULL,
 datalong int(11) DEFAULT NULL,
 datalong2 int(11) DEFAULT NULL,
 buddy_entry int(11) DEFAULT NULL,
 search_radius int(11) DEFAULT NULL,
 data_flags int(11) DEFAULT NULL,
 dataint int(11) DEFAULT NULL,
 dataint2 int(11) DEFAULT NULL,
 dataint3 int(11) DEFAULT NULL,
 dataint4 int(11) DEFAULT NULL,
 x double DEFAULT NULL,
 y double DEFAULT NULL,
 z double DEFAULT NULL,
 o double DEFAULT NULL,
 comments varchar(255) DEFAULT NULL,
 PRIMARY KEY (script_guid)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

DROP TABLE IF EXISTS warden;
CREATE TABLE warden (
 id int(11) NOT NULL,
 groupid int(11) DEFAULT NULL,
 build int(11) DEFAULT NULL,
 type int(11) DEFAULT NULL,
 data varchar(255) DEFAULT NULL,
 str varchar(255) DEFAULT NULL,
 address int(11) DEFAULT NULL,
 length int(11) DEFAULT NULL,
 result varchar(255) DEFAULT NULL,
 comment varchar(255) DEFAULT NULL,
 PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
EXIT
EOF

######################################
## create databases
######################################

mysql -uroot -pmangos < install_create_databases.sql

######################################
## set up realm database
######################################

mysql -uroot -pmangos realmd < Realm/Setup/realmdLoadDB.sql
mysql -uroot -pmangos realmd < Tools/updateRealm.sql

# default realm address is set to localhost; but we are running server to be accessed from LAN,
# so we need to tell it our static ip. if we don't do this, when WoW client logs in you will get
# a realm list loop
mysql -uroot -pmangos -e "UPDATE realmlist SET address = '192.168.3.25' WHERE id = 1;" realmd

######################################
## set up character and world databases
######################################

mysql -uroot -pmangos characters0 < Character/Setup/characterLoadDB.sql
mysql -uroot -pmangos mangos0 < World/Setup/mangosdLoadDB.sql

# run missing tables (develop21 fix)
mysql -uroot -pmangos mangos0 < install_create_missing_tables_dev21.sql

# add full db
cat World/Setup/FullDB/*.sql >> install_add_world_data.sql
mysql -uroot -pmangos mangos0 < install_add_world_data.sql

####################################################################################
## we are done
####################################################################################

echo "Done!"
echo "Go to 'cd /opt/mangos', then run './mangosd' and './realmd' in separate consoles "
echo "to test that things are working properly."

exit 0

[antz]

Hi.... I have just committed a fix to the table creation script:

[Fix] Base DB load fixed for both Windows/Linux. Thanks to mrrpes for… * mangoszero/database@6fcb4c4 * GitHub

Also looking at your script you are missing this file to create the actual tables:

mangosdLoadDB.sql

This should be imported before each of the separate .SQL files are loaded

[mjrpes]

Thanks antz. the mangod sql is in my script, but it's a long script and tucked in.

Do you have any ideas about the bot issue I mentioned?

[antz]

Assuming the following:

* you have built the core with playerbots enabled

* you have enabled playerbots in aiplayerbot.conf

the command is normally.....

.bot add

etc.

[mjrpes]

That's what doesn't seem to work. Everything else works. 200 bots are spawning in the world and making the console speed by with a million messages per second. I can see the bots and interact with them and add them to my party. They respond to commands like ~follow and ~grind. However when I type ".bot add name" it treats my text like chat text, suggesting the command is not being caught by the bot program. Tried

.bot add name

/.bot add name

~.bot add name

~bot add name

Maybe I will try diving into the source code.

[mjrpes]

OK got it to work. Didn't realize I needed to be an admin