[10167] vmap rewrite

[ilija]

Could someone tell me about the stability changes it makes?

[faramir118]

Ehm merging with mmaps will hardly be as easy as removing all "<<<<", "====" and ">>>>" lines...and please use paste2.org or similar next time...

The mmap generator uses the old vmap code, so that won't work. But if you extract the vmaps the old way and use them to generate mmaps, mangos should run just fine

@faramir118: Hm are you really sure you used a fresh compile of the extractor and had no old files in Building dir?

It was the latter. Which begs the question why the assembler tried to convert liquid data when there was none

[Lynx3d]

Well the original code already exported something...although that stuff didn't make any sense IMHO. The assembler just skipped that chunk, as it does with some other chunk that i have no idea what it was even intended to be used for.

Maybe i should implement a check if the dirs are clean before starting...

[yasuno]

So lynx3d how close do you think you are to completion with this rewrite? Just wondering...

Also at this point (stopped following topic a bit) is there a patch, or do i have to merge the branch?

[Unkle Nuke]

You can do a merge, but generating your own diff for later patching is ridiculously easy.

[Lynx3d]

So lynx3d how close do you think you are to completion with this rewrite? Just wondering...

Well, how shall i give an answer to that question when all i get back is questions like "are you done yet?", "why do i have to use git?" or "does it work with <insert random WIP branch>?"...

Yet i see people are happily merging it in their forks or even their "own" projects (== mangos forks with funky names and claiming all code to themselves...)

I don't know if no one is giving it a real try, or just no one cares to give any feedback (except 2 or 3 guys earlier in the thread), but i don't have any feedback from my last update...

Functionality wise, pretty much everything is there now, but that's all i can test/decide myself...

[przemratajczak]

i can give you Htop and top -H results but without comparition with "standard" vmaps it would be useless isn't it? It's because turning on them on continents at previous (mangos) version is improssible on weaker machines (CPU reach 100% and app is closed by freeze detector). Merging your work, while vmaps enabled on 2 of 4 continents seems that CPU does not even noticed any change i think ;-) If you wish I can submit comparition turned ON<->OFF

i don't know if it is related to this patch or MaNGOS core itself but .reload conf does not turn on/off vmaps on certain maps - reboot is required

also i was told that some creatures fall under ground since new vmaps implemented and some GOs (like doors in Naxxramas) are "transparent" for spells (but it was also bugged with MaNGOS version)

If you just say what you need we will do what ever we can to help you.

[yasuno]

I'm sorry Lynx3d i was not trying to make you upset, or ask any "newbie" questions. I was just wondering where you were at with this amazing project, as i stopped following it for a little bit. Thanks for the update, and again sorry to upset you.

Please keep working on it btw, its a great project, best one i have seen besides npc pathing.

[Unkle Nuke]

I'm a bit slow with implementing updates on my server these days, so I have yet to test vmap_rewrite. My server platform is currently Windows XP, so I'll have to give feedback on faramir's branch until I learn Linux well enough to compile and install my server for that OS.

My apologies for not being more helpful, Lynx3d, but things have just gotten in the way of patching and updating my server from 3.2.2a to 3.3.3a.

[KAPATEJIb]

Faramir118, can you update your fork by merging with master, not by cherry-pick of commits? It's very annoying to resolve conflicts because of different mangos commits. Also it adds many garbage into diff that i generated with this "git diff master vmap_rewrite -p > vmap_rewrite.patch"

It also won't compile

it gives couple of same errors at libmpq compile (VS90)

1>c:\\faramir118-mangos-439cb22\\contrib\\libmpq\\libmpq\\mpq.h(36) : fatal error C1083: Cannot open include file: 'stdint.h': No such file or directory

[KAPATEJIb]

Can someone upload recent Vmaps for Lynx3d branch? Or new extractor bins...

[shauren]

replace #include <stdint.h> with the following as visual studio doesnt have stdint

#ifdef _MSC_VER
typedef __int8 int8_t;
typedef unsigned __int8 uint8_t;
typedef __int16 int16_t;
typedef unsigned __int16 uint16_t;
typedef __int32 int32_t;
typedef unsigned __int32 uint32_t;
typedef __int64 int64_t;
typedef unsigned __int32 uint64_t;
#else
#include <stdint.h>
#endif

[KAPATEJIb]

replace #include <stdint.h> with the following as visual studio doesnt have stdint

#ifdef _MSC_VER
typedef __int8 int8_t;
typedef unsigned __int8 uint8_t;
typedef __int16 int16_t;
typedef unsigned __int16 uint16_t;
typedef __int32 int32_t;
typedef unsigned __int32 uint32_t;
typedef __int64 int64_t;
typedef unsigned __int32 uint64_t;
#else
#include <stdint.h>
#endif

with this error count is decreased, but still...

Libmpq_VC90

1>extract.c
1>..\\..\\libmpq\\extract.c(27) : fatal error C1083: Cannot open include file: 'zlib.h': No such file or directory

vmap_assembler

1>Linking...
1>WorldModel.obj : error LNK2001: unresolved external symbol "double __cdecl G3D::inf(void)" (?inf@G3D@@YANXZ)
1>MapTree.obj : error LNK2019: unresolved external symbol "double __cdecl G3D::inf(void)" (?inf@G3D@@YANXZ) referenced in function "public: float __thiscall VMAP::StaticMapTree::getHeight(class G3D::Vector3 const &)const " (?getHeight@StaticMapTree@VMAP@@QBEMABVVector3@G3D@@@Z)
1>ModelInstance.obj : error LNK2001: unresolved external symbol "double __cdecl G3D::inf(void)" (?inf@G3D@@YANXZ)
1>TileAssembler.obj : error LNK2001: unresolved external symbol "double __cdecl G3D::inf(void)" (?inf@G3D@@YANXZ)
1>VMapManager2.obj : error LNK2001: unresolved external symbol "double __cdecl G3D::inf(void)" (?inf@G3D@@YANXZ)
1>MapTree.obj : error LNK2019: unresolved external symbol "public: void __thiscall G3D::Ray::set(class G3D::Vector3 const &,class G3D::Vector3 const &)" (?set@Ray@G3D@@QAEXABVVector3@2@0@Z) referenced in function "public: __thiscall G3D::Ray::Ray(class G3D::Vector3 const &,class G3D::Vector3 const &)" (??0Ray@G3D@@QAE@ABVVector3@1@0@Z)
1>ModelInstance.obj : error LNK2001: unresolved external symbol "public: void __thiscall G3D::Ray::set(class G3D::Vector3 const &,class G3D::Vector3 const &)" (?set@Ray@G3D@@QAEXABVVector3@2@0@Z)
1>WorldModel.obj : error LNK2001: unresolved external symbol "public: void __thiscall G3D::Ray::set(class G3D::Vector3 const &,class G3D::Vector3 const &)" (?set@Ray@G3D@@QAEXABVVector3@2@0@Z)
1>ModelInstance.obj : error LNK2019: unresolved external symbol "public: class G3D::Matrix3 __thiscall G3D::Matrix3::inverse(float)const " (?inverse@Matrix3@G3D@@QBE?AV12@M@Z) referenced in function "public: __thiscall VMAP::ModelInstance::ModelInstance(class VMAP::ModelSpawn const &,class VMAP::WorldModel *)" (??0ModelInstance@VMAP@@QAE@ABVModelSpawn@1@PAVWorldModel@1@@Z)
1>ModelInstance.obj : error LNK2019: unresolved external symbol "public: static class G3D::Matrix3 __cdecl G3D::Matrix3::fromEulerAnglesZYX(float,float,float)" (?fromEulerAnglesZYX@Matrix3@G3D@@SA?AV12@MMM@Z) referenced in function "public: __thiscall VMAP::ModelInstance::ModelInstance(class VMAP::ModelSpawn const &,class VMAP::WorldModel *)" (??0ModelInstance@VMAP@@QAE@ABVModelSpawn@1@PAVWorldModel@1@@Z)
1>TileAssembler.obj : error LNK2001: unresolved external symbol "public: static class G3D::Matrix3 __cdecl G3D::Matrix3::fromEulerAnglesZYX(float,float,float)" (?fromEulerAnglesZYX@Matrix3@G3D@@SA?AV12@MMM@Z)
1>ModelInstance.obj : error LNK2019: unresolved external symbol "public: float __thiscall G3D::Ray::intersectionTime(class G3D::AABox const &)const " (?intersectionTime@Ray@G3D@@QBEMABVAABox@2@@Z) referenced in function "public: bool __thiscall VMAP::ModelInstance::intersectRay(class G3D::Ray const &,float &,bool)const " (?intersectRay@ModelInstance@VMAP@@QBE_NABVRay@G3D@@AAM_N@Z)
1>TileAssembler.obj : error LNK2019: unresolved external symbol "public: static class G3D::ReferenceCountedPointer<class G3D::MemoryManager> __cdecl G3D::MemoryManager::create(void)" (?create@MemoryManager@G3D@@SA?AV?$ReferenceCountedPointer@VMemoryManager@G3D@@@2@XZ) referenced in function "public: __thiscall G3D::Table<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,unsigned int,struct HashTrait<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > >,struct EqualsTrait<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > > >::Table<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,unsigned int,struct HashTrait<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > >,struct EqualsTrait<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > > >(void)" (??0?$Table@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@IU?$HashTrait@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@@U?$EqualsTrait@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@@@G3D@@QAE@XZ)
1>TileAssembler.obj : error LNK2019: unresolved external symbol "public: void __thiscall G3D::ReferenceCountedObject::ReferenceCountedObject_zeroWeakPointers(void)" (?ReferenceCountedObject_zeroWeakPointers@ReferenceCountedObject@G3D@@QAEXXZ) referenced in function "private: void __thiscall G3D::ReferenceCountedPointer<class G3D::MemoryManager>::zeroPointer(void)" (?zeroPointer@?$ReferenceCountedPointer@VMemoryManager@G3D@@@G3D@@AAEXXZ)
1>TileAssembler.obj : error LNK2019: unresolved external symbol "public: enum G3D::Vector3::Axis __thiscall G3D::Vector3::PrimaryAxis(void)const " (?primaryAxis@Vector3@G3D@@QBE?AW4Axis@12@XZ) referenced in function "protected: void __thiscall BIH::subdivide(int,int,class std::vector<unsigned int,class std::allocator<unsigned int> > &,struct BIH::buildData &,struct AABound &,struct AABound &,int,int,class BIH::BuildStats &)" (?subdivide@BIH@@IAEXHHAAV?$vector@IV?$allocator@I@std@@@std@@AAUbuildData@1@AAUAABound@@2HHAAVBuildStats@1@@Z)
1>WorldModel.obj : error LNK2001: unresolved external symbol "public: enum G3D::Vector3::Axis __thiscall G3D::Vector3::PrimaryAxis(void)const " (?primaryAxis@Vector3@G3D@@QBE?AW4Axis@12@XZ)
1>TileAssembler.obj : error LNK2019: unresolved external symbol "float __cdecl G3D::fnan(void)" (?fnan@G3D@@YAMXZ) referenced in function "protected: void __thiscall BIH::subdivide(int,int,class std::vector<unsigned int,class std::allocator<unsigned int> > &,struct BIH::buildData &,struct AABound &,struct AABound &,int,int,class BIH::BuildStats &)" (?subdivide@BIH@@IAEXHHAAV?$vector@IV?$allocator@I@std@@@std@@AAUbuildData@1@AAUAABound@@2HHAAVBuildStats@1@@Z)
1>WorldModel.obj : error LNK2001: unresolved external symbol "float __cdecl G3D::fnan(void)" (?fnan@G3D@@YAMXZ)
1>..\\bin\\Win32_Release\\/vmap_assembler.exe : fatal error LNK1120: 9 unresolved externals
1>Build log was saved at "file://c:\\faramir118-mangos-439cb22\\contrib\\vmap_assembler\\VC90\\bin\\vmap_assembler__Win32_Release\\BuildLog.htm"
1>vmap_assembler - 19 error(s), 50 warning(s)

wmapExtractor3_VC90

1>ProgressBar.cpp
1>c1xx : fatal error C1083: Cannot open source file: '..\\..\\misc\\ProgressBar.cpp': No such file or directory

I'm using last Faramir's sources, i didn't use a pull/merge, just clone sources and start trying to compile

Vmap_assembler compiles fine with Lynx3d sources... but libmpq and vmap_extractor have same errors

[shauren]

project file isnt perfect. as for progressbar - i just removed them from project completely (didnt have to change sources)

try with these dependencies http://filebeam.com/acea9e0bbbf13092d803a2bedc9e871a - just extract that to main folder (trunk)

[KAPATEJIb]

project file isnt perfect. as for progressbar - i just removed them from project completely (didnt have to change sources)

try with these dependencies http://filebeam.com/acea9e0bbbf13092d803a2bedc9e871a - just extract that to main folder (trunk)

1>mpq.c
1>..\\..\\libmpq\\mpq.c(22) : fatal error C1083: Cannot open include file: 'config.h': No such file or directory
1>extract.c
1>..\\..\\libmpq\\extract.c(27) : fatal error C1083: Cannot open include file: 'zlib.h': No such file or directory

i will try it with VS 2010 later...

But anyway better to just upload recent extractor bins

[shauren]

sure thing, http://filebeam.com/5257952917815e976387bb30c8b5865a - built in release

included libmpq.lib should you ever need it

[Unkle Nuke]

VS2010?

That's half your trouble, right there. Numerous compile issues with that IDE and I'm not just speaking of faramir's branch.

Also, I believe faramir's Windows port of vmap_rewrite was developed using VC++ 2008, not VC++ 2010. Try using 9.0 to build faramir's source before complaining about errors with an unsupported beta compiler (VS 2010).

[zergtmn]

VS2010 is RTM already.

[Skirnir]

Gave it a try yesterday night (yeah had only 4 hours of sleep ...), so far: works as expected. Super fast vmap extraction, vmap size of ~500MB. Liquid type works, server worked smooth, thumbs up.

Tested with Debian Testing x86_64 i7, 4gb ram.

Regards

Skirnir

Edit: I forgot to say, I merged this branch with mangos master: http://github.com/Lynx3d/mangos/commit/0be80326d330f995e70536a03e28a6a17c390e05

[Lynx3d]

Hm okay faramir didn't really adapt VC90 files to his changes...

Should work again now with my repo.

Make sure you first build the libmpq solution, and then extractor with the same configuration.

It's because turning on them on continents at previous (mangos) version is improssible on weaker machines (CPU reach 100% and app is closed by freeze detector). Merging your work, while vmaps enabled on 2 of 4 continents seems that CPU does not even noticed any change i think ;-) If you wish I can submit comparition turned ON<->OFF

I'm not sure i understand you correctly, you haven't been using the "old" vmaps on continents recently because they would kill the core too often?

I know the old code was not quite that optimal, but i'd be surprised if the difference is as dramatic as "no noticeable load increase" and "server fails due to CPU load".

Of course a direct comparison would be great, but i can understand if switching back and forth is too much interruption for your players.

i don't know if it is related to this patch or MaNGOS core itself but .reload conf does not turn on/off vmaps on certain maps - reboot is required

Good question, i don't know if it ever worked, will try to look at it.

If you just say what you need we will do what ever we can to help you.

Hm you're right, maybe i should just ask the right questions myself to find out what all this code is worth...

Okay things particular interesting:

General

• * compiling, using: anything that does not work as expected?
  * stability: does memory leak indefinitely, or the core crash more often?
  * regressions: do LoS or height checks fail somewhere where they did work before? If so, where?
  

New Functionality

• * indoor check: do you get unmounted somewhere where you clearly should not get unmounted on retail, or vice versa?
  * area detection (need .gps priviledges to test): is reported zone/area faulty? (note that server does not evaluate "micro-zone" names like individual shop names in cities etc.)
  * water: do you drown somewhere you shouldn't, or do you not drown when you should?
  * fishing: any fishable water that does not work? Do fish swarms still work properly?
  

-edit-

Oh and forgot to say, please also mention platform you use, and revisions you tested (hash of my last commit, and other things merged in)

Merging with MaNGOS master should be fine, that's expected to work, of course you can try other branches, but understand that i will not try to support all combinations you may come up with.

[Diablox]

* stability: does memory leak indefinitely, or the core crash more often?

Last few weeks i been switching from old to new and back to old and now i am compiling again new vmaps. About memory leak i didnt notice any. When my mangos is warmed up it is using about 2 - 2.1GB of RAM and with new vmaps it is always there (right up to the moment when i need to restart it becose instance locks wont reset) so as far as i can tell it hasn't got any leaks. If i on other hand i use old vmaps server usualy is dead before he get even close to 2GB and today i had so many crashes with old vmaps that i had to turn them off. My OS is Ubuntu 10.04 x32. Vmaps are turned on on all maps and im using clean mangos with my fork of sd2 and ah bot. And at the end about cpu and memory usage in my case has gone down 5% - 10% depending on what was my playerd doing. Botom line is it is very stable and i didnt notice any memory leaks. This is just my expirience with this project maybe someone alse did had some isues but for me it is working (and working very good). Thx for this awsome patch

EDIT: idk if this is related to this patch but i do not have this problem with clean mangos. Dalaran now is not a sanctuary. Server does not even say that you are in dalaran. Sorry if it is not related

[xeross155]

Will try to merge it into my branch with my next update, however that might be a while before I will do that.

[Artkeep]

EDIT: idk if this is related to this patch but i do not have this problem with clean mangos. Dalaran now is not a sanctuary. Server does not even say that you are in dalaran. Sorry if it is not related

The same problem, fault maps, have any ideas?

[Lynx3d]

Did you disable vmaps on continents?

The old area hacks had been removed by arrai...probably need to find a solution fall back to those hacks when the required vmap is disabled.

[Diablox]

So it will only work with vmaps turned on for northend? Btw my vmaps are now off(need to upload new version of vmaps).