Rare crash when launching multiple clients

[Reag]

Background

We're using Cesium to build a multiplayer system in Unity. As part of our testing protocol, we have multiple clients that will boot up and connect to a shared server.

Error

When our test script is executed, occasionally we will get the following stack trace:

0x00007FFD236353AC (KERNELBASE) RaiseException
0x00007FFCEB3D6BA7 (VCRUNTIME140) CxxThrowException
0x00007FFC9447B5E9 (CesiumForUnityNative-Runtime) DotNet_CesiumForUnity_CesiumPolygonRasterOverlay_CreateImplementation
0x00007FFC94479271 (CesiumForUnityNative-Runtime) DotNet_CesiumForUnity_CesiumPolygonRasterOverlay_CreateImplementation
0x00007FFC943EF8DD (CesiumForUnityNative-Runtime) DotNet_CesiumForUnity_CesiumPolygonRasterOverlay_CreateImplementation
0x00007FFC943EF5DD (CesiumForUnityNative-Runtime) DotNet_CesiumForUnity_CesiumPolygonRasterOverlay_CreateImplementation
0x00007FFC93A2B381 (CesiumForUnityNative-Runtime) DotNet_CesiumForUnity_CesiumPolygonRasterOverlay_CreateImplementation
0x00007FFC93A2BB15 (CesiumForUnityNative-Runtime) DotNet_CesiumForUnity_CesiumPolygonRasterOverlay_CreateImplementation
0x00007FFC939DB8CA (CesiumForUnityNative-Runtime) DotNet_CesiumForUnity_CesiumPolygonRasterOverlay_CreateImplementation
0x0000012EF928BE14 (Mono JIT Code) (wrapper managed-to-native) CesiumForUnity.Cesium3DTileset:DotNet_CesiumForUnity_Cesium3DTileset_Update (intptr,CesiumForUnity.Cesium3DTileset/ImplementationHandle)
0x0000012EF928BCC3 (Mono JIT Code) CesiumForUnity.Cesium3DTileset:Update ()

This crash only seems to affect one of the clients we spawn in our testing script. The testing script is below:

start "Test1" Client.exe -playerName 1 -serverIp 127.0.0.1 -hideDebugGUI true
start "Test2" Client.exe -playerName 2 -serverIp 127.0.0.1 -hideDebugGUI true

This crash seems to happen when ever we launch two or more clients within about 5 seconds of each other. If we wait longer than that, the crash does not seem to ever occur.

Technical Details

Unity Version: 22.3.18f1
CesiumForUnity Version: 1.9.0
We could not replicate the crash in the CesiumForUnity sample project

Conclusion

This is a rather annoying, but likely low priority, crash in our testing pipeline. It appears to be caused by something in the C++ implementation of the code, though my C++ skills are not sufficient to determine what. If there is a simple workaround we can implement, that would help us out a lot! The full stack trace and memory dump are available on request.

[kring]

It appears from the call stack that a regular C++ exception is being thrown, so it should be possible to find out which one, and the associated message, using the Visual Studio debugger. The trick will be to reproduce the problem when running in the debugger. This page may help:
https://learn.microsoft.com/en-us/visualstudio/debugger/debug-multiple-processes?view=vs-2022#BKMK_Automatically_start_an_process_in_the_debugger

It explains how to attach the debugger to a process even when it is launched by some other process.

In Visual Studio, if you go to Debug -> Windows -> Exception Settings and check the box next to "C++ Exceptions", the debugger should stop as soon as such an exception is raised. From there, look in the "Locals" tab for an exception message (you may need to move up a few levels in the call stack).

Better yet, if you can reproduce this in an isolated project that you can send to us, we can take look ourselves.

[Reag]

So after much testing, we finally were able to make a proper debug version of the cesium package. As such, we where able to get a more complete stack trace:

0x00007FFFDBAD543C (KERNELBASE) RaiseException
0x00007FFFCF65BBF1 (VCRUNTIME140D) CxxThrowException
0x00007FFF39C1BA21 (CesiumForUnityNative-Runtime) [C:\DEV\CesiumDevelopment\Packages\com.cesium.unity\native~\extern\cesium-native\CesiumAsync\src\SqliteCache.cpp:233] CesiumAsync::SqliteCache::createConnection 
0x00007FFF39C1776E (CesiumForUnityNative-Runtime) [C:\DEV\CesiumDevelopment\Packages\com.cesium.unity\native~\extern\cesium-native\CesiumAsync\src\SqliteCache.cpp:206] CesiumAsync::SqliteCache::SqliteCache 
0x00007FFF39ACF62E (CesiumForUnityNative-Runtime) [C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.38.33130\include\xutility:257] std::_Construct_in_place<CesiumAsync::SqliteCache,std::shared_ptr<spdlog::logger>,std::basic_string<char,std::char_traits<char>,std::allocator<char> > &,unsigned __int64 &> 
0x00007FFF39ACEEFE (CesiumForUnityNative-Runtime) [C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.38.33130\include\memory:2094] std::_Ref_count_obj2<CesiumAsync::SqliteCache>::_Ref_count_obj2<CesiumAsync::SqliteCache><std::shared_ptr<spdlog::logger>,std::basic_string<char,std::char_traits<char>,std::allocator<char> > &,unsigned __int64 &> 
0x00007FFF39AD04C9 (CesiumForUnityNative-Runtime) [C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.38.33130\include\memory:2769] std::make_shared<CesiumAsync::SqliteCache,std::shared_ptr<spdlog::logger>,std::basic_string<char,std::char_traits<char>,std::allocator<char> > &,unsigned __int64 &> 
0x00007FFF39ACE6CF (CesiumForUnityNative-Runtime) [C:\DEV\CesiumDevelopment\Packages\com.cesium.unity\native~\Runtime\src\UnityTilesetExternals.cpp:43] CesiumForUnityNative::getAssetAccessor 
0x00007FFF39ACEA7C (CesiumForUnityNative-Runtime) [C:\DEV\CesiumDevelopment\Packages\com.cesium.unity\native~\Runtime\src\UnityTilesetExternals.cpp:95] CesiumForUnityNative::createTilesetExternals 
0x00007FFF388F1DDF (CesiumForUnityNative-Runtime) [C:\DEV\CesiumDevelopment\Packages\com.cesium.unity\native~\Runtime\src\Cesium3DTilesetImpl.cpp:565] CesiumForUnityNative::Cesium3DTilesetImpl::LoadTileset 
0x00007FFF388F0838 (CesiumForUnityNative-Runtime) [C:\DEV\CesiumDevelopment\Packages\com.cesium.unity\native~\Runtime\src\Cesium3DTilesetImpl.cpp:118] CesiumForUnityNative::Cesium3DTilesetImpl::Update 
0x00007FFF38880444 (CesiumForUnityNative-Runtime) [C:\DEV\CesiumDevelopment\Packages\com.cesium.unity\native~\Runtime\generated-Standalone\src\DotNet\CesiumForUnity\Cesium3DTileset.cpp:742] DotNet_CesiumForUnity_Cesium3DTileset_Update 
0x000002777A5BB6DF (Mono JIT Code) (wrapper managed-to-native) CesiumForUnity.Cesium3DTileset:DotNet_CesiumForUnity_Cesium3DTileset_Update (intptr,CesiumForUnity.Cesium3DTileset/ImplementationHandle)
0x000002777A5BB173 (Mono JIT Code) [.\Library\PackageCache\com.cesium.unity@7e5a30123a17\Runtime\generated\Reinterop\Reinterop.RoslynSourceGenerator\Cesium3DTileset-generated.cs:270] CesiumForUnity.Cesium3DTileset:Update () 
0x0000027773536898 (Mono JIT Code) (wrapper runtime-invoke) object:runtime_invoke_void__this__ (object,intptr,intptr,intptr)

Opening the offending file (SqliteCache.cpp), we find the following segment:

  // create cache tables if not exist. Key -> Cache table: one-to-many
  // relationship
  char* createTableError = nullptr;
  status = CESIUM_SQLITE(sqlite3_exec)(
      this->_pImpl->_pConnection.get(),
      CREATE_CACHE_TABLE_SQL.c_str(),
      nullptr,
      nullptr,
      &createTableError);
  if (status != SQLITE_OK) {
    std::string errorStr(createTableError);
    CESIUM_SQLITE(sqlite3_free)(createTableError);
    throw std::runtime_error(errorStr);
  }

It would appear that if we get a status that isn't SQLITE_OK, we throw a runtime error. While this line might be fine in a native c++ environment, the effect of this seems to be that Unity simply commits die.