Load Checkpoint Fails

[yusufcananar]

Hello,

I use Ubuntu 24.04 and latest master branch of trick (>= 19.7.2). I just wanted to dump and load checkpoint using your sim control panel with one of trick sims trick/trick_sims/Cannon/SIM_cannon_analytic but I got several errors when loading checkpoint.

Note1: I have clean build of trick. Only change I made is from the Issue #1729 where @hchen99 is suggested to change a line from configure file.

Note2: I have tried with Ubuntu 22.04 19.6.0 release and I was able to load checkpoint without any issue. (No change made on trick source)

Steps to reproduce:
1- trick-CP at this sim trick/trick_sims/Cannon/SIM_cannon_analytic
2- run ./S_main_*.exe ./RUN_test/input.py
3- Click Start and wait a few seconds.
4- Click Freeze
5- Click Dump Chkpnt and save sth like that chkpnt_2.80
6- Click Start and wait a few seconds.
7- Click Freeze
8- Click Load Chkpnt and load the one you saved chkpnt_2.80
9- after waiting a sec, sim time should show the checkpoint time
10- Click Start
And this is the resulting error:

Trick Build Process Complete
fjorin@fjorin-PC:~/trick/trick_sims/Cannon/SIM_cannon_analytic$ ./S_main_Linux_13.2_x86_64.exe ./RUN_test/input.py 
/home/fjorin/trick/bin/trick-simcontrol   fjorin-PC 42765 &
|L   1|2024/08/09,12:21:13|fjorin-PC| |T 0|0.000000| Freeze ON. Simulation time holding at 0.000000 seconds.
|L   1|2024/08/09,12:21:20|fjorin-PC| |T 0|0.000000| Freeze OFF.
|L   1|2024/08/09,12:21:21|fjorin-PC| |T 0|1.100000| Freeze ON. Simulation time holding at 1.100000 seconds.
|L   1|2024/08/09,12:21:26|fjorin-PC| |T 0|1.100000| Dumped ASCII Checkpoint chkpnt_1.10.
|L   1|2024/08/09,12:21:28|fjorin-PC| |T 0|1.100000| Freeze OFF.
|L   1|2024/08/09,12:21:31|fjorin-PC| |T 0|3.700000| Freeze ON. Simulation time holding at 3.700000 seconds.
|L   1|2024/08/09,12:21:37|fjorin-PC| |T 0|3.700000| Load checkpoint file /home/fjorin/trick/trick_sims/Cannon/SIM_cannon_analytic/RUN_test/chkpnt_1.10.
|L   1|2024/08/09,12:21:37|fjorin-PC| |T 0|3.700000| Finished loading checkpoint file.  Calling restart jobs.
restart message port = 41831
|L   1|2024/08/09,12:21:37|fjorin-PC| |T 0|1.100000| restart variable server message port = 42765
|L   1|2024/08/09,12:21:39|fjorin-PC| |T 0|1.100000| Freeze OFF.
Process terminated by signal SIGSEGV

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[New LWP 1793686]
[New LWP 1793643]
[New LWP 1793538]
[New LWP 1793533]
[New LWP 1793532]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fa9a29107e3 in __GI___wait4 (pid=1794044, stat_loc=stat_loc@entry=0x7ffc34a24ba8, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30

warning: 30     ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory

Thread 6 (Thread 0x7fa9a0c006c0 (LWP 1793532) "MessageListen"):
#0  0x00007fa9a2926cd7 in __GI___select (nfds=6, readfds=0x7fa9a0bffd70, writefds=0x0, exceptfds=0x0, timeout=0x7fa9a0bffd60) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x00005793fe32ea23 in Trick::MessageTCDeviceListenThread::thread_body() ()
#2  0x00005793fe36a7fe in Trick::ThreadBase::thread_helper(void*) ()
#3  0x00007fa9a289ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#4  0x00007fa9a2929c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 5 (Thread 0x7fa9a02006c0 (LWP 1793533) "VarServListen"):
#0  0x00007fa9a2926cd7 in __GI___select (nfds=7, readfds=0x7fa9a01ff850, writefds=0x0, exceptfds=0x0, timeout=0x7fa9a01ff840) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x00005793fe408368 in SystemInterface::select(int, fd_set*, fd_set*, fd_set*, timeval*) ()
#2  0x00005793fe7042f6 in Trick::TCPClientListener::checkForNewConnections() ()
#3  0x00005793fe372c02 in Trick::VariableServerListenThread::thread_body() ()
#4  0x00005793fe36a7fe in Trick::ThreadBase::thread_helper(void*) ()
#5  0x00007fa9a289ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#6  0x00007fa9a2929c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 4 (Thread 0x7fa99f8006c0 (LWP 1793538) "DR_Writer"):
#0  0x00007fa9a2898d61 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5793fea67168 <trick_data_record+616>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5793fea67168 <trick_data_record+616>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5793fea67168 <trick_data_record+616>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007fa9a289b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5793fea67170 <trick_data_record+624>, cond=0x5793fea67140 <trick_data_record+576>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x5793fea67140 <trick_data_record+576>, mutex=0x5793fea67170 <trick_data_record+624>) at ./nptl/pthread_cond_wait.c:627
#5  0x00005793fe2d8fe6 in Trick::DRDWriterThread::thread_body() ()
#6  0x00005793fe36a7fe in Trick::ThreadBase::thread_helper(void*) ()
#7  0x00007fa9a289ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#8  0x00007fa9a2929c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 3 (Thread 0x7fa99ee006c0 (LWP 1793643) "VS_SimControl"):
#0  0x00007fa9a28ecadf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa99edffd50, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x00007fa9a28f9a27 in __GI___nanosleep (req=req@entry=0x7fa99edffd50, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x00007fa9a292972c in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
#3  0x00005793fe37f00e in Trick::VariableServerSessionThread::thread_body() ()
#4  0x00005793fe36a7fe in Trick::ThreadBase::thread_helper(void*) ()
#5  0x00007fa9a289ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#6  0x00007fa9a2929c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 2 (Thread 0x7fa99e4006c0 (LWP 1793686) "VS_SimControl2"):
#0  0x00007fa9a28ecadf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa99e3ffd50, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x00007fa9a28f9a27 in __GI___nanosleep (req=req@entry=0x7fa99e3ffd50, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x00007fa9a292972c in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
#3  0x00005793fe37f00e in Trick::VariableServerSessionThread::thread_body() ()
#4  0x00005793fe36a7fe in Trick::ThreadBase::thread_helper(void*) ()
#5  0x00007fa9a289ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#6  0x00007fa9a2929c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 1 (Thread 0x7fa9a38c8740 (LWP 1793516) "S_main_Linux_13"):
#0  0x00007fa9a29107e3 in __GI___wait4 (pid=1794044, stat_loc=stat_loc@entry=0x7ffc34a24ba8, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
#1  0x00007fa9a29108eb in __GI___waitpid (pid=<optimized out>, stat_loc=stat_loc@entry=0x7ffc34a24ba8, options=options@entry=0) at ./posix/waitpid.c:38
#2  0x00007fa9a28585ab in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:172
#3  0x00005793fe2ff28c in Trick::Executive::signal_handler(int) ()
#4  0x00005793fe3004e9 in sig_hand(int) ()
#5  <signal handler called>
#6  0x00005793fe2deb6e in Trick::DataRecordGroup::data_record(double) ()
#7  0x00005793fe2dc25e in Trick::DataRecordGroup::call_function(Trick::JobData*) ()
#8  0x00005793fe3023b5 in Trick::FrameDataRecordGroup::call_function(Trick::JobData*) ()
#9  0x00005793fe364c93 in Trick::JobData::call() ()
#10 0x00005793fe2f5e2f in Trick::Executive::loop_single_thread() ()
#11 0x00005793fe2f593e in Trick::Executive::loop() ()
#12 0x00005793fe41ab35 in master(int, char**) ()
#13 0x00005793fe21f870 in main ()

A debugging session is active.

        Inferior 1 [process 1793516] will be detached.

Quit anyway? (y or n) [answered Y; input not from terminal]
[Inferior 1 (process 1793516) detached]

[rendayigit]

Hi!

I'm facing the same issue and I use Ubuntu 22.04. With the same scenario (And latest trick from the main branch at the time of writing this) I get the following error output:

pc@pc:~/trick/trick_sims/Cannon/SIM_cannon_analytic$ ./S_main_Linux_11.4_x86_64.exe RUN_test/input.py 
/home/pc/trick//bin/trick-simcontrol   pc 43895 &
|L   1|2024/08/09,14:27:42|pc| |T 0|0.000000| Freeze ON. Simulation time holding at 0.000000 seconds.
Gtk-Message: 14:27:43.256: Failed to load module "canberra-gtk-module"
|L   1|2024/08/09,14:27:50|pc| |T 0|0.000000| Freeze OFF.
|L   1|2024/08/09,14:27:52|pc| |T 0|1.300000| Freeze ON. Simulation time holding at 1.300000 seconds.
|L   1|2024/08/09,14:27:57|pc| |T 0|1.300000| Dumped ASCII Checkpoint chkpnt_1.30.
|L   1|2024/08/09,14:28:00|pc| |T 0|1.300000| Freeze OFF.
|L   1|2024/08/09,14:28:01|pc| |T 0|2.100000| Freeze ON. Simulation time holding at 2.100000 seconds.
|L   1|2024/08/09,14:28:10|pc| |T 0|2.100000| Load checkpoint file /home/pc/trick/trick_sims/Cannon/SIM_cannon_analytic/RUN_test/chkpnt_1.30.
|L   1|2024/08/09,14:28:10|pc| |T 0|2.100000| Finished loading checkpoint file.  Calling restart jobs.
restart message port = 41829
|L   1|2024/08/09,14:28:10|pc| |T 0|1.300000| restart variable server message port = 43895
|L   1|2024/08/09,14:28:12|pc| |T 0|1.300000| Freeze OFF.
Process terminated by signal SIGSEGV
[New LWP 85064]
[New LWP 85065]
[New LWP 85070]
[New LWP 85113]
[New LWP 85142]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007d86920ea42f in __GI___wait4 (pid=85268, stat_loc=stat_loc@entry=0x7ffc80d46768, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30      ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.

Thread 6 (Thread 0x7d868dcc8640 (LWP 85142) "VS_SimControl2"):
#0  0x00007d86920e57f8 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7d868dcc7ce0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x00007d86920ea677 in __GI___nanosleep (req=req@entry=0x7d868dcc7ce0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x00007d869211bf2f in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
#3  0x00006055ed571b68 in Trick::VariableServerSessionThread::thread_body() ()
#4  0x00006055ed55d572 in Trick::ThreadBase::thread_helper(void*) ()
#5  0x00007d8692094ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007d8692126850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7d868e4c9640 (LWP 85113) "VS_SimControl"):
#0  0x00007d86920e57f8 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7d868e4c8ce0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x00007d86920ea677 in __GI___nanosleep (req=req@entry=0x7d868e4c8ce0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x00007d869211bf2f in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
#3  0x00006055ed571b68 in Trick::VariableServerSessionThread::thread_body() ()
#4  0x00006055ed55d572 in Trick::ThreadBase::thread_helper(void*) ()
#5  0x00007d8692094ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007d8692126850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7d868ecca640 (LWP 85070) "DR_Writer"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x6055edc34288 <trick_data_record+616>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x6055edc34288 <trick_data_record+616>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x6055edc34288 <trick_data_record+616>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007d8692093a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x6055edc34290 <trick_data_record+624>, cond=0x6055edc34260 <trick_data_record+576>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x6055edc34260 <trick_data_record+576>, mutex=0x6055edc34290 <trick_data_record+624>) at ./nptl/pthread_cond_wait.c:627
#5  0x00006055ed4ca798 in Trick::DRDWriterThread::thread_body() ()
#6  0x00006055ed55d572 in Trick::ThreadBase::thread_helper(void*) ()
#7  0x00007d8692094ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007d8692126850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7d868f4cb640 (LWP 85065) "MessageListen"):
#0  0x00007d869211b63d in __GI___select (nfds=6, readfds=0x7d868f4cad00, writefds=0x0, exceptfds=0x0, timeout=0x7d868f4cacf0) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x00006055ed5206eb in Trick::MessageTCDeviceListenThread::thread_body() ()
#2  0x00006055ed55d572 in Trick::ThreadBase::thread_helper(void*) ()
#3  0x00007d8692094ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#4  0x00007d8692126850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7d868fccc640 (LWP 85064) "VarServListen"):
#0  0x00007d869211b63d in __GI___select (nfds=7, readfds=0x7d868fccb7f0, writefds=0x0, exceptfds=0x0, timeout=0x7d868fccb7e0) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x00006055ed5ebb9c in SystemInterface::select(int, fd_set*, fd_set*, fd_set*, timeval*) ()
#2  0x00006055ed8c7582 in Trick::TCPClientListener::checkForNewConnections() ()
#3  0x00006055ed565cd2 in Trick::VariableServerListenThread::thread_body() ()
#4  0x00006055ed55d572 in Trick::ThreadBase::thread_helper(void*) ()
#5  0x00007d8692094ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007d8692126850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7d8692e1d740 (LWP 85063) "S_main_Linux_11"):
#0  0x00007d86920ea42f in __GI___wait4 (pid=85268, stat_loc=stat_loc@entry=0x7ffc80d46768, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
#1  0x00007d86920ea3ab in __GI___waitpid (pid=<optimized out>, stat_loc=stat_loc@entry=0x7ffc80d46768, options=options@entry=0) at ./posix/waitpid.c:38
#2  0x00007d8692050bdb in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:171
#3  0x00006055ed4f1110 in Trick::Executive::signal_handler(int) ()
#4  0x00006055ed4f2603 in sig_hand(int) ()
#5  <signal handler called>
#6  0x00006055ed4d003e in Trick::DataRecordGroup::data_record(double) ()
#7  0x00006055ed4cd8bc in Trick::DataRecordGroup::call_function(Trick::JobData*) ()
#8  0x00006055ed4f4789 in Trick::FrameDataRecordGroup::call_function(Trick::JobData*) ()
#9  0x00006055ed557bdf in Trick::JobData::call() ()
#10 0x00006055ed4e7504 in Trick::Executive::loop_single_thread() ()
#11 0x00006055ed4e6fa2 in Trick::Executive::loop() ()
#12 0x00006055ed5fd000 in master(int, char**) ()
#13 0x00006055ed41948b in main ()

A debugging session is active.

        Inferior 1 [process 85063] will be detached.

Quit anyway? (y or n) [answered Y; input not from terminal]
[Inferior 1 (process 85063) detached]

[hchen99]

Did you mean that trick 19.6.0 works for the stated dump checkpoint and load checkpoint scenario while the lasted trick doesn't?

[yusufcananar]

Yes.

[hchen99]

Ok, looks to me 19.7.2 and earlier work for the stated scenario however the latest is having an issue. Will look into this.

[hchen99]

Yes.

Give it a try by turning off frame log

[yusufcananar]

I have turned frame log off (I think this is the right way please correct me if i used wrong https://nasa.github.io/trick/documentation/simulation_capabilities/Frame-Logging.html ) and Here is my input.py ->

trick.var_server_set_port(5556)
trick.frame_log_on()
trick.frame_log_off()

exec(open("Modified_data/realtime.py").read())
#exec(open("RUN_test/asdr.py").read())

print("Current HOST:PORT => {}:{} ".format(trick.var_server_get_hostname(), trick.var_server_get_port()))

I got the same result:

fjorin@fjorin-PC:~/Workspace/satellite-sil-simulation$ ./S_main_Linux_13.2_x86_64.exe ./RUN_test/input.py 
Current HOST:PORT => fjorin-PC:5556 
/home/fjorin/trick/bin/trick-simcontrol   fjorin-PC 5556 &
|L   1|2024/08/13,09:33:21|fjorin-PC| |T 0|0.000000| Freeze ON. Simulation time holding at 0.000000 seconds.
|L   1|2024/08/13,09:33:24|fjorin-PC| |T 0|0.000000| Freeze OFF.
|L   1|2024/08/13,09:33:25|fjorin-PC| |T 0|1.900000| Freeze ON. Simulation time holding at 1.900000 seconds.
|L   1|2024/08/13,09:33:27|fjorin-PC| |T 0|1.900000| Freeze OFF.
|L   1|2024/08/13,09:33:30|fjorin-PC| |T 0|4.300000| Freeze ON. Simulation time holding at 4.300000 seconds.
|L   1|2024/08/13,09:33:34|fjorin-PC| |T 0|4.300000| Dumped ASCII Checkpoint chkpnt_4.30.
|L   1|2024/08/13,09:33:36|fjorin-PC| |T 0|4.300000| Freeze OFF.
|L   1|2024/08/13,09:33:40|fjorin-PC| |T 0|8.400000| Freeze ON. Simulation time holding at 8.400000 seconds.
|L   1|2024/08/13,09:33:45|fjorin-PC| |T 0|8.400000| Load checkpoint file /home/fjorin/Workspace/satellite-sil-simulation/RUN_test/chkpnt_4.30.
|L   1|2024/08/13,09:33:45|fjorin-PC| |T 0|8.400000| Finished loading checkpoint file.  Calling restart jobs.
restart message port = 38959
|L   1|2024/08/13,09:33:45|fjorin-PC| |T 0|4.300000| variable server restart user_port requested set fjorin-PC:5556
|L   1|2024/08/13,09:33:46|fjorin-PC| |T 0|4.300000| Freeze OFF.
Process terminated by signal SIGSEGV

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[New LWP 2553918]
[New LWP 2553891]
[New LWP 2553847]
[New LWP 2553842]
[New LWP 2553841]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x0000767f5cd107e3 in __GI___wait4 (pid=2554044, stat_loc=stat_loc@entry=0x7ffefd1c3228, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30

warning: 30     ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory

Thread 6 (Thread 0x767f5b0006c0 (LWP 2553841) "MessageListen"):
#0  0x0000767f5cd26cd7 in __GI___select (nfds=6, readfds=0x767f5afffd70, writefds=0x0, exceptfds=0x0, timeout=0x767f5afffd60) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x000059d0badf2ba9 in Trick::MessageTCDeviceListenThread::thread_body() ()
#2  0x000059d0bae2e82e in Trick::ThreadBase::thread_helper(void*) ()
#3  0x0000767f5cc9ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#4  0x0000767f5cd29c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 5 (Thread 0x767f5a6006c0 (LWP 2553842) "VarServListen"):
#0  0x0000767f5cd26cd7 in __GI___select (nfds=7, readfds=0x767f5a5ff850, writefds=0x0, exceptfds=0x0, timeout=0x767f5a5ff840) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x000059d0baecb324 in SystemInterface::select(int, fd_set*, fd_set*, fd_set*, timeval*) ()
#2  0x000059d0bb1c72b2 in Trick::TCPClientListener::checkForNewConnections() ()
#3  0x000059d0bae36c32 in Trick::VariableServerListenThread::thread_body() ()
#4  0x000059d0bae2e82e in Trick::ThreadBase::thread_helper(void*) ()
#5  0x0000767f5cc9ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#6  0x0000767f5cd29c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 4 (Thread 0x767f59c006c0 (LWP 2553847) "DR_Writer"):
#0  0x0000767f5cc98d61 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x59d0c0529dec <trick_data_record+620>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x59d0c0529dec <trick_data_record+620>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x59d0c0529dec <trick_data_record+620>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000767f5cc9b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x59d0c0529df0 <trick_data_record+624>, cond=0x59d0c0529dc0 <trick_data_record+576>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x59d0c0529dc0 <trick_data_record+576>, mutex=0x59d0c0529df0 <trick_data_record+624>) at ./nptl/pthread_cond_wait.c:627
#5  0x000059d0bad9d16c in Trick::DRDWriterThread::thread_body() ()
#6  0x000059d0bae2e82e in Trick::ThreadBase::thread_helper(void*) ()
#7  0x0000767f5cc9ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#8  0x0000767f5cd29c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 3 (Thread 0x767f592006c0 (LWP 2553891) "VS_SimControl"):
#0  0x0000767f5ccecadf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x767f591ffd50, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x0000767f5ccf9a27 in __GI___nanosleep (req=req@entry=0x767f591ffd50, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x0000767f5cd2972c in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
#3  0x000059d0bae4303e in Trick::VariableServerSessionThread::thread_body() ()
#4  0x000059d0bae2e82e in Trick::ThreadBase::thread_helper(void*) ()
#5  0x0000767f5cc9ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#6  0x0000767f5cd29c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 2 (Thread 0x767f53e006c0 (LWP 2553918) "VS_SimControl2"):
#0  0x0000767f5ccecadf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x767f53dffd50, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x0000767f5ccf9a27 in __GI___nanosleep (req=req@entry=0x767f53dffd50, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x0000767f5cd2972c in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
#3  0x000059d0bae4303e in Trick::VariableServerSessionThread::thread_body() ()
#4  0x000059d0bae2e82e in Trick::ThreadBase::thread_helper(void*) ()
#5  0x0000767f5cc9ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#6  0x0000767f5cd29c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 1 (Thread 0x767f5dccd740 (LWP 2553833) "S_main_Linux_13"):
#0  0x0000767f5cd107e3 in __GI___wait4 (pid=2554044, stat_loc=stat_loc@entry=0x7ffefd1c3228, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
#1  0x0000767f5cd108eb in __GI___waitpid (pid=<optimized out>, stat_loc=stat_loc@entry=0x7ffefd1c3228, options=options@entry=0) at ./posix/waitpid.c:38
#2  0x0000767f5cc585ab in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:172
#3  0x000059d0badc3412 in Trick::Executive::signal_handler(int) ()
#4  0x000059d0badc466f in sig_hand(int) ()
#5  <signal handler called>
#6  0x000059d0bada2cf4 in Trick::DataRecordGroup::data_record(double) ()
#7  0x000059d0bada03e4 in Trick::DataRecordGroup::call_function(Trick::JobData*) ()
#8  0x000059d0badc653b in Trick::FrameDataRecordGroup::call_function(Trick::JobData*) ()
#9  0x000059d0bae28cc3 in Trick::JobData::call() ()
#10 0x000059d0badb9fb5 in Trick::Executive::loop_single_thread() ()
#11 0x000059d0badb9ac4 in Trick::Executive::loop() ()
#12 0x000059d0baeddaf1 in master(int, char**) ()
#13 0x000059d0baa1ba89 in main ()

A debugging session is active.

        Inferior 1 [process 2553833] will be detached.

Quit anyway? (y or n) [answered Y; input not from terminal]
[Inferior 1 (process 2553833) detached]

[hchen99]

You got frame logging commands right. By default, it's off. However, Modified_data/realtime.py also has trick.frame_log_on() which needs to be commented out.

[yusufcananar]

Got it. I also commented out in realtime.py and it worked for cannon analytic.

Btw just curious, what is frame log for?

[hchen99]

The Frame Logging page you visited earlier and Frame-Logging section in Realtime Best Practices should have the info you're looking for

[sharmeye]

We're still investigating this but I wanted to mention it so that you don't think we've forgotten about it.