PyMem_Realloc, PyObject_Free, and other "Invalid read" valgrind errors in example sims

[ddj116]

The errors

Error signature looks like this, about 126 instances with SIM_ball_L1:

==18841== Invalid read of size 4
==18841==    at 0x513D86F: PyMem_Realloc (in /usr/lib64/libpython3.6m.so.1.0)
==18841==    by 0x50D7DD0: PyImport_ExtendInittab (in /usr/lib64/libpython3.6m.so.1.0)
==18841==    by 0x50D7E49: PyImport_AppendInittab (in /usr/lib64/libpython3.6m.so.1.0)
==18841==    by 0x5AD291: init_swig_modules (in /local/trick_issue_XYZ/trick/trick_sims/Ball/SIM_ball_L1/S_main_Linux_4.8_x86_64.exe)
==18841==    by 0x72969F: Trick::IPPython::init() (in /local/trick_issue_XYZ/trick/trick_sims/Ball/SIM_ball_L1/S_main_Linux_4.8_x86_64.exe)
==18841==    by 0x58B50D: InputProcessorSimObject::call_function(Trick::JobData*) (in /local/trick_issue_XYZ/trick/trick_sims/Ball/SIM_ball_L1/S_main_Linux_4.8_x86_64.exe)
==18841==    by 0x6A085A: Trick::JobData::call() (in /local/trick_issue_XYZ/trick/trick_sims/Ball/SIM_ball_L1/S_main_Linux_4.8_x86_64.exe)
==18841==    by 0x6480D9: Trick::Executive::call_input_processor() (in /local/trick_issue_XYZ/trick/trick_sims/Ball/SIM_ball_L1/S_main_Linux_4.8_x86_64.exe)
==18841==    by 0x64D375: Trick::Executive::init() (in /local/trick_issue_XYZ/trick/trick_sims/Ball/SIM_ball_L1/S_main_Linux_4.8_x86_64.exe)
==18841==    by 0x721B65: master(int, char**) (in /local/trick_issue_XYZ/trick/trick_sims/Ball/SIM_ball_L1/S_main_Linux_4.8_x86_64.exe)
==18841==    by 0x59018D: main (in /local/trick_issue_XYZ/trick/trick_sims/Ball/SIM_ball_L1/S_main_Linux_4.8_x86_64.exe)
==18841==  Address 0x101f2020 is 12 bytes after a block of size 4 alloc'd
==18841==    at 0x4C29F73: malloc (vg_replace_malloc.c:309)
==18841==    by 0x5C89D17: ??? (in /usr/lib64/libudunits2.so.0.1.0)
==18841==    by 0x5C89662: ??? (in /usr/lib64/libudunits2.so.0.1.0)
==18841==    by 0x5C8FC09: uaiNew (in /usr/lib64/libudunits2.so.0.1.0)
==18841==    by 0x5C8ECDA: ??? (in /usr/lib64/libudunits2.so.0.1.0)
==18841==    by 0x5C8EF79: ut_map_name_to_unit (in /usr/lib64/libudunits2.so.0.1.0)
==18841==    by 0x5C931FE: ??? (in /usr/lib64/libudunits2.so.0.1.0)
==18841==    by 0x5C948EF: ??? (in /usr/lib64/libudunits2.so.0.1.0)
==18841==    by 0x5C954A0: ??? (in /usr/lib64/libudunits2.so.0.1.0)
==18841==    by 0x76DC1CF: ??? (in /usr/lib64/libexpat.so.1.6.0)
==18841==    by 0x76DCB30: ??? (in /usr/lib64/libexpat.so.1.6.0)
==18841==    by 0x76DEBAE: XML_ParseBuffer (in /usr/lib64/libexpat.so.1.6.0)

How to replicate

This little script will replicate the issue. I ran this on FSL (CentOS 7.9) with gcc 4.8.5 but have also observed it with gcc 8.3.0 in my Gateway simulation and gcc 4.8.5 in my Ramtares simulation.

mkdir trick_issue_1312/ && cd trick_issue_1312        # Create a workspace
python3 -m venv .venv && source .venv/bin/activate  # Create python3 venv
git clone https://github.com/nasa/trick.git         # Get trick
cd trick && git checkout 19.4.0                     # Checkout to latest
./configure && make                                 # Build trick
cd trick_sims/Ball/SIM_ball_L1 && ../../../bin/trick-CP  # Build sim
valgrind ./S_main_Linux*.exe RUN_test/input.py >& valgrind_out.txt

This appears to be related to use of python3 -- with python2 I don't see any of these errors in my Ramtares project.

---

FYI @spfennell @jmpenn @jdeans289 -- If you have trouble replicating or other questions please let me know.

[sfennell2]

@ddj116 I followed your steps exactly and I am not getting an invalid read. Can I please see the output of your ./configure step?

[ddj116]

Sure, here's the output of ./configure:

checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... -std=gnu11
checking for g++... g++
checking whether the compiler supports GNU C++... yes
checking whether g++ accepts -g... yes
checking for g++ option to enable C++11 features... none needed
checking how to run the C++ preprocessor... g++ -E
checking for X... libraries , headers 
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for X11/Intrinsic.h... yes
checking for trick_source/data_products/fermi-ware... no
checking for main in -lxml2... yes
checking for libxml/tree.h... yes
checking for gawk... gawk
checking for gcc... (cached) gcc
checking whether the compiler supports GNU C... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to enable C11 features... (cached) -std=gnu11
checking whether the compiler supports GNU C++... (cached) yes
checking whether g++ accepts -g... (cached) yes
checking for g++ option to enable C++11 features... (cached) none needed
checking gcc version... 4.8.5
checking gcc version >= 4.8... yes
which: invalid option -- 's'
which: invalid option -- 't'
which: invalid option -- 'd'
which: invalid option -- '='
which: invalid option -- 'g'
which: invalid option -- 'n'
which: invalid option -- 'u'
which: invalid option -- '1'
which: invalid option -- '1'
checking for tee... /usr/bin/tee
checking for ld... /usr/bin/ld
checking for flex... flex
checking for lex output file root... lex.yy
checking for lex library... none needed
checking for library containing yywrap... no
checking whether yytext is a pointer... yes
checking for bison... /usr/bin/bison
checking for curl... /usr/bin/curl
checking for perl... /usr/bin/perl
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for perl version... 5.16.3
checking for zip... /usr/bin/zip
checking for python3... /local/<user>/trick_issue_XYZ/.venv/bin/python3
checking for python3.6-config... /usr/bin/python3.6-config
checking for gnuplot... /usr/bin/gnuplot
checking for swig... /usr/bin/swig
checking SWIG version... 2.0.10
checking for SWIG library... /usr/share/swig/2.0.10
checking for gawk... (cached) gawk
checking for javac... javac
checking javac version >= 1.8... yes
checking for mvn... /usr/bin/mvn
checking how to run the C preprocessor... /usr/bin/gcc -E
checking for egrep... /usr/bin/grep -E
checking whether /usr/bin/gcc is Clang... no
checking whether pthreads work with "-pthread" and "-lpthread"... yes
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking whether more special flags are required for pthreads... no
checking for PTHREAD_PRIO_INHERIT... yes
checking for perl module Text::Balanced... ok
checking for perl module Digest::MD5... ok
checking for llvm-config... /bin/llvm-config
checking for /usr/include/clang/Basic/LLVM.h... yes
checking for /usr/lib64/llvm/libLLVMSupport.a... yes
checking for clang... /usr/bin/clang
checking for /usr/lib64/llvm/libclangFrontend.a... yes
checking clang version >= 3.4.2... yes
checking if zlib is wanted... yes
checking for inflateEnd in -lz... yes
checking for zlib.h... yes
checking for udunits2.h... no
checking for udunits2/udunits2.h... yes
checking for main in -ludunits2... yes
checking for hdf5.h... yes
checking for main in -lhdf5... yes
checking for gsl/gsl_rng.h... yes
checking for main in -lgsl... yes
checking for gtest/gtest.h... yes
checking for civetweb.h... no
configure: creating ./config.status
config.status: creating share/trick/makefiles/config_user.mk
configure script successfully completed

And here's how the virtual environment manages the python3 version:

(.venv) user@user: /local/user/trick_issue_XYZ > ll .venv/bin/
total 32
-rw-r--r-- 1 user user 2215 Jul 22 13:01 activate
-rw-r--r-- 1 user user 1271 Jul 22 13:01 activate.csh
-rw-r--r-- 1 user user 2435 Jul 22 13:01 activate.fish
-rwxr-xr-x 1 user user  267 Jul 22 13:01 easy_install
-rwxr-xr-x 1 user user  267 Jul 22 13:01 easy_install-3.6
-rwxr-xr-x 1 user user  239 Jul 22 13:01 pip 
-rwxr-xr-x 1 user user  239 Jul 22 13:01 pip3
-rwxr-xr-x 1 user user  239 Jul 22 13:01 pip3.6
lrwxrwxrwx 1 user user    7 Jul 22 13:01 python -> python3
lrwxrwxrwx 1 user user   16 Jul 22 13:01 python3 -> /usr/bin/python3

(.venv) user@user: /local/user/trick_issue_XYZ > which python
/local/user/trick_issue_XYZ/.venv/bin/python
(.venv) user@user: /local/user/trick_issue_XYZ > python --version
Python 3.6.8

[ddj116]

I've got some more information since y'all are having trouble reproducing this. In the Trick lab the machine cordonbleu is CentOS 7.9, closely resembling the platform in which I see these errors in the FSL. I was unable to produce the error on that machine, which was surprising. Comparing the output of trick's ./configure I noticed the only real difference being SWIG versions:

In FSL, the system swig is 2.0.10 which configure picks up by default. Unfortunately, even when switching swig over to 3.0.12 (/software/x86_64/swig/3.0.12/bin/) I'm still getting 121 Invalid read errors similar to those shown above. The following script can be run in the FSL to reproduce this issue:

#!/bin/bash
mkdir trick_issue_1312/ && cd trick_issue_1312      # Create a workspace
python3 -m venv .venv && source .venv/bin/activate  # Create python3 venv
git clone https://github.com/nasa/trick.git         # Get trick
cd trick && git checkout 19.4.0                     # Checkout to latest
./configure --with-swig=/software/x86_64/swig/3.0.12/bin/ && make >& trick_build.txt   # Build trick
cd trick_sims/Ball/SIM_ball_L1 && ../../../bin/trick-CP  # Build sim
valgrind ./S_main_Linux*.exe RUN_test/input.py >& valgrind_out.txt

[spfennell]

https://stackoverflow.com/questions/1519276/is-it-normal-that-running-python-under-valgrind-shows-many-errors-with-memory

https://svn.python.org/projects/python/trunk/Misc/README.valgrind