Parsing a variadic macro makes clang2py crash

[FloSim82]

Hello, following issues described in ticket #100, I confirm that I have an issue under Ubuntu 18.04 when parsing a variadic #define, with a specific include dependence between parsed files.

Let me explain, we want to parse 3 header files :

• in file lib1.h we have functions and the variadic macro
• in file lib2, we have a function
• in file lib3.h, we have a function AND include "lib1.h"

Now, here is the behaviour of clang2py:

• clang2py -k emstu --debug lib1.h => no issue (== we have "all" at the end)
• clang2py -k emstu --debug lib1.h lib2.h lib3.h => parsing issue, no "output"

In the second case, we have the following debug traces :

DEBUG:clangparser:ARCH sizes: long:c_int64 longdouble:c_long_double_t
DEBUG:codegen:Parsing input file lib1.h
DEBUG:clangparser:lib1.h:2: Found a MACRO_DEFINITION|LIB1_H|LIB1_H
DEBUG:handler:get_unique_name: name "LIB1_H"
DEBUG:utils:MACRO_DEFINITION: displayname:'LIB1_H'
DEBUG:handler:get_unique_name: name "LIB1_H"
DEBUG:handler:get_unique_name: name "LIB1_H"
DEBUG:utils:_literal_handling: displayname:'LIB1_H'
DEBUG:cursorhandler:literal has 1 tokens.[ LIB1_H ]
DEBUG:cursorhandler:cursor.type:INVALID
DEBUG:cursorhandler:token:LIB1_H tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:MACRO: #define LIB1_H True
DEBUG:clangparser:register: LIB1_H 
DEBUG:clangparser:lib1.h:15: Found a MACRO_DEFINITION|log_print_def|log_print_def
DEBUG:handler:get_unique_name: name "log_print_def"
DEBUG:utils:MACRO_DEFINITION: displayname:'log_print_def'
DEBUG:handler:get_unique_name: name "log_print_def"
DEBUG:handler:get_unique_name: name "log_print_def"
DEBUG:utils:_literal_handling: displayname:'log_print_def'
DEBUG:cursorhandler:literal has 23 tokens.[ log_print_def ( fmt , ... ) log_print ( NULL , 0 , DEBUG_INFO , NULL , LOG_DEF_PRINT , fmt , ## __VA_ARGS__ ) ]
DEBUG:cursorhandler:cursor.type:INVALID
DEBUG:cursorhandler:token:log_print_def tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:( tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:fmt tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:Undefined MACRO_DEFINITION token identifier : fmt
DEBUG:cursorhandler:token:, tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:... tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:) tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:log_print tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:Undefined MACRO_DEFINITION token identifier : log_print
DEBUG:cursorhandler:token:( tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:NULL tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:Undefined MACRO_DEFINITION token identifier : NULL
DEBUG:cursorhandler:token:, tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:0 tk.kd:    LITERAL tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:, tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:DEBUG_INFO tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:Undefined MACRO_DEFINITION token identifier : DEBUG_INFO
DEBUG:cursorhandler:token:, tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:NULL tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:Undefined MACRO_DEFINITION token identifier : NULL
DEBUG:cursorhandler:token:, tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:LOG_DEF_PRINT tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:Undefined MACRO_DEFINITION token identifier : LOG_DEF_PRINT
DEBUG:cursorhandler:token:, tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:fmt tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:Undefined MACRO_DEFINITION token identifier : fmt
DEBUG:cursorhandler:token:, tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:## tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:__VA_ARGS__ tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:Undefined MACRO_DEFINITION token identifier : __VA_ARGS__
DEBUG:cursorhandler:token:) tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:_literal_handling final_value: ['log_print_def', '(', UndefinedIdentifier(name=fmt), ',', '...', ')', UndefinedIdentifier(name=log_print), '(', UndefinedIdentifier(name=NULL), ',', '0', ',', UndefinedIdentifier(name=DEBUG_INFO), ',', UndefinedIdentifier(name=NULL), ',', UndefinedIdentifier(name=LOG_DEF_PRINT), ',', UndefinedIdentifier(name=fmt), ',', '##', UndefinedIdentifier(name=__VA_ARGS__), ')']
DEBUG:cursorhandler:MACRO: #define log_print_def(fmt, ...) log_print(NULL,0,DEBUG_INFO,NULL,LOG_DEF_PRINT,fmt,##__VA_ARGS__)
DEBUG:clangparser:register: log_print_def 
DEBUG:clangparser:lib1.h:12: Found a FUNCTION_DECL|lib1_init()|lib1_init
DEBUG:handler:get_unique_name: name "lib1_init"
DEBUG:utils:FUNCTION_DECL: displayname:'lib1_init'
DEBUG:handler:get_unique_name: name "lib1_init"
DEBUG:clangparser:register: lib1_init 
DEBUG:clangparser:lib1.h:14: Found a FUNCTION_DECL|log_print(const char *, int, int, const char *, int, const char *, ...)|log_print
DEBUG:handler:get_unique_name: name "log_print"
DEBUG:utils:FUNCTION_DECL: displayname:'log_print'
DEBUG:handler:get_unique_name: name "log_print"
DEBUG:handler:get_unique_name: name "module"
DEBUG:utils:PARM_DECL: displayname:'module'
DEBUG:handler:get_unique_name: name "const char *"
DEBUG:utils:POINTER: displayname:'const char *'
DEBUG:handler:get_unique_name: name "const char"
DEBUG:typehandler:POINTER: size:8 align:8 typ:TypeKind.CHAR_S
DEBUG:typehandler:POINTER: pointee type_name:'const char'
DEBUG:handler:get_unique_name: name "options"
DEBUG:utils:PARM_DECL: displayname:'options'
DEBUG:handler:get_unique_name: name "severity"
DEBUG:utils:PARM_DECL: displayname:'severity'
DEBUG:handler:get_unique_name: name "color"
DEBUG:utils:PARM_DECL: displayname:'color'
DEBUG:handler:get_unique_name: name "const char *"
DEBUG:utils:POINTER: displayname:'const char *'
DEBUG:handler:get_unique_name: name "const char"
DEBUG:typehandler:POINTER: size:8 align:8 typ:TypeKind.CHAR_S
DEBUG:typehandler:POINTER: pointee type_name:'const char'
DEBUG:handler:get_unique_name: name "output"
DEBUG:utils:PARM_DECL: displayname:'output'
DEBUG:handler:get_unique_name: name "fmt"
DEBUG:utils:PARM_DECL: displayname:'fmt'
DEBUG:handler:get_unique_name: name "const char *"
DEBUG:utils:POINTER: displayname:'const char *'
DEBUG:handler:get_unique_name: name "const char"
DEBUG:typehandler:POINTER: size:8 align:8 typ:TypeKind.CHAR_S
DEBUG:typehandler:POINTER: pointee type_name:'const char'
DEBUG:clangparser:register: log_print 
DEBUG:codegen:Parsing input file lib2.h
DEBUG:clangparser:lib2.h:2: Found a MACRO_DEFINITION|LIB2_H|LIB2_H
DEBUG:handler:get_unique_name: name "LIB2_H"
DEBUG:utils:MACRO_DEFINITION: displayname:'LIB2_H'
DEBUG:handler:get_unique_name: name "LIB2_H"
DEBUG:handler:get_unique_name: name "LIB2_H"
DEBUG:utils:_literal_handling: displayname:'LIB2_H'
DEBUG:cursorhandler:literal has 1 tokens.[ LIB2_H ]
DEBUG:cursorhandler:cursor.type:INVALID
DEBUG:cursorhandler:token:LIB2_H tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:MACRO: #define LIB2_H True
DEBUG:clangparser:register: LIB2_H 
DEBUG:clangparser:lib2.h:4: Found a INCLUSION_DIRECTIVE|lib3.h|lib3.h
WARNING:handler:INCLUSION_DIRECTIVE is not handled
DEBUG:handler:get_unique_name: name "lib3.h"
WARNING:handler:_do_nothing for INCLUSION_DIRECTIVE/lib3.h
DEBUG:clangparser:./lib3.h:2: Found a MACRO_DEFINITION|LIB3_H|LIB3_H
DEBUG:handler:get_unique_name: name "LIB3_H"
DEBUG:utils:MACRO_DEFINITION: displayname:'LIB3_H'
DEBUG:handler:get_unique_name: name "LIB3_H"
DEBUG:handler:get_unique_name: name "LIB3_H"
DEBUG:utils:_literal_handling: displayname:'LIB3_H'
DEBUG:cursorhandler:literal has 1 tokens.[ LIB3_H ]
DEBUG:cursorhandler:cursor.type:INVALID
DEBUG:cursorhandler:token:LIB3_H tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:MACRO: #define LIB3_H True
DEBUG:clangparser:register: LIB3_H 
DEBUG:clangparser:./lib3.h:4: Found a INCLUSION_DIRECTIVE|lib1.h|lib1.h
DEBUG:handler:get_unique_name: name "lib1.h"
WARNING:handler:_do_nothing for INCLUSION_DIRECTIVE/lib1.h
DEBUG:clangparser:./lib1.h:2: Found a MACRO_DEFINITION|LIB1_H|LIB1_H
DEBUG:handler:get_unique_name: name "LIB1_H"
DEBUG:utils:MACRO_DEFINITION: displayname:'LIB1_H'
DEBUG:handler:get_unique_name: name "LIB1_H"
DEBUG:handler:get_unique_name: name "LIB1_H"
DEBUG:utils:_literal_handling: displayname:'LIB1_H'
DEBUG:cursorhandler:literal has 1 tokens.[ LIB1_H ]
DEBUG:cursorhandler:cursor.type:INVALID
DEBUG:cursorhandler:token:LIB1_H tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:MACRO: #define LIB1_H True
INFO:cursorhandler:Redefinition of LIB1_H None->True
DEBUG:clangparser:./lib1.h:15: Found a MACRO_DEFINITION|log_print_def|log_print_def
DEBUG:handler:get_unique_name: name "log_print_def"
DEBUG:utils:MACRO_DEFINITION: displayname:'log_print_def'
DEBUG:handler:get_unique_name: name "log_print_def"
DEBUG:handler:get_unique_name: name "log_print_def"
DEBUG:utils:_literal_handling: displayname:'log_print_def'
DEBUG:cursorhandler:literal has 23 tokens.[ log_print_def ( fmt , ... ) log_print ( NULL , 0 , DEBUG_INFO , NULL , LOG_DEF_PRINT , fmt , ## __VA_ARGS__ ) ]
DEBUG:cursorhandler:cursor.type:INVALID
DEBUG:cursorhandler:token:log_print_def tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:( tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:fmt tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:Undefined MACRO_DEFINITION token identifier : fmt
DEBUG:cursorhandler:token:, tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:... tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:) tk.kd:PUNCTUATION tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION
DEBUG:cursorhandler:token:log_print tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION

Thanks for your support

[FloSim82]

To help here are the 3 header files:

$ cat lib1.h

#ifndef LIB1_H
#define LIB1_H

#ifdef __cplusplus

extern "C"
{
#endif

int     lib1_init();

void    log_print(const char * module, int options, int severity, const char * color, int output, const char * fmt, ...);
#define log_print_def(fmt, ...)     log_print(NULL, 0, DEBUG_INFO, NULL, LOG_DEF_PRINT, fmt, ##__VA_ARGS__)


#ifdef __cplusplus
}
#endif

#endif

$ cat lib2.h

#ifndef LIB2_H
#define LIB2_H

#include "lib3.h"

#ifdef __cplusplus

extern "C"
{
#endif

int lib2_init();

#ifdef __cplusplus
}
#endif

#endif

$ cat lib3.h

#ifndef LIB3_H
#define LIB3_H

#include "lib1.h"

#ifdef __cplusplus

extern "C"
{
#endif

int lib3_init(void);

#ifdef __cplusplus
}
#endif

#endif