Initial sanity check failed in ooprolog

[h4-ppy]

Hi, I'm trying to recover C++ object informations from 32bit tightvnc server with OOAnalyzer. (it's open source and you can check it's source code in the link)

I used following commands to analyze it, and an error occurs at the last command (ooprolog).

partition --serialize=tvnserver.ser --maximum-memory=128000 --no-semantics ./tvnserver.exe

ooanalyzer --serialize=tvnserver.ser --maximum-memory 128000 --no-semantics --prolog-facts=tvnserver-facts.pl --threads=16 --per-function-timeout=60000 ./tvnserver.exe

ooprolog --facts tvnserver-facts.pl --results tvnserver-results.pl --log-level=6 >ooprog.log

I got following error message and log.

  [29] prolog_stack:get_prolog_backtrace(100,[frame(29,clause(<clause>(0x5616ff3d5990),6),_23986616)|_23986604],[goal_term_depth(100)]) at /usr/local/lib/swipl/library/prolog_stack.pl:137
  [28] throw_with_backtrace(error(system_error(initialSanityChecks))) at /usr/local/share/pharos/prolog/oorules/util.pl:185
  [26] solve_internal at /usr/local/share/pharos/prolog/oorules/setup.pl:679
  [25] catch(user:solve_internal,_23986840,user:((_23986908=error(resource_error(private_table_space),_23986922)->complain_table_space(ooscript);_23986972=error(resource_error(stack),_23986986)->complain_stack_size(ooscript);true),throw(_23987018))) at /usr/local/lib/swipl/boot/init.pl:562
  [24] solve(ooscript) at /usr/local/share/pharos/prolog/oorules/setup.pl:617
  [23] psolve_no_halt('<garbage_collected>') at /usr/local/share/pharos/prolog/oorules/report.pl:23
  [22] catch(user:psolve_no_halt(stream(<stream>(0x5616ff400280))),_23987192,user:(print_message(error,_23987258),(globalHalt->halt(1);true))) at /usr/local/lib/swipl/boot/init.pl:562
  [21] catch_with_backtrace('<garbage_collected>','<garbage_collected>','<garbage_collected>') at /usr/local/lib/swipl/boot/init.pl:629
  [20] run_with_backtrace('<garbage_collected>') at /usr/local/bin/ooprolog:177
  [19] <meta call>
  [18] with_output_to(<stream>(0x5616ff5785d0),run_with_backtrace(psolve_no_halt(stream(<stream>(0x5616ff400280))))) <foreign>
  [17] setup_call_catcher_cleanup(user:(var('tvnserver-results.pl')->open_null_stream(<stream>(0x5616ff5785d0));open('tvnserver-results.pl',write,<stream>(0x5616ff5785d0))),user:with_output_to(<stream>(0x5616ff5785d0),run_with_backtrace(psolve_no_halt(stream(<stream>(0x5616ff400280))))),_23987610,user:close(<stream>(0x5616ff5785d0))) at /usr/local/lib/swipl/boot/init.pl:663
  [15] setup_call_catcher_cleanup(user:open('tvnserver-facts.pl',read,<stream>(0x5616ff400280)),user:setup_call_cleanup((var('tvnserver-results.pl')->open_null_stream(<stream>(0x5616ff5785d0));open('tvnserver-results.pl',write,<stream>(0x5616ff5785d0))),with_output_to(<stream>(0x5616ff5785d0),run_with_backtrace(psolve_no_halt(stream(<stream>(0x5616ff400280))))),close(<stream>(0x5616ff5785d0))),_23987820,user:close(<stream>(0x5616ff400280))) at /usr/local/lib/swipl/boot/init.pl:663
  [12] run([script('/usr/local/bin/ooprolog'),json(_23988098),ground(_23988118),rtti(true),guess(true),config(_23988178),stacklimit(200000000000),tablespace(200000000000),oorulespath(_23988238),halt(true),load_only(false),help(_23988298),facts('tvnserver-facts.pl'),results('tvnserver-results.pl'),loglevel(6)]) at /usr/local/bin/ooprolog:235
   [9] catch(user:main(['/usr/local/bin/ooprolog','--facts','tvnserver-facts.pl','--results','tvnserver-results.pl','--log-level=6']),_23988422,user:(print_message(error,_23988552),halt(1))) at /usr/local/lib/swipl/boot/init.pl:562
   [7] catch(user:main,_23988626,'$toplevel':true) at /usr/local/lib/swipl/boot/init.pl:562
   [6] catch_with_backtrace('<garbage_collected>','<garbage_collected>','<garbage_collected>') at /usr/local/lib/swipl/boot/init.pl:629

Note: some frames are missing due to last-call optimization.
Re-run your program in debug mode (:- debug.) to get more detail.
ERROR: tvnserver-facts.pl:132464:
ERROR:    Unknown message: error(system_error(initialSanityChecks))

$ tail ooprog.log
Starting reasonForward.
reasonMergeClasses_K(0x4bf2f0, 0x4bb6b8, 0x4737c0).
reasonMergeClasses_K(0x4bb6b8, 0x4bf2f0, 0x498550).
reasonForwardAsManyTimesAsPossible complete.
Entering stage 'Initial reasoning complete'.
failed.
Consistency checks failed.
Contradictory information about merging classes: Method1=0x4bf2f0 Method2=0x4bb6b8
Initial sanity check failed, indicating the OO rules are incorrect.
Please report this failure to the Pharos developers!

These files are tightvnc server executable that I used, and facts extracted from --prolog-facts option.

• tvnserver.zip
• tvnserver-facts.zip

---

Also I have some additional questions.

1. Could you explain me briefly, what is technical challanges in ooanalyzer to support 64-bit executable?
2. I observed a following warning message during step two.

OOAN[WARN ]: Unable to find parameter for new() call at 0x004047E2

push    98h             ; Size
mov     [ebp+68h+arg_8], ecx
; 125:   v47 = operator new(0x98u);
call    ??2@YAPAXI@Z    ; operator new(uint)

This is a disassembly and decompiled code at 0x004047E2 from IDA.
I think libpharos cannot recognize push instruction before new call, due to mov instruction between them. (this pattern often observed in optimized executable)
Is it possible to make a patch for this problem?

[sei-ccohen]

The 64-bit question has come up before, for obvious reasons. I'll link to an answer I've provided previously. There's no expectation that you would have been able to find it, I just didn't want to have to type it again. ;-) See this issue comment. If you have more specific questions about that I can answer them.

There's a known bug that's been interfering with recovering new() parameters for a while. I think we may have recently fixed it, so please try again with the commits we'll be releasing in the next few days.

The TightVNC program looks like it might make a good test program for OOAnalyzer, so thanks for that pointer.

[h4-ppy]

Thanks for your kind reply.
I had checked some related issues, but I missed the one you mentioned. Thanks :)

It seems that the new related warning message is not a root cause of this error..
Could you check why Contradictory information about merging classes error occured during ooprolog of tightvnc?

[sei-eschwartz]

I will look into the prolog issue with tightvnc. First I'll try to make sure I can reproduce the problem on the newly released version.

[sei-eschwartz]

I was able to reproduce the problem using your facts file on the new version.

0x4bf2f0 is AnsiStringStorage::vftable, and 0x4bb6b8 is StringStorage::vftable.

This is what reasonMergeClassesK says:

% If two methods are present on the same object, and we know neither method's class has a base
% class, the methods must be on the same object.

[sei-eschwartz]

reasonMergeClasses_K(0x4bb6b8, 0x4bf2f0, 0x498550).

0x4bb6b8 (StringStorage::vftable) is related to 0x498550, but is currently on class 0x4bf2f0 (AnsiStringStorage::vftable).

How is 0x4bb6b8 related to 0x498550?

1390162: Concluding factClassRelatedMethod(0x4737c0, 0x498550).
1397482: Merging class 0x4737c0 into 0x4bb6b8 ...
1397505: Retracting factClassRelatedMethod(0x4737c0, 0x498550) and asserting factClassRelatedMethod(0x4bb6b8, 0x498550) ...

reasonClassRelatedMethod_B(function=0x405a20, class1=0x4737c0, class2=0x498550, method1=0x4737c0, method2=0x498550).

Ok, I have a hypothesis of what is going on here. In 0x405a20, there are some conditional branches that control which constructor is invoked.

In one case, StringStorage::StringStorage is invoked. In another, AnsiStringStorage::AnsiStringStorage is invoked. We say that these methods are related to each other because they are both (statically) called on the same thisptr. But both calls cannot occur in the same execution because of the control flow.

I will need to think a little more about what to do about this.

[sei-eschwartz]

To be clear, this is a problem with reasonClassRelatedMethod_B.