Merge 4-0-85 to backup_sync #88

They may fail throwing an unrecoverable exception in case of unresolved includes etc. Also, some minor clean-up.

For: https://shiftleftinc.atlassian.net/jira/software/c/projects/SEN/issues/SEN-2777

For: https://shiftleftinc.atlassian.net/browse/SEN-2838

This PR handles: * Move parser tests from the `deprecated` frontend to the new `ruby` frontend. * Fixed parser issues with HashLiterals * Fixed parser issues with one-liner class definitions * Fixed parser issues with arguments in functions

Brings in latest babel and typescript to astgen.

* [ruby] Arrow Lambda Parameter Fix Tests various lambdas and fixes `self` parameter name in methods.

partly to minify the flatgraph diff

* minor work on reachingDef * fmt --------- Co-authored-by: Michael Pollmeier <[email protected]>

This astgen version skipps giant, unparsable files with EMSCRIPTEN code now by default. For: https://shiftleftinc.atlassian.net/browse/SEN-2797

…joernio#4719) For: https://shiftleftinc.atlassian.net/browse/SEN-2777

Also handle fullnames with generics correctly when stubbing types For: https://shiftleftinc.atlassian.net/browse/SEN-2840

This PR adds a few more parser tests based on tests in the `querying/` folder for Ruby.

Inheritance via `<` in Ruby can be arbitrary extensions which warrant post-processing analysis, so this removes any attempt to resolve the type at AST creation to allow for a post-processing pass to handle this instead.

Does not exist anymore.

As the `<body>` call is synthetic and meant to be immediately deterministic, so there is no reason it should be re-determined.

* upgrade cpg and adapt * use released cpg

…joernio#4729) ``` [warn] -- [E092] Pattern Match Unchecked Warning: /home/mp/Projects/shiftleft/joern.1/joern-cli/frontends/pysrc2cpg/src/test/scala/io/joern/pysrc2cpg/PySrc2CpgFixture.scala:74:61 [warn] 74 | path.resultPairs().collect { case (firstElement: String, secondElement: Option[Integer]) => [warn] | ^ [warn] |the type test for Option[Integer] cannot be checked at runtime because its type arguments can't be determined from Option[Int] [warn] | [warn] | longer explanation available when compiling with `-explain` } ```

Earlier, if we pass the directory path which contains multiple go modules. Processing was done with all the `.go` files mapped to single `go.mod` file. With this change, we have segregated the processing by first isolating all the files mapped to respective `go.mod`. This will also make sure to cleanup the memory footprint after every module is being processed. However, this will increase the processing when used with download dependency as it will process all the `go.mod` files for identifying and processing used dependencies.

This PR changes the modelling of Singleton methods on objects. Ex: ```ruby class Animal; end animal = Animal.new class << animal def bark "woof" end end ``` Creates a lambda method for `animal.bark`, and assigns `animal.bark = methodRef(bark)` Resolves joernio#4721

- we had one stackoverflow in fullname - evaluation.getOverload may return null

…oernio#4733) * Fix always-crashing array initializer type resolution * Fix exception when trying to cast ArrayType to ClassOrInterfaceType * Fix formatting

…g merge (joernio#4737) * minify the flatgraph diff by bringing various things in before the big merge * fmt * revert accidental change

For new frontends with potentially malformed ASTs, type recovery must safely handle instances where members don't have AST parents.

The parser emits calls to `super` as different from simple calls, this PR handles them.

Also: no more empty method fullnames

* Safely handles the case when `super` call has a `null` argument from the parser * Shadows keywords when they are used as keys in association keys for named arguments in calls * Handles singleton methods in implicit returns

This PR fixes a bug where method members were not correctly linked to surrounding methods' bound type decls. Additionally, this handles `return` statements without any proceeding expression. Resolves joernio#4732

The initial implementation of the edge creation and captured local node was wrongly added to the `Ast` object instead of the diff graph. This PR rectifies this.

Again for: ShiftLeftSecurity/codescience#7730

Due to Ruby method references being called via a `.call()` method, the type ref that should be emitted, is one that contains the `.call` method, but is not necessarily the same type as the one bound to the actual lambda. This makes that change, however, data-flow no longer works in the open-source data-flow tracker as this is not supported.

* [ruby] Remodelled methods defined on singleton objects to no longer be lambdas * [ruby] Review comments

* Renamed `:program` to `<main>` * Replaced `:` method separator to `.` * Removed `<global>` from full names of types and methods

In the case of a type or method re-definition, the full name is ensured to be unique by a set that tracks all full-names for that compilation unit, and a counter. Resolves joernio#4742

This PR remodels `yield` calls as explicit invocations of the implicit or explicit block parameter of the surrounding method. Resolves joernio#4760

Getters and setters were modelled as their CPG operations in isolation, and not using the ordinary AST creator hooks. This PR lowers the getters and setters as their `RubyNode` equivalents, and hands off the AST creation to `astForMethodDeclaration`.

* [php2cpg] Support array/list unpacking in assignment * [php2cpg] Rename method and fix some tests * [php2cpg] code clean and improved test * [php2cpg] improved test

) * IF Cfg creation for if-statements with empty `then` block. * Update joern-cli/frontends/c2cpg/src/test/scala/io/joern/c2cpg/passes/cfg/CfgCreationPassTests.scala Co-authored-by: maltek <[email protected]> --------- Co-authored-by: maltek <[email protected]>

- Changed test constructs used to test the CFG creation to return List instead of Set. This avoids deduplication which is required for correct tests because the number of edges between two nodes matters. - The above change unveiled a problem with the for-statement CFG: Duplicate edge between loop condition and body

* [ruby] Moved precedence for keywords down, added special handling on local identifiers to check for keywords and member access * [ruby] Finished test for member call with reserved keyword * [ruby] remove debug print from RubyNodeCreator * [ruby] Fixed edge case for reserved keywords * [ruby] Added more checks for keyword handling, added test cases for :: syntax and having a base with the same name as a reserved keyword * [ruby] Added antlr debug flag to ruby config * [ruby] Added edge case for keyword handling

* [php2cpg] Support array/list unpacking in assignment * [php2cpg] Rename method and fix some tests * [php2cpg] code clean and improved test * [php2cpg] improved test * [php2cpg] lowering the init part of foreach statement

…modifier (joernio#4768) Fixes these org.eclipse.cdt.core.dom.ast.ExpansionOverlapsBoundaryException for `.getSyntax` calls.

* [ruby] Added file content and offset to method and type decls * [ruby] Added test case for file content * [ruby] Remove empty curlys

…ernio#4780) * [php2cpg] Support array/list unpacking in assignment * [php2cpg] Rename method and fix some tests * [php2cpg] code clean and improved test * [php2cpg] improved test * [php2cpg] lowering the init part of foreach statement * Add a new operator to mark the creation point of an array. improved array unpacking. * Use `array()` call instead of emptyArray operator

With Ruby attribute assignments being calls, this can lead to costly data-flow tracking and these setters are rarely overridden with custom logic. This PR simplifies this model by representing the attribute assignment from a setter call to a direct field assignment. A similar adjustment is done to member getters. Misc: Simplified a parser bug workaround and linked it to an issue.

…arents) (joernio#4778)

* Added a script that parses and creates a sensible string from a slice that can be tested against. * Fixed a slicing bug where slicing direction was in the opposite direction of neighbour retrieval and ended up ignoring certain nodes. * Added entry in the `test-scripts` job to run the slice script, parse the slice, and assert the expected flow with `grep`. Resolves joernio#4783

There was a case that would mistakenly prepend `@` on a member that start with a capitalized first letter.

on stage: remove module-info.class from dependency jars - a hacky workaround for a scala3 compiler bug: scala/scala3#20421 Fixes joernio#4625

Methods were wrong for that.

`.getArrayModifiers` may contain null values for empty array init

This may throw (FileSystemLoopException, or any other reason why this dir may not be readable). We use Files.walkFileTree now. The only solution where a safe continue mechanism can be implemented. Fixes: https://shiftleftinc.atlassian.net/browse/SEN-2976

- implicit this param for CPP functions - identifiers that are actual member accesses are now transformed, e.g., this->varname if varname is a member and we are in the corresponding context

1. Added the support in the CfgCreator. A THROW control structure now breaks control flow. 2. Adjust c2cpg to generate such a control structure instead of a CALL node.

…#4808)

In this PR joernio#4807 the CFG for `throw` statements was changes in the way that they have no outgoing CFG edges. Since this might break assumptions of some CFG consuming code, we now create outgoing CFG edges from `throw` statements to the method exit nodes.

…ecl (joernio#4817)

* [ruby] Initial commit for new Ast Printer, basic methods implemented * [ruby] Initial AstPrinter finished * [ruby] String parser tests working on new AstPrinter * [ruby] ArrayParserTests, SingleAssignmentTests, BeginExpressionTests moved over to new AstPrinter for parsing tests * [ruby] BooleanParserTests moved to new parser test * [ruby] CaseConditionParserTests moved * [ruby] ControlStructureParserTests moved * [ruby] FieldAccess, HashLiteral, IndexAccess and InvocationWithoutParentheses parser tests moved * [ruby] InvocationWithParentheses moved * [ruby] MethodDef and ProcDef parser tests moved * [ruby] Range, regex, require parser tests moved * [ruby] Return and ternary parser tests moved * [ruby] UnlessStmt parser tests moved * [ruby] Ensure statement parser test moved * [ruby] DoBlock parser tests moved * [ruby] RescueClause parser tests moved * [ruby] MultipleAssignment moved * [ruby] ClassDecl moved * [ruby] Module parser tests moved * [ruby] Working on interpolations * [ruby] Cleanup * [ruby] Cleanup for PR * [ruby] removed newMatch flag

* [ruby] Added handling for String interpolated array literals * [ruby] Added type check in tests * cleanup * [ruby] Fixed parser tests with new parser test framework * [ruby] removed if check on element content

* [ruby] handling added for command literal. Modelled as exec call * [ruby] fixed failing parser test * [ruby] Removed print * [ruby] Added expanded command literal to new parser tests

Using the old SerializedCpg API hat no effect anymore.

* [ruby2cpg] Fix ImplicitRequirePass. - Fix method lookup via AST edges. The approach of looking up module methods via is not possible at the execution time of the pass because AST is not yet linked. I replaced this buy a fullname based regex lookup. We should likely just change the execution time to after AST linking. - The other changes are just for better readability and debugability. * Address review remarks. * Fix field access lookup. We are only interested on those field accesses which operator on "self" since those describe the exported/imported objects.

* [ruby] Fixed bracket assignments * [ruby] Add parser test for BracketAssignment

* [ruby] Added filter to remove file with empty contents on AstCreator side * [ruby] lowered log to info, changed expression * formatting

Symbols/identifiers used as based class where not taken into account for the implicit import creation.

* [ruby] Fix assignment parser test * [ruby] Move assigment parser test to correct subset

…oernio#4832)

* [ruby] Fixed regex parser tests * [ruby] Fixed last parser tests * [ruby] Use pattern matching and string interpolation for building initMethod span * fmt * remove comment

Although the parser rule for picking up pseudo-variables seems fine, it may sometimes choose to represent it as a local variable. This fixes this and adds the related tests.

* [ruby] `self` Param `REF` Edge Fixed issue where no `REF` edges between self params and their identifiers. * Fixed the rest of the cases where `self` would have no refs * Updated test expectation

As Ruby may consider `<<` as either shifting left (if LHS is a number) or as an append (if LHS is an array), one cannot statically approximate the operation. We thus consider this as a call named `<<`.

The implicit require calls that are generated after the AST creation had an older/out-dated format and lead to identifiers not being linked via REF edges. Changed require to `require_relative`

* [ruby] Fixed incorrect class/module name, added NamespaceDeclaration node type for classes/modules defined straight in a namespace * [ruby] Added Namespace block to diffgraph, updated class test * [ruby] update module test * revert changes * [ruby] Added file as parent node of namespace block generated in class/module decl in namespace * [ruby] review changes * [ruby] Remove unused fucntions * [ruby] removed unused import

* [ruby] Constructor Lowering Fix * Fixed constructor lowering structure as per joernio#4822 * Tested and fixed issue in parenthesis-less `new` member calls (could use a better long-term fix in the parser) * Testing and handling an expression base of constructor * Fixed some imports * Using `code(node)`

…4844)

* [python] Include Pipfile[.lock] * scalafmt

`self` parameters within the scope of methods reverts back to initial surrounding type information, and keeps `ANY` otherwise, e.g. `<main>` methods.

* Reverted implicit requires to use `require` instead of `require_relative` * Removed requirement for `ImplicitRequirePass` to need `programSummary` argument * Moved import processing to post-processing

Following the other frontends, migrates the non-deprecated post-processing passes to `x2cpg`

* [ruby] initial commit * [ruby] Added test for locals * [ruby] cleanup * [ruby] removed check for methodFullName on scope * [ruby] removed check for methodFullName on scope * [ruby] rename function

* [ruby] Changed methodFullName on callNode for Initialize to DynamicUnknown instead of Any * [ruby] test fixes * [ruby] fixed failing type recovery tests * remove println

…eResolver` (joernio#4855)

joernio#4856)

Fixed an issue where classes defined in namespaces had different fullNames for the class and singleton instance defined in the namespace. Also resolved an issue where methods defined in class defined in a namespace was not being defined under the new `TypeDecl`

* Modified `raise` calls to be control structures of type `THROW` * If a literal argument is given, this is then explicitly represented as a `StandardError.new`

…xpression` (joernio#4860) * [ruby] Updated rescue expressions to use TRY/CATCH/ELSE/FINALLY controlStructureNodes * [ruby] fixed failing control structure test

joernio#4861)

* Handles various access modifiers private/public/protected * Also considers file-level methods are private and initialize methods cannot be anything other than private * Accounts for nested types where their modifiers are separate from surrounding type

* [ruby] Lowered ||= and &&= assignment operators to required if statements * [ruby] remove empty params in scaladocs

Some calls such as `select` may be commonly shadowed by auto-loaded calls, thus, on an example basis, we will continue to comment out similar calls.

* [ruby] Added operatorExpression to commandArgument * [ruby] removed operatorExpression, added only ternaryExpression to command

For chained calls such as `a().b()`, where the base/receiver of `b` involves a call, to avoid invoking the call in both the base and receiver, we now assign a temporary variable to the first invocation and refer to it on the second invocation.

* [php2cpg] Improve performance. This php parser we use seems to have some warmup phase in which it has very poor performance which makes it very expensive to feed it single files like we so far did. This change invokes the php parser with groups of 20 files which on my machine give ~40% performance increase for the total frontend runtime on large projects. Bigger groups of files strangely only show marginal performance increases despite the fact that profiling the frontend indicates that there is still a lot of time wasted by the large amount of individual parser invocations. For now we keep it like this because for bigger groups we would run into argument length limitations which are especially a pain on systems like Windows. Sadly the php parser does not provide means to specify the to be parsed files other than as a list on the command line. So there would need to be some change there first before we could increase the groups size in a meaningful way.

…oernio#4871) * [php2cpg] Fixed the init method of static/const members of class * clean code

The current `foreach` implementation is based on an older approach that doesn't capture the semantics quite as well. This PR implements the newer `forAst` construct, which encourages de-sugaring and representing what is semantically happening underneath the `foreach` statement. Resolves joernio#4674 (comment)

* ghidra 11.1.2 * drop unused private class which doesn't compile any longer after an api change * avoid `+` which breaks some scripts

* [ruby] Fixed single-assignments without basic variables * [ruby] review comments * [ruby] fix failing test

joernio#4884) * [ruby] Add handling for ArrayLiteral used directly as argument in call * [ruby] Moved parser rule to indexingArgumentList, added ANTLR debugging flag to RubyParserTEst * rename ANTLR rule

* [ruby] Add handling for ArrayLiteral used directly as argument in call * [ruby] Moved parser rule to indexingArgumentList, added ANTLR debugging flag to RubyParserTEst * rename ANTLR rule * [ruby] Updated ANTLR rule for array elements to include decimal literals * [ruby] removed comment

* [ruby] update parameterList rule * [ruby] Add fixed parser test

…4879) * Allows a user of the pass to add additional type identifiers to consider for implicit imports (preference to internal types, motivation is that internal types are generally loaded later which works with zeitwerk's newest-first logic) * Handles qualified type names to be detected and imported * Added tests and better handling for constructors * Deduplicates require calls * Add core gems to `DependencyPass`

* [ruby] fixed naked splat on LHS * [ruby] fixed multiple RHS splat args * [ruby] Added test for middle splat arg

…ferent types of arguments (joernio#4890)

…tatements (joernio#4891)

* [ruby] Temp Var Call `code` Fix Fixed bug where splitting on `.` alone does not create a reasonable code property if parameters are not accounted for. * Improved code property construction

* Added `GroupedParameter` type and handling for new type * Fixed ANTLR grammar rules for different combinations of block parameters * Updated relevant parser tests

* Reduced redundant match done by `AstForTypesCreator->handleDefaultConstructor` * Refactored control structure handling into `AstForControlStructuresCreator` * Re-named `RubyNode` to `RubyExpression` as most-if-not-all Ruby constructs can be evaluated to some value, even statements (assignments, control structures, method definitions, etc. all return a value), and added `RubyStatement` to try to subtly define constructs that stand on their own more often that not -> this will be more fleshed out in Pt. 2 * Noted areas where more clean-up and refactoring is required, but want to keep this diff small-ish

* [semanticcpg] remove `runPass` * further remove "unnecessary" variable declarations

* Handles implicit returns of multi-assignments by returning an array of the LHS assignment targets. * For implicit returns of multi-assignments created as desugaring of splatted parameters, returns `nil` as per what is evaluated in the Ruby interpreter.

@main

…oernio#4900) as discussed in joernio#4896, we currently fail hard when the user passes additional frontend arguments, which we forward to XTypeRecoveryConfig.parse With this case we still print a warning, but at least don't error. Potentially better would be to filter out everything _but_ the expected arguments in this case... simple test case: foo.sc: ``` @main def main() = { importCode.python("tests/code/pythonsrc/simple.py", args=List("--exclude-regex", "test*")) } ``` ``` sbt stage ./joern --script foo.sc ```

Added handling for named arguments when using hash rocket syntax with "symbol" keys.

* All `require`-like call names are now a set under `ImportsPass` * These calls are now static calls with only the arguments as children

* [ruby] Fixed parser rules for lambda expressions to allow parameters without parentheses * [ruby] update parser rule

…oernio#4907) * [ruby] Added parser rules and AST lowering for beginless and endless range expressions * [ruby] changed ordering in indexing argument list

such loops are bugs that should be fixed. But they happen occasionally, and right now they lead to OOM and infinite loops. I'd prefer it if we handled that a bit more gracefully. The cost of the set in `.dedup` should be pretty low.

joernio#4908) * [ruby] add methodInvocationWithoutParentheses to RETURN argumentList * [ruby] Added check for arguments of jsonArg

Only static calls with import related names 'require', 'load" and so on are linked to import nodes. Before that we ended up with import nodes and links for calls like `Marshal.load(...)`.

Fixed an instance where something like `foo.bar` would represent `foo` as a `self.foo` field access instead of a `foo()` call, given that no `foo` variable is present.

* update cpg/flatgraph * empty commit to trigger GH actions

…4919)

…#4921)

If an annotation type full name cannot be resolved we now use `<unresolvedNamespace>.ClassName` instead of `ANY`.

In case of a field access after a map/array access, the following field access had invalid argument indicies because it triggered the code path for dynamic receiver calls. This is now fixed.

* Fixed issue where access modifiers in expression positions would be unhandled * Handling identifiers on LHS of associations as symbols

…nio#4920) * Refactor `Semantics` -> `FullNameSemantics` * Refactor `FlowSemantic`/`FlowNode`/`FlowPath` out of the FullNameSemantics parser * Rename `Parser` -> `FullNameSemanticsParser` * Introduce `Semantics` trait with `initialize` and `forMethod`

)

* [kotlin2cpg] Fix "nacked" call representation. Fix representation of "nacked" calls like `someFunc(someArg)`. These so far have always being represented as statically dispatched which is obviously not correct. * Handle static dispatch with instance argument case.

as brought up in joernio#4936 it can lead to confusing situations if type parameters are shadowed, so for the sake of readability and disambiguity alone we should enable this compiler warning IMO. That being said, I'd like to stress that it's not something fundamentally complicated, afaics. Simple example copied from joernio#4936: ```scala class Example[NodeType <: Object](node: NodeType) extends AnyVal{ def foo[NodeType](x: List[NodeType]): List[NodeType] = x } ``` These two type parameters in your Example class are not related at all. They happen to have the same name, which only means that within `def foo` we cannot reference the `NodeType` type from the `class Example`, but apart from that there's no connection at all between them. Semantically it's similar to something like this on the value level: the two `bar` variables are not related at all, they just happen to have the same name, and therefor we cannot reference the class-level `bar` variable from within `def baz`. ```scala class Foo { val bar = 42 def baz: Unit = { val bar = "123" () } } ```

* Added ANTLR profiling with the `--antlrProfile` frontend argument. * If enabled, will gather parser metrics and log them right next to the file that was parsed, as well as print any syntax ambiguities.

Removes the old frontend and related CLI arguments. Hard deadline 20 Sept 2024.

Profiled `ArrayTests` to detect ambiguity and decisions with high lookaheads and modified test fixture to print profiler logs if enabled. This led to converting certain array rules to use more specific rules and fall back to more general rules less often. Some small improvements on `railsgoat` measured with `time` command on `joern-parse`: ``` // With ambiguity 75.58s user 1.98s system 356% cpu 21.762 total 73.56s user 2.61s system 492% cpu 15.452 total 66.52s user 2.01s system 387% cpu 17.667 total // With reduced ambiguity 65.42s user 1.94s system 443% cpu 15.189 total 74.58s user 2.01s system 557% cpu 13.744 total 74.39s user 1.75s system 560% cpu 13.595 total ```

…and detect project and configuration names automatically (joernio#4934) Fix android project detection and automatic name detection for gradle dependency fetcher

Merge 4-0-85 to backup_sync #88

Are you sure you want to change the base?

Merge 4-0-85 to backup_sync #88

Commits on Jul 1, 2024

Commits on Jul 2, 2024

Commits on Jul 3, 2024

Commits on Jul 4, 2024

Commits on Jul 5, 2024

Commits on Jul 8, 2024

Commits on Jul 9, 2024

Commits on Jul 10, 2024

Commits on Jul 11, 2024

Commits on Jul 12, 2024

Commits on Jul 15, 2024

Commits on Jul 16, 2024

Commits on Jul 17, 2024

Commits on Jul 18, 2024

Commits on Jul 19, 2024

Commits on Jul 22, 2024

Commits on Jul 24, 2024

Commits on Jul 25, 2024

Commits on Jul 29, 2024

Commits on Jul 30, 2024

Commits on Jul 31, 2024

Commits on Aug 1, 2024

Commits on Aug 2, 2024

Commits on Aug 5, 2024

Commits on Aug 6, 2024

Commits on Aug 7, 2024

Commits on Aug 12, 2024

Commits on Aug 13, 2024

Commits on Aug 14, 2024

Commits on Aug 15, 2024

Commits on Aug 16, 2024

Commits on Aug 19, 2024

Commits on Aug 20, 2024

Commits on Aug 21, 2024

Commits on Aug 22, 2024

Commits on Aug 23, 2024

Commits on Aug 26, 2024

Commits on Aug 27, 2024

Commits on Aug 28, 2024

Commits on Aug 29, 2024

Commits on Aug 30, 2024

Commits on Sep 2, 2024

Commits on Sep 3, 2024

Commits on Sep 4, 2024

Commits on Sep 5, 2024

Commits on Sep 6, 2024

Commits on Sep 9, 2024

Commits on Sep 10, 2024

Commits on Sep 11, 2024

Commits on Sep 12, 2024

Commits on Sep 13, 2024

Commits on Sep 14, 2024

Commits on Sep 16, 2024

Commits on Sep 17, 2024

Commits on Sep 18, 2024

Commits on Sep 19, 2024

Commits on Sep 20, 2024

Commits on Sep 23, 2024

Commits on Sep 24, 2024