Merge 4-0-93 to backup_sync #95
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
They may fail throwing an unrecoverable exception in case of unresolved includes etc. Also, some minor clean-up.
This PR handles: * Move parser tests from the `deprecated` frontend to the new `ruby` frontend. * Fixed parser issues with HashLiterals * Fixed parser issues with one-liner class definitions * Fixed parser issues with arguments in functions
Brings in latest babel and typescript to astgen.
* [ruby] Arrow Lambda Parameter Fix Tests various lambdas and fixes `self` parameter name in methods.
partly to minify the flatgraph diff
* minor work on reachingDef * fmt --------- Co-authored-by: Michael Pollmeier <[email protected]>
This astgen version skipps giant, unparsable files with EMSCRIPTEN code now by default. For: https://shiftleftinc.atlassian.net/browse/SEN-2797
Also handle fullnames with generics correctly when stubbing types For: https://shiftleftinc.atlassian.net/browse/SEN-2840
This PR adds a few more parser tests based on tests in the `querying/` folder for Ruby.
Inheritance via `<` in Ruby can be arbitrary extensions which warrant post-processing analysis, so this removes any attempt to resolve the type at AST creation to allow for a post-processing pass to handle this instead.
Does not exist anymore.
As the `<body>` call is synthetic and meant to be immediately deterministic, so there is no reason it should be re-determined.
* upgrade cpg and adapt * use released cpg
…joernio#4729) ``` [warn] -- [E092] Pattern Match Unchecked Warning: /home/mp/Projects/shiftleft/joern.1/joern-cli/frontends/pysrc2cpg/src/test/scala/io/joern/pysrc2cpg/PySrc2CpgFixture.scala:74:61 [warn] 74 | path.resultPairs().collect { case (firstElement: String, secondElement: Option[Integer]) => [warn] | ^ [warn] |the type test for Option[Integer] cannot be checked at runtime because its type arguments can't be determined from Option[Int] [warn] | [warn] | longer explanation available when compiling with `-explain` } ```
Earlier, if we pass the directory path which contains multiple go modules. Processing was done with all the `.go` files mapped to single `go.mod` file. With this change, we have segregated the processing by first isolating all the files mapped to respective `go.mod`. This will also make sure to cleanup the memory footprint after every module is being processed. However, this will increase the processing when used with download dependency as it will process all the `go.mod` files for identifying and processing used dependencies.
This PR changes the modelling of Singleton methods on objects. Ex:
```ruby
class Animal; end
animal = Animal.new
class << animal
def bark
"woof"
end
end
```
Creates a lambda method for `animal.bark`, and assigns `animal.bark = methodRef(bark)`
Resolves joernio#4721
- we had one stackoverflow in fullname - evaluation.getOverload may return null
…oernio#4733) * Fix always-crashing array initializer type resolution * Fix exception when trying to cast ArrayType to ClassOrInterfaceType * Fix formatting
…g merge (joernio#4737) * minify the flatgraph diff by bringing various things in before the big merge * fmt * revert accidental change
For new frontends with potentially malformed ASTs, type recovery must safely handle instances where members don't have AST parents.
The parser emits calls to `super` as different from simple calls, this PR handles them.
Also: no more empty method fullnames
In case of a field access after a map/array access, the following field access had invalid argument indicies because it triggered the code path for dynamic receiver calls. This is now fixed.
* Fixed issue where access modifiers in expression positions would be unhandled * Handling identifiers on LHS of associations as symbols
…nio#4920) * Refactor `Semantics` -> `FullNameSemantics` * Refactor `FlowSemantic`/`FlowNode`/`FlowPath` out of the FullNameSemantics parser * Rename `Parser` -> `FullNameSemanticsParser` * Introduce `Semantics` trait with `initialize` and `forMethod`
* [kotlin2cpg] Fix "nacked" call representation. Fix representation of "nacked" calls like `someFunc(someArg)`. These so far have always being represented as statically dispatched which is obviously not correct. * Handle static dispatch with instance argument case.
as brought up in joernio#4936 it can lead to confusing situations if type parameters are shadowed, so for the sake of readability and disambiguity alone we should enable this compiler warning IMO. That being said, I'd like to stress that it's not something fundamentally complicated, afaics. Simple example copied from joernio#4936: ```scala class Example[NodeType <: Object](node: NodeType) extends AnyVal{ def foo[NodeType](x: List[NodeType]): List[NodeType] = x } ``` These two type parameters in your Example class are not related at all. They happen to have the same name, which only means that within `def foo` we cannot reference the `NodeType` type from the `class Example`, but apart from that there's no connection at all between them. Semantically it's similar to something like this on the value level: the two `bar` variables are not related at all, they just happen to have the same name, and therefor we cannot reference the class-level `bar` variable from within `def baz`. ```scala class Foo { val bar = 42 def baz: Unit = { val bar = "123" () } } ```
* Added ANTLR profiling with the `--antlrProfile` frontend argument. * If enabled, will gather parser metrics and log them right next to the file that was parsed, as well as print any syntax ambiguities.
Removes the old frontend and related CLI arguments. Hard deadline 20 Sept 2024.
Profiled `ArrayTests` to detect ambiguity and decisions with high lookaheads and modified test fixture to print profiler logs if enabled. This led to converting certain array rules to use more specific rules and fall back to more general rules less often. Some small improvements on `railsgoat` measured with `time` command on `joern-parse`: ``` // With ambiguity 75.58s user 1.98s system 356% cpu 21.762 total 73.56s user 2.61s system 492% cpu 15.452 total 66.52s user 2.01s system 387% cpu 17.667 total // With reduced ambiguity 65.42s user 1.94s system 443% cpu 15.189 total 74.58s user 2.01s system 557% cpu 13.744 total 74.39s user 1.75s system 560% cpu 13.595 total ```
…and detect project and configuration names automatically (joernio#4934) Fix android project detection and automatic name detection for gradle dependency fetcher
There were some unused parser rules adding unnecessary complexity to the parser
Added `simpleCommandArgumentList` to allow multiple call args for `methodInvocationWithoutParentheses`
* fix assignmentModulo
* fix assignmentXor
* fix assignmentOr
* fix assignmentAnd
* fix assignmentExponentiation
* fix assignmentShiftLeft
* fix assignment{Logical,Arithmetic}ShiftRight
* scalafmt
…#4954) These synthetic METHOD nodes did not have an offset at all. For: https://shiftleftinc.atlassian.net/browse/SEN-3250
…o#4955) As soon as `FullNameSemantics` started logging duplicate semantics (in joernio#4954), it was observed that `SemanticTestCpg` was passing a fresh Semantics instance to `EngineContext` instead of passing the one used by `OssDataFlow`. By coincidence, however, they were the same. Nevertheless, this patch guarantees they are the same.
Using the CDT API (getContainingFilename) now. For: joernio#4924
* [ruby] ANTLR Profiler Summary * Introduced global profiling to summarize rule and parse performance across the project * Added a shutdown hook to dump a summary of the profiler rules in a file `antlr_summary.log` at the root of the project with this information * Check parent exists before dumping. May not be there during test cases * Added note around shutdown hook
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Created by Github action