Merge 4-0-85 to backup_sync #88
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
They may fail throwing an unrecoverable exception in case of unresolved includes etc. Also, some minor clean-up.
This PR handles: * Move parser tests from the `deprecated` frontend to the new `ruby` frontend. * Fixed parser issues with HashLiterals * Fixed parser issues with one-liner class definitions * Fixed parser issues with arguments in functions
Brings in latest babel and typescript to astgen.
* [ruby] Arrow Lambda Parameter Fix Tests various lambdas and fixes `self` parameter name in methods.
partly to minify the flatgraph diff
* minor work on reachingDef * fmt --------- Co-authored-by: Michael Pollmeier <[email protected]>
This astgen version skipps giant, unparsable files with EMSCRIPTEN code now by default. For: https://shiftleftinc.atlassian.net/browse/SEN-2797
Also handle fullnames with generics correctly when stubbing types For: https://shiftleftinc.atlassian.net/browse/SEN-2840
This PR adds a few more parser tests based on tests in the `querying/` folder for Ruby.
Inheritance via `<` in Ruby can be arbitrary extensions which warrant post-processing analysis, so this removes any attempt to resolve the type at AST creation to allow for a post-processing pass to handle this instead.
Does not exist anymore.
As the `<body>` call is synthetic and meant to be immediately deterministic, so there is no reason it should be re-determined.
* upgrade cpg and adapt * use released cpg
…joernio#4729) ``` [warn] -- [E092] Pattern Match Unchecked Warning: /home/mp/Projects/shiftleft/joern.1/joern-cli/frontends/pysrc2cpg/src/test/scala/io/joern/pysrc2cpg/PySrc2CpgFixture.scala:74:61 [warn] 74 | path.resultPairs().collect { case (firstElement: String, secondElement: Option[Integer]) => [warn] | ^ [warn] |the type test for Option[Integer] cannot be checked at runtime because its type arguments can't be determined from Option[Int] [warn] | [warn] | longer explanation available when compiling with `-explain` } ```
Earlier, if we pass the directory path which contains multiple go modules. Processing was done with all the `.go` files mapped to single `go.mod` file. With this change, we have segregated the processing by first isolating all the files mapped to respective `go.mod`. This will also make sure to cleanup the memory footprint after every module is being processed. However, this will increase the processing when used with download dependency as it will process all the `go.mod` files for identifying and processing used dependencies.
This PR changes the modelling of Singleton methods on objects. Ex:
```ruby
class Animal; end
animal = Animal.new
class << animal
def bark
"woof"
end
end
```
Creates a lambda method for `animal.bark`, and assigns `animal.bark = methodRef(bark)`
Resolves joernio#4721
- we had one stackoverflow in fullname - evaluation.getOverload may return null
…oernio#4733) * Fix always-crashing array initializer type resolution * Fix exception when trying to cast ArrayType to ClassOrInterfaceType * Fix formatting
…g merge (joernio#4737) * minify the flatgraph diff by bringing various things in before the big merge * fmt * revert accidental change
For new frontends with potentially malformed ASTs, type recovery must safely handle instances where members don't have AST parents.
The parser emits calls to `super` as different from simple calls, this PR handles them.
Also: no more empty method fullnames
* Handles implicit returns of multi-assignments by returning an array of the LHS assignment targets. * For implicit returns of multi-assignments created as desugaring of splatted parameters, returns `nil` as per what is evaluated in the Ruby interpreter.
…oernio#4900) as discussed in joernio#4896, we currently fail hard when the user passes additional frontend arguments, which we forward to XTypeRecoveryConfig.parse With this case we still print a warning, but at least don't error. Potentially better would be to filter out everything _but_ the expected arguments in this case... simple test case: foo.sc: ``` @main def main() = { importCode.python("tests/code/pythonsrc/simple.py", args=List("--exclude-regex", "test*")) } ``` ``` sbt stage ./joern --script foo.sc ```
Added handling for named arguments when using hash rocket syntax with "symbol" keys.
* All `require`-like call names are now a set under `ImportsPass` * These calls are now static calls with only the arguments as children
* [ruby] Fixed parser rules for lambda expressions to allow parameters without parentheses * [ruby] update parser rule
…oernio#4907) * [ruby] Added parser rules and AST lowering for beginless and endless range expressions * [ruby] changed ordering in indexing argument list
such loops are bugs that should be fixed. But they happen occasionally, and right now they lead to OOM and infinite loops. I'd prefer it if we handled that a bit more gracefully. The cost of the set in `.dedup` should be pretty low.
joernio#4908) * [ruby] add methodInvocationWithoutParentheses to RETURN argumentList * [ruby] Added check for arguments of jsonArg
Only static calls with import related names 'require', 'load" and so on are linked to import nodes. Before that we ended up with import nodes and links for calls like `Marshal.load(...)`.
Fixed an instance where something like `foo.bar` would represent `foo` as a `self.foo` field access instead of a `foo()` call, given that no `foo` variable is present.
* update cpg/flatgraph * empty commit to trigger GH actions
If an annotation type full name cannot be resolved we now use `<unresolvedNamespace>.ClassName` instead of `ANY`.
In case of a field access after a map/array access, the following field access had invalid argument indicies because it triggered the code path for dynamic receiver calls. This is now fixed.
* Fixed issue where access modifiers in expression positions would be unhandled * Handling identifiers on LHS of associations as symbols
…nio#4920) * Refactor `Semantics` -> `FullNameSemantics` * Refactor `FlowSemantic`/`FlowNode`/`FlowPath` out of the FullNameSemantics parser * Rename `Parser` -> `FullNameSemanticsParser` * Introduce `Semantics` trait with `initialize` and `forMethod`
* [kotlin2cpg] Fix "nacked" call representation. Fix representation of "nacked" calls like `someFunc(someArg)`. These so far have always being represented as statically dispatched which is obviously not correct. * Handle static dispatch with instance argument case.
as brought up in joernio#4936 it can lead to confusing situations if type parameters are shadowed, so for the sake of readability and disambiguity alone we should enable this compiler warning IMO. That being said, I'd like to stress that it's not something fundamentally complicated, afaics. Simple example copied from joernio#4936: ```scala class Example[NodeType <: Object](node: NodeType) extends AnyVal{ def foo[NodeType](x: List[NodeType]): List[NodeType] = x } ``` These two type parameters in your Example class are not related at all. They happen to have the same name, which only means that within `def foo` we cannot reference the `NodeType` type from the `class Example`, but apart from that there's no connection at all between them. Semantically it's similar to something like this on the value level: the two `bar` variables are not related at all, they just happen to have the same name, and therefor we cannot reference the class-level `bar` variable from within `def baz`. ```scala class Foo { val bar = 42 def baz: Unit = { val bar = "123" () } } ```
* Added ANTLR profiling with the `--antlrProfile` frontend argument. * If enabled, will gather parser metrics and log them right next to the file that was parsed, as well as print any syntax ambiguities.
Removes the old frontend and related CLI arguments. Hard deadline 20 Sept 2024.
Profiled `ArrayTests` to detect ambiguity and decisions with high lookaheads and modified test fixture to print profiler logs if enabled. This led to converting certain array rules to use more specific rules and fall back to more general rules less often. Some small improvements on `railsgoat` measured with `time` command on `joern-parse`: ``` // With ambiguity 75.58s user 1.98s system 356% cpu 21.762 total 73.56s user 2.61s system 492% cpu 15.452 total 66.52s user 2.01s system 387% cpu 17.667 total // With reduced ambiguity 65.42s user 1.94s system 443% cpu 15.189 total 74.58s user 2.01s system 557% cpu 13.744 total 74.39s user 1.75s system 560% cpu 13.595 total ```
…and detect project and configuration names automatically (joernio#4934) Fix android project detection and automatic name detection for gradle dependency fetcher
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Created by Github action