Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dummy PR #60

Draft
wants to merge 92 commits into
base: master
Choose a base branch
from
Draft

Dummy PR #60

wants to merge 92 commits into from

Conversation

pandurangpatil
Copy link
Member

No description provided.

max-leuthaeuser and others added 30 commits May 29, 2024 14:54
…nio#4612)

Just some fixed MAX_BATCH_SIZE until we come up with something better.

Fixes: joernio#4611
…ariables (joernio#4617)

* Add reproducing test

* Add tests for new lambdas

* Never add Unknown node for captured variable

* Add todos to comments about lambda parameter capturing
* [python] prevent overtainting dictLiterals

* use allAssignmentTypes in isAssignment

* use methodFullNameExact instead of nameExact in isAssignment
As pointed out in joernio#4240, combining this nested immutable map-like structure has a quadratic performance, and the more performant strategy would be to use nested data-structures to merge.

For now, I've decided not to opt for a builder pattern, but rather keep the underlying structure mutable, and accessor methods return immutable structures.
)

This PR does two things:
1) DEBUG logs the frontend name, version and raw arguments passed down
2) INFO logs the CPG output path if not set explicitly by the user
Added a recursive aspect to the call type recovery if the receiver of the call is possibly recoverable (and it's receiver if that's a call and so on)
- log cpg creation for each file in DEBUG only instead of INFO
 - log the astgen path only once with INFO
 - removed logging that astgen runs now in the given input folder
- Fix last error token position if error token itself is semicolon,
  newline or EOF.
- Also log the parser exception message.
Moved parser logging of memory clearing from INFO to DEBUG
A common pattern in JavaScript is something like

```javascript
var foo = {};
foo.bar = {};
foo.bar.someFunc = function someFunc() {};

foo.bar.someFunc();
```

This PR adds a post-processing pass to find instances where the definition and the call live in the same file and to link them. This "common file" limitation aims to reduce false linking.
also to minify the diff for the flatgraph port
Allow the frontend to continue analysis if the type stubs for built-ins is missing.
The `new` operator can prefix an ordinary call, e.g.
```javascript
var Print = function(str) {
	console.log(str)
}

new Print("Hello")
```
The type recovery pass only considers object instantiations from type declarations on `new` calls, but this change allows it to consider functions too.

Other changes include adding constants for `:program` and `<operator>.new`
Added implicit self receiver to simple calls.
In ruby as for all other dynamic language there is no vtable and thus
only a single entry in the binding tables with name=="" and
signature=="".
This change moves the type/method identifier references for entities exportable from the script to prefix the respective entity at the definition.
this caused me some headaches as I had an old jar lying around there,
likely from a bisect session...
This change moves the type/method identifier references for entities exportable from the script to prefix the respective entity at the definition.
ICPPASTExpression.getEvaluation may fail throwing an unrecoverable exception in case of unresolved includes etc.
* Added `Member` nodes for each method, to relate to their respective bound `TypeDecl` nodes.
* Added a type decl for "fake methods" that include the methods and classes exported by the file
To simplify the call graph, methods resolved to the `Kernel` class are dispatched as static.

Additionally, replaced `__builtin` with `Kernel` to map more closely to Ruby's internals and implemented `MemberAccessCommandContext`.
pandurangpatil and others added 30 commits June 26, 2024 20:06
joern-cli/frontends/gosrc2cpg/src/main/scala/io/joern/gosrc2cpg/GoSrc2Cpg.scala
joern-cli/frontends/gosrc2cpg/src/main/scala/io/joern/gosrc2cpg/passes/BasePassForAstProcessing.scala
…rnio#4698)

* Builtins are renamed from `__builtin` -> `__core` to more closely resemble the core gem in Ruby
* Removed surrounding `<` and `>` tags from builtin packages
* Handling `Array:[]` type calls as `Array.[]` calls
…4693)

* use scala primitives (e.g. Integer -> Int) and some cleanup

motivation: minify diff for flatgraph migration

* latest cpg

* latest cpg

* upgrade cpg

* released cpg
…o#4699)

* Cpg.newDiffGraphBuilder: use new api to minify flatgraph diff

* import
Given the code block below:
```ruby
def foo &block
 puts block.call
end

foo do
 "world!"
end

Proc parameters from do-blocks (interpreted with the usual CPG lambda handling), suggests that there is a `call` member on the type bound to the type decl.

This PR adds this member, with the dynamic type hint referring to the type decl bound to this method.

```
Resolves joernio#4700
…nio#4703)

Earlier changes to include packages with include regex, didn't filter all the packages and had a limitation to include all sub folders/packages if root package is being used. However, that wasn't the case. Hence, we introduced a mechanism to check the entire package folder path with `goastgen` with a separate flag `-includePacakges` and updated integration.
* upgrade deps

* upgrade cpg

* upgrade and adapt
They may fail throwing an unrecoverable exception in case of unresolved includes etc.
Also, some minor clean-up.
This PR handles:
 * Move parser tests from the `deprecated` frontend to the new `ruby` frontend.
 * Fixed parser issues with HashLiterals
 * Fixed parser issues with one-liner class definitions
 * Fixed parser issues with arguments in functions
Brings in latest babel and typescript to astgen.
* [ruby] Arrow Lambda Parameter Fix
Tests various lambdas and fixes `self` parameter name in methods.
partly to minify the flatgraph diff
* minor work on reachingDef

* fmt

---------

Co-authored-by: Michael Pollmeier <[email protected]>
This astgen version skipps giant, unparsable files with EMSCRIPTEN code now by default.

For: https://shiftleftinc.atlassian.net/browse/SEN-2797
Earlier, if we pass the directory path which contains multiple go
modules. Processing was done with all the `.go` files mapped to single
`go.mod` file.

With this change, we have segregated the processing by first isloating
all the files mapped to respective `go.mod`. This will also make sure to
cleanup the memory footprint after every module is being processed.
However this will increase the processing when used with download
dependency as it will process all the `go.mod` files for identifying and
processing used dpendencies.
joern-cli/frontends/gosrc2cpg/src/main/scala/io/joern/gosrc2cpg/GoSrc2Cpg.scala
joern-cli/frontends/gosrc2cpg/src/main/scala/io/joern/gosrc2cpg/passes/DownloadDependenciesPass.scala
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.