-
Notifications
You must be signed in to change notification settings - Fork 42
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #47 from mattyjones/feature/full_integration
massive uplift from gitlab and salesforce
- Loading branch information
Showing
60 changed files
with
4,773 additions
and
2,459 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,98 +1,111 @@ | ||
| <p align="center"> | ||
| <img src="https://github.com/michenriksen/gitrob/raw/master/static/images/gopher_full.png" alt="Gitrob" width="200" /> | ||
| <img src="./static/images/gopher_full.png" alt="Gitrob" width="200" /> | ||
| </p> | ||
| <br /> | ||
| <br /> | ||
| <br /> | ||
|
|
||
| # Gitrob: Putting the Open Source in OSINT | ||
|  | ||
|
|
||
|  | ||
| Gitrob is a tool to help find potentially sensitive information pushed to repositories on GitLab or Github. Gitrob will clone repositories belonging to a user or group/organization down to a configurable depth and iterate through the commit history and flag files and/or commit content that match signatures for potentially sensitive information. The findings will be presented through a web interface for easy browsing and analysis. | ||
|
|
||
| [](https://travis-ci.org/mattyjones/gitrob) | ||
| ## Features | ||
|
|
||
| Gitrob is a tool to help find potentially sensitive files pushed to public repositories on Github. Gitrob will clone repositories belonging to a user or organization down to a configurable depth and iterate through the commit history and flag files that match signatures for potentially sensitive files. The findings will be presented through a web interface for easy browsing and analysis. | ||
| - Scan the following sources: | ||
| - Gitlab repositories | ||
| - Github.com repositories | ||
| - Exclude files, paths, and extensions | ||
| - Web interface for real-time results | ||
| - Configurable commit depth | ||
| - Use environment variables, a config file, or flags | ||
| - Uses sub-commands for easier, more modular, functionality | ||
| - Clone a repo to memory instead of disk | ||
|
|
||
| This currently in beta, check the [roadmap][1] for planned functionality | ||
|
|
||
| ## Usage | ||
|
|
||
| gitrob [options] target [target2] ... [targetN] | ||
| For a full list of use cases and configuration options use the included help functionality. | ||
|
|
||
| ### Options | ||
| `gitrob --help` | ||
|
|
||
| ``` | ||
| -bind-address string | ||
| Address to bind web server to (default "127.0.0.1") | ||
| -commit-depth int | ||
| Number of repository commits to process (default 500) | ||
| -debug | ||
| Print debugging information | ||
| -enterprise-upload-url string | ||
| Upload URL for Github Enterprise (defaults to the URL set in -enterprise-url if any) | ||
| -enterprise-url string | ||
| URL for Github Enterprise | ||
| -enterprise-user string | ||
| Username for Github Enterprise (defaults to first target) | ||
| -github-access-token string | ||
| GitHub access token to use for API requests | ||
| -include-forks | ||
| Include forked repositories in scan | ||
| -load string | ||
| Load session file | ||
| -no-expand-orgs | ||
| Don't add members to targets when processing organizations | ||
| -no-server | ||
| Disables web server | ||
| -port int | ||
| Port to run web server on (default 9393) | ||
| -save string | ||
| Save session to file | ||
| -silent | ||
| Suppress all output except for errors | ||
| -threads int | ||
| Number of concurrent threads (default number of logical CPUs) | ||
| -gather-all | ||
| Specify whether to pull all repositories from the domain | ||
|
|
||
| ## Configuration | ||
|
|
||
| **IMPORTANT** If you are targeting a GitLab group, please give the **group ID** as the target argument. You can find the group ID just below the group name in the GitLab UI. Otherwise, names with suffice for the target arguments. This id can be found on the group homepage. | ||
|
|
||
| There are multiple was to configure the tool for a scan. The easiest way is via commandline flags. To get a full list of available flags and their purpose use `gitrob <subcommand> --help`. This will pring out a list of flags and how they interact with the base scan. You can also set all flags as environment variables or use a static config file in YAML format. This config file can be used to store targets for multiple scan targets. | ||
|
|
||
| The order of precendence with each item taking precedence over the item below it is: | ||
|
|
||
| - explicit call to Set | ||
| - commandline flag | ||
| - environment variable | ||
| - configuration file | ||
| - key/value store | ||
| - default value | ||
|
|
||
| The various values are configured independently of each other so if you set all values in a config file, you can then override just the ones you want on the commandline. A sample config file looks like: | ||
|
|
||
| ```yaml | ||
| --- | ||
| commit-depth: 0 | ||
| gitlab-targets: | ||
| - mattyjones1 | ||
| - 8692959 | ||
| silent: false | ||
| debug: false | ||
| gitlab-api-token: <token> | ||
| github-api-token: <token> | ||
| github-targets: | ||
| - mattyjones | ||
| - ansible | ||
| ignore-path: cmd/, docs/ | ||
| ignore-extension: .go,.log | ||
| in-mem-clone: true | ||
| ``` | ||
| ### Saving session to a file | ||
| ## Examples | ||
| By default, gitrob will store its state for an assessment in memory. This means that the results of an assessment is lost when Gitrob is closed. You can save the session to a file by using the `-save` option: | ||
| Scan a GitLab group assuming your access token has been added to the environment variable or a config file. Look for file signature matches only: | ||
| gitrob -save ~/gitrob-session.json acmecorp | ||
| gitrob scanGitlab <gitlab_group_id> | ||
| Gitrob will save all the gathered information to the specified file path as a special JSON document. The file can be loaded again for browsing at another point in time, shared with other analysts or parsed for custom integrations with other tools and systems. | ||
| Scan a multiple GitLab groups assuming your access token has been added to the environment variable or a config file. Clone repositories into memory for faster analysis. Set the scan mode to 2 to scan each file match for a content match before creating a result.: | ||
| ### Loading session from a file | ||
| gitrob scanGitlab -in-mem-clone -mode 2 "<gitlab_group_id_1> <gitlab_group_id_2>" | ||
| A session stored in a file can be loaded with the `-load` option: | ||
| Scan a GitLab groups assuming your access token has been added to the environment variable or a config file. Clone repositories into memory for faster analysis. Set the scan mode to 3 to scan each commit for content matches only.: | ||
| gitrob -load ~/gitrob-session.json | ||
| gitrob scanGitlab -in-mem-clone -mode 3 "<gitlab_group_id>" | ||
| Gitrob will start its web interface and serve the results for analysis. | ||
| Scan a Github user setting your Github access token as a parameter. Clone repositories into memory for faster analysis. | ||
| gitrob scangithub -github-access-token <token> -in-mem-clone "<github_user_name>" | ||
| ### Use with Github Enterprise | ||
| ### Editing File and Content Regular Expressions | ||
| To configure Gitrob for Github Enterprise, the following switches can be used: | ||
| Regular expressions are included in the [filesignatures.json](./rules/filesignatures.json) and [contentsignatures.json](./rules/contentsignatures.json) files respectively. Edit these files to adjust your scope and fine-tune your results. | ||
| - `enterprise-url`: Must be specified; this is the URL where the path `/api/v3/` exists. This is usually the URL where the Github web interface can be found. Example: `-enterprise-url=https://github.yourcompany.com` | ||
| - `enterprise-upload-url:` Optional, defaults to `enterprise-url`; full path to the upload URL if different from the main Github Enterprise URL. Example: `-enterprise-upload-url=https://github.yourcompany.com/api/v3/upload` | ||
| - `enterprise-user`: Optional, defaults to the first target. Example: `-enterprise-user=your.username` | ||
| Gitrob will start its web interface and serve the results for analysis. | ||
| ## Installation | ||
| A [precompiled version is available](https://github.com/michenriksen/gitrob/releases) for each release, alternatively you can use the latest version of the source code from this repository in order to build your own binary. | ||
| At this stage the only option is to build from source from this repository. | ||
| Make sure you have a correctly configured **Go >= 1.11** environment and that `$GOPATH/bin` is in your `$PATH` | ||
| To install from source, make sure you have a correctly configured **Go >= 1.14** environment and that `$GOPATH/bin` is in your `$PATH`. | ||
|
|
||
| $ go get github.com/michenriksen/gitrob | ||
| $ git clone [email protected]:mattyjones1/gitrob.git | ||
| $ cd ~/go/src/gitrob | ||
| $ make build | ||
| $ ./bin/gitrob-<ARCH> <sub-command> | ||
|
|
||
| In the future there will be binary releases of the code | ||
|
|
||
| This command will download gitrob, install its dependencies, compile it and move the `gitrob` executable to `$GOPATH/bin`. | ||
| ## Access Tokens | ||
|
|
||
| ### Github access token | ||
| Gitrob will need either a GitLab or Github access token in order to interact with the appropriate API. You can create a [GitLab personal access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html), or [a Github personal access token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) and save it in an environment variable in your `.bashrc` or similar shell configuration file: | ||
|
|
||
| Gitrob will need a Github access token in order to interact with the Github API. [Create a personal access token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) and save it in an environment variable in your `.bashrc` or similar shell configuration file: | ||
| export GITROB_GITLAB_ACCESS_TOKEN=deadbeefdeadbeefdeadbeefdeadbeefdeadbeef | ||
| export GITROB_GITHUB_ACCESS_TOKEN=deadbeefdeadbeefdeadbeefdeadbeefdeadbeef | ||
|
|
||
| export GITROB_ACCESS_TOKEN=deadbeefdeadbeefdeadbeefdeadbeefdeadbeef | ||
| Alternatively you can specify the access token with the `-gitlab-access-token` or `-github-access-token` option on the command line, but watch out for your command history! A configuration file can also be used, an example is provided above. | ||
|
|
||
| Alternatively you can specify the access token with the `-github-access-token` option, but watch out for your command history! | ||
| [1]: docs/development/roadmap.md |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| If the issues is non-critical then open an issue on Github with | ||
| * a full explanation of the issue | ||
| * what versions it applies to | ||
| * any third-party libraries it may effect | ||
|
|
||
| If the matter is critical send any details to urlugalATgmail using the following key if necessary | ||
|
|
||
| -----BEGIN PGP PUBLIC KEY BLOCK----- | ||
|
|
||
| mQENBF8fi7oBCADl862/KBPZFRb1A4t8b38BDY7juQ4Xj/gEgPSQ2GcqOvnBa+mX | ||
| tCsKimanAGNiS8c2TtSDCejgggBWqKF7GgR1XP5a8a60PDWpsXZn2duzABeEOYhc | ||
| qWHmmJTTgi3K7+gd+VSDK3rfXmWf6oSQIE/q03/fQJGphVTaQQfdXvfp+m3rwl9U | ||
| A6EtIaNLyIn7LC6oBXGKGZnFx4MeZCfrPOF5GHvMq0bt0LQ4N46hkMK0oo+9eFb5 | ||
| LEvvOWbDQthpUBA12a/3Sw+XwlbowqnT5vxiCMDsehrEgENeZZ8zS8tY+YBtrt+R | ||
| J+DZg4EtxS4cGDYlopntlXXMvmbCfYGBaOMXABEBAAG0Hk1hdHQgSm9uZXMgPHVy | ||
| bHVnYWxAZ21haWwuY29tPokBTgQTAQgAOBYhBDJRM9sNhtVxJYvvU+2gu2mdq3WU | ||
| BQJfH4u6AhsDBQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAAAoJEO2gu2mdq3WUsHAH | ||
| /1BvzYOkOUdd3WIN9XaUlOctqzhGn4r6Wcj+K6zpZBrbnf6lemNvK4czdBhyl7KK | ||
| YWNR5Lv3h2CHzxZPswcud+BjrAlDq77ZGjp3lqWyysJXKit5bEMxWqKMU6JSpD3j | ||
| vO4j3oDard9qTcbApG87iGGQalHjk/IqeicChp/WwGb0GFm7brZDMlRO8JZ5yWHH | ||
| KpMP8IAO/wQhZ9VFn1Zw02JVViHrBB83Cwbf1c3RFb61gs5V1Z+9t0v9Hto8IChD | ||
| 2xsBt2nqnpqOPagSwSdkYO8oPTBQUK632URzH57ZhNJhWhDTcrWlNK/uh5fwTyRA | ||
| AjM94uyesiFAtmHLq7sM6GW5AQ0EXx+LugEIALoSoKrx+VJFWPSCyGpOS7oRqkF1 | ||
| CbanZYOaUWAXlyPI8NdoA+1kiEQyuz2sz0Q4Vpiuc0KkH3rK8fOT5s6KwYIC0o+L | ||
| +EszoBckS5OMNOUrYtoJdp+TkDCVvOflsqdp8BXqDRzDDxQcK51qxRWrWEvQ7eau | ||
| KTsXd1wWgFssuujd3kpqHD7fl6LNaPSYOt+vyV/pBSay+HhTjdOTV2G2c6lQEgm/ | ||
| TtUFse95SyWnNdmOpiN8pLUlm+yKiO34ks2ODJ9HRmI7DoofE8yskRavWYTn94Gw | ||
| ZusltWCV0onCj8l0WJkpahZY7E/3rxXJEhreWNre45sFyxm1E4Y8VJmvG+EAEQEA | ||
| AYkBNgQYAQgAIBYhBDJRM9sNhtVxJYvvU+2gu2mdq3WUBQJfH4u6AhsMAAoJEO2g | ||
| u2mdq3WUvZsIAN5qFoCBH1SbITMfvzDRdKj7WOOKGj7PFz9UfTK0kyoI34iktX6B | ||
| j5g6Yi4NtbnuWPG1GblR3nrHQcOtS8mw3D63AsgOMTsaCcro0K7wy3bTEbaFeYFw | ||
| EuXJ+/FtjWAR/EhrMp7CunDS2l56oiu4lbP6vAroHQurgEiIdm5Sma5IEraGJOFo | ||
| xL+aGT5m3QSkn+5xz4w+KrCx+QkYnV8OafSCwGU8ubvqFbnKSZavrUt4a/MB5FiA | ||
| 9LzdQW9eTLF4BJBFZrNaZ77aNGQS4k9bzFTCJXoXTY4UrLXs6MsYanEgMnU8YFIm | ||
| pQXV3D9FRG+vp2yFdhvOfOp5qFwwKxaNP4U= | ||
| =bQRW | ||
| -----END PGP PUBLIC KEY BLOCK----- |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| // Package cmd represents the specific commands that the user will execute. Only specific code related to the command | ||
| // should be in these files. As much of the code as possible should be pushed to other packages. | ||
| package cmd | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "github.com/spf13/cobra" | ||
| "gitrob/version" | ||
| "os" | ||
| ) | ||
|
|
||
| // rootCmd represents the base command when called without any subcommands | ||
| var rootCmd = &cobra.Command{ | ||
| Use: "gitrob", | ||
| Short: "A tool to scan for secrets in various digital hiding spots", | ||
| Long: "A tool to scan for secrets in various digital hiding spots - v" + version.AppVersion(), // TODO write a better long description | ||
| } | ||
|
|
||
| // Execute adds all child commands to the root command and sets flags appropriately. | ||
| // This is called by main.main(). It only needs to happen once to the rootCmd. | ||
| func Execute() { | ||
| if err := rootCmd.Execute(); err != nil { | ||
| fmt.Println(err) | ||
| os.Exit(1) | ||
| } | ||
| } | ||
|
|
||
| func init() {} |
Oops, something went wrong.