diff --git a/components/resc-vcs-scanner/README.md b/components/resc-vcs-scanner/README.md index 33f9b175..964b29c2 100644 --- a/components/resc-vcs-scanner/README.md +++ b/components/resc-vcs-scanner/README.md @@ -36,7 +36,7 @@ These instructions will help you to get a copy of the project up and running on **Prerequisites:** * RabbitMQ and RESC web service must be up and running locally.</br> If you have already deployed RESC through helm in Kubernetes, then rabbitmq and resc webservice are already running for you.</br> - * Install Gitleaks [v8.8.8](https://github.com/zricethezav/gitleaks/releases/tag/v8.8.8) on your system. + * Install Gitleaks [v8.18.0](https://github.com/zricethezav/gitleaks/releases/tag/v8.18.0) on your system. * Download the rule config toml file to `/tmp/temp_resc_rule.toml` location by running below command from a Git Bash terminal. * Send some repositories to 'repositories' topics of RabbitMQ server by referring the README of RESC-VCS-SCRAPER component. diff --git a/components/resc-vcs-scanner/gitleaks_config/LICENSE b/components/resc-vcs-scanner/gitleaks_config/LICENSE new file mode 100644 index 00000000..3c270b36 --- /dev/null +++ b/components/resc-vcs-scanner/gitleaks_config/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2019 Zachary Rice + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/components/resc-vcs-scanner/gitleaks_config/README.md b/components/resc-vcs-scanner/gitleaks_config/README.md new file mode 100644 index 00000000..93eb29e7 --- /dev/null +++ b/components/resc-vcs-scanner/gitleaks_config/README.md @@ -0,0 +1,418 @@ +# Gitleaks + +``` +┌─○───┐ +│ │╲ │ +│ │ ○ │ +│ ○ ░ │ +└─░───┘ +``` + +<p align="left"> + <p align="left"> + <a href="https://github.com/zricethezav/gitleaks/actions/workflows/test.yml"> + <img alt="Github Test" src="https://github.com/zricethezav/gitleaks/actions/workflows/test.yml/badge.svg"> + </a> + <a href="https://hub.docker.com/r/zricethezav/gitleaks"> + <img src="https://img.shields.io/docker/pulls/zricethezav/gitleaks.svg" /> + </a> + <a href="https://github.com/zricethezav/gitleaks-action"> + <img alt="gitleaks badge" src="https://img.shields.io/badge/protected%20by-gitleaks-blue"> + </a> + <a href="https://twitter.com/intent/follow?screen_name=zricethezav"> + <img src="https://img.shields.io/twitter/follow/zricethezav?label=Follow%20zricethezav&style=social&color=blue" alt="Follow @zricethezav" /> + </a> + </p> +</p> + +### Join our Discord! [](https://discord.gg/sydS6AHTUP) + +Gitleaks is a SAST tool for **detecting** and **preventing** hardcoded secrets like passwords, api keys, and tokens in git repos. Gitleaks is an **easy-to-use, all-in-one solution** for detecting secrets, past or present, in your code. + +``` +➜ ~/code(master) gitleaks detect --source . -v + + ○ + │╲ + │ ○ + ○ ░ + ░ gitleaks + + +Finding: "export BUNDLE_ENTERPRISE__CONTRIBSYS__COM=cafebabe:deadbeef", +Secret: cafebabe:deadbeef +RuleID: sidekiq-secret +Entropy: 2.609850 +File: cmd/generate/config/rules/sidekiq.go +Line: 23 +Commit: cd5226711335c68be1e720b318b7bc3135a30eb2 +Author: John +Email: john@users.noreply.github.com +Date: 2022-08-03T12:31:40Z +Fingerprint: cd5226711335c68be1e720b318b7bc3135a30eb2:cmd/generate/config/rules/sidekiq.go:sidekiq-secret:23 +``` + +## Getting Started + +Gitleaks can be installed using Homebrew, Docker, or Go. Gitleaks is also available in binary form for many popular platforms and OS types on the [releases page](https://github.com/zricethezav/gitleaks/releases). In addition, Gitleaks can be implemented as a pre-commit hook directly in your repo or as a GitHub action using [Gitleaks-Action](https://github.com/gitleaks/gitleaks-action). + +### Installing + +```bash +# MacOS +brew install gitleaks + +# Docker (DockerHub) +docker pull zricethezav/gitleaks:latest +docker run -v ${path_to_host_folder_to_scan}:/path zricethezav/gitleaks:latest [COMMAND] --source="/path" [OPTIONS] + +# Docker (ghcr.io) +docker pull ghcr.io/gitleaks/gitleaks:latest +docker run -v ${path_to_host_folder_to_scan}:/path ghcr.io/gitleaks/gitleaks:latest [COMMAND] --source="/path" [OPTIONS] + +# From Source +git clone https://github.com/gitleaks/gitleaks.git +cd gitleaks +make build +``` + +### GitHub Action + +Check out the official [Gitleaks GitHub Action](https://github.com/gitleaks/gitleaks-action) + +``` +name: gitleaks +on: [pull_request, push, workflow_dispatch] +jobs: + scan: + name: gitleaks + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + with: + fetch-depth: 0 + - uses: gitleaks/gitleaks-action@v2 + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GITLEAKS_LICENSE: ${{ secrets.GITLEAKS_LICENSE}} # Only required for Organizations, not personal accounts. +``` + +### Pre-Commit + +1. Install pre-commit from https://pre-commit.com/#install +2. Create a `.pre-commit-config.yaml` file at the root of your repository with the following content: + + ``` + repos: + - repo: https://github.com/gitleaks/gitleaks + rev: v8.16.1 + hooks: + - id: gitleaks + ``` + + for a [native execution of GitLeaks](https://github.com/zricethezav/gitleaks/releases) or use the [`gitleaks-docker` pre-commit ID](https://github.com/zricethezav/gitleaks/blob/master/.pre-commit-hooks.yaml) for executing GitLeaks using the [official Docker images](#docker) + +3. Auto-update the config to the latest repos' versions by executing `pre-commit autoupdate` +4. Install with `pre-commit install` +5. Now you're all set! + +``` +➜ git commit -m "this commit contains a secret" +Detect hardcoded secrets.................................................Failed +``` + +Note: to disable the gitleaks pre-commit hook you can prepend `SKIP=gitleaks` to the commit command +and it will skip running gitleaks + +``` +➜ SKIP=gitleaks git commit -m "skip gitleaks check" +Detect hardcoded secrets................................................Skipped +``` + +## Usage + +``` +Usage: + gitleaks [command] + +Available Commands: + completion generate the autocompletion script for the specified shell + detect detect secrets in code + help Help about any command + protect protect secrets in code + version display gitleaks version + +Flags: + -b, --baseline-path string path to baseline with issues that can be ignored + -c, --config string config file path + order of precedence: + 1. --config/-c + 2. env var GITLEAKS_CONFIG + 3. (--source/-s)/.gitleaks.toml + If none of the three options are used, then gitleaks will use the default config + --exit-code int exit code when leaks have been encountered (default 1) + -h, --help help for gitleaks + -l, --log-level string log level (trace, debug, info, warn, error, fatal) (default "info") + --max-target-megabytes int files larger than this will be skipped + --no-color turn off color for verbose output + --no-banner suppress banner + --redact redact secrets from logs and stdout + -f, --report-format string output format (json, csv, junit, sarif) (default "json") + -r, --report-path string report file + -s, --source string path to source (default ".") + -v, --verbose show verbose output from scan + +Use "gitleaks [command] --help" for more information about a command. +``` + +### Commands + +There are two commands you will use to detect secrets; `detect` and `protect`. + +#### Detect + +The `detect` command is used to scan repos, directories, and files. This command can be used on developer machines and in CI environments. + +When running `detect` on a git repository, gitleaks will parse the output of a `git log -p` command (you can see how this executed +[here](https://github.com/zricethezav/gitleaks/blob/7240e16769b92d2a1b137c17d6bf9d55a8562899/git/git.go#L17-L25)). +[`git log -p` generates patches](https://git-scm.com/docs/git-log#_generating_patch_text_with_p) which gitleaks will use to detect secrets. +You can configure what commits `git log` will range over by using the `--log-opts` flag. `--log-opts` accepts any option for `git log -p`. +For example, if you wanted to run gitleaks on a range of commits you could use the following command: `gitleaks detect --source . --log-opts="--all commitA..commitB"`. +See the `git log` [documentation](https://git-scm.com/docs/git-log) for more information. + +You can scan files and directories by using the `--no-git` option. + +#### Protect + +The `protect` command is used to scan uncommitted changes in a git repo. This command should be used on developer machines in accordance with +[shifting left on security](https://cloud.google.com/architecture/devops/devops-tech-shifting-left-on-security). +When running `protect` on a git repository, gitleaks will parse the output of a `git diff` command (you can see how this executed +[here](https://github.com/zricethezav/gitleaks/blob/7240e16769b92d2a1b137c17d6bf9d55a8562899/git/git.go#L48-L49)). You can set the +`--staged` flag to check for changes in commits that have been `git add`ed. The `--staged` flag should be used when running Gitleaks +as a pre-commit. + +**NOTE**: the `protect` command can only be used on git repos, running `protect` on files or directories will result in an error message. + +### Creating a baseline + +When scanning large repositories or repositories with a long history, it can be convenient to use a baseline. When using a baseline, +gitleaks will ignore any old findings that are present in the baseline. A baseline can be any gitleaks report. To create a gitleaks report, run gitleaks with the `--report-path` parameter. + +``` +gitleaks detect --report-path gitleaks-report.json # This will save the report in a file called gitleaks-report.json +``` + +Once as baseline is created it can be applied when running the detect command again: + +``` +gitleaks detect --baseline-path gitleaks-report.json --report-path findings.json +``` + +After running the detect command with the --baseline-path parameter, report output (findings.json) will only contain new issues. + +### Verify Findings + +You can verify a finding found by gitleaks using a `git log` command. +Example output: + +``` +Finding: aws_secret="AKIAIMNOJVGFDXXXE4OA" +RuleID: aws-access-token +Secret AKIAIMNOJVGFDXXXE4OA +Entropy: 3.65 +File: checks_test.go +Line: 37 +Commit: ec2fc9d6cb0954fb3b57201cf6133c48d8ca0d29 +Author: Zachary Rice +Email: z@email.com +Date: 2018-01-28T17:39:00Z +Fingerprint: ec2fc9d6cb0954fb3b57201cf6133c48d8ca0d29:checks_test.go:aws-access-token:37 +``` + +We can use the following format to verify the leak: + +``` +git log -L {StartLine,EndLine}:{File} {Commit} +``` + +So in this example it would look like: + +``` +git log -L 37,37:checks_test.go ec2fc9d6cb0954fb3b57201cf6133c48d8ca0d29 +``` + +Which gives us: + +``` +commit ec2fc9d6cb0954fb3b57201cf6133c48d8ca0d29 +Author: zricethezav <thisispublicanyways@gmail.com> +Date: Sun Jan 28 17:39:00 2018 -0500 + + [update] entropy check + +diff --git a/checks_test.go b/checks_test.go +--- a/checks_test.go ++++ b/checks_test.go +@@ -28,0 +37,1 @@ ++ "aws_secret= \"AKIAIMNOJVGFDXXXE4OA\"": true, + +``` + +## Pre-Commit hook + +You can run Gitleaks as a pre-commit hook by copying the example `pre-commit.py` script into +your `.git/hooks/` directory. + +## Configuration + +Gitleaks offers a configuration format you can follow to write your own secret detection rules: + +```toml +# Title for the gitleaks configuration file. +title = "Gitleaks title" + +# Extend the base (this) configuration. When you extend a configuration +# the base rules take precedence over the extended rules. I.e., if there are +# duplicate rules in both the base configuration and the extended configuration +# the base rules will override the extended rules. +# Another thing to know with extending configurations is you can chain together +# multiple configuration files to a depth of 2. Allowlist arrays are appended +# and can contain duplicates. +# useDefault and path can NOT be used at the same time. Choose one. +[extend] +# useDefault will extend the base configuration with the default gitleaks config: +# https://github.com/zricethezav/gitleaks/blob/master/config/gitleaks.toml +useDefault = true +# or you can supply a path to a configuration. Path is relative to where gitleaks +# was invoked, not the location of the base config. +path = "common_config.toml" + +# An array of tables that contain information that define instructions +# on how to detect secrets +[[rules]] + +# Unique identifier for this rule +id = "awesome-rule-1" + +# Short human readable description of the rule. +description = "awesome rule 1" + +# Golang regular expression used to detect secrets. Note Golang's regex engine +# does not support lookaheads. +regex = '''one-go-style-regex-for-this-rule''' + +# Golang regular expression used to match paths. This can be used as a standalone rule or it can be used +# in conjunction with a valid `regex` entry. +path = '''a-file-path-regex''' + +# Array of strings used for metadata and reporting purposes. +tags = ["tag","another tag"] + +# Int used to extract secret from regex match and used as the group that will have +# its entropy checked if `entropy` is set. +secretGroup = 3 + +# Float representing the minimum shannon entropy a regex group must have to be considered a secret. +entropy = 3.5 + +# Keywords are used for pre-regex check filtering. Rules that contain +# keywords will perform a quick string compare check to make sure the +# keyword(s) are in the content being scanned. Ideally these values should +# either be part of the idenitifer or unique strings specific to the rule's regex +# (introduced in v8.6.0) +keywords = [ + "auth", + "password", + "token", +] + +# You can include an allowlist table for a single rule to reduce false positives or ignore commits +# with known/rotated secrets +[rules.allowlist] +description = "ignore commit A" +commits = [ "commit-A", "commit-B"] +paths = [ + '''go\.mod''', + '''go\.sum''' +] +# note: (rule) regexTarget defaults to check the _Secret_ in the finding. +# if regexTarget is not specified then _Secret_ will be used. +# Acceptable values for regexTarget are "match" and "line" +regexTarget = "match" +regexes = [ + '''process''', + '''getenv''', +] +# note: stopwords targets the extracted secret, not the entire regex match +# like 'regexes' does. (stopwords introduced in 8.8.0) +stopwords = [ + '''client''', + '''endpoint''', +] + + +# This is a global allowlist which has a higher order of precedence than rule-specific allowlists. +# If a commit listed in the `commits` field below is encountered then that commit will be skipped and no +# secrets will be detected for said commit. The same logic applies for regexes and paths. +[allowlist] +description = "global allow list" +commits = [ "commit-A", "commit-B", "commit-C"] +paths = [ + '''gitleaks\.toml''', + '''(.*?)(jpg|gif|doc)''' +] + +# note: (global) regexTarget defaults to check the _Secret_ in the finding. +# if regexTarget is not specified then _Secret_ will be used. +# Acceptable values for regexTarget are "match" and "line" +regexTarget = "match" + +regexes = [ + '''219-09-9999''', + '''078-05-1120''', + '''(9[0-9]{2}|666)-\d{2}-\d{4}''', +] +# note: stopwords targets the extracted secret, not the entire regex match +# like 'regexes' does. (stopwords introduced in 8.8.0) +stopwords = [ + '''client''', + '''endpoint''', +] +``` + +Refer to the default [gitleaks config](https://github.com/zricethezav/gitleaks/blob/master/config/gitleaks.toml) for examples or follow the [contributing guidelines](https://github.com/zricethezav/gitleaks/blob/master/README.md) if you would like to contribute to the default configuration. Additionally, you can check out [this gitleaks blog post](https://blog.gitleaks.io/stop-leaking-secrets-configuration-2-3-aeed293b1fbf) which covers advanced configuration setups. + +### Additional Configuration + +#### gitleaks:allow + +If you are knowingly committing a test secret that gitleaks will catch you can add a `gitleaks:allow` comment to that line which will instruct gitleaks +to ignore that secret. Ex: + +``` +class CustomClass: + discord_client_secret = '8dyfuiRyq=vVc3RRr_edRk-fK__JItpZ' #gitleaks:allow + +``` + +#### .gitleaksignore + +You can ignore specific findings by creating a `.gitleaksignore` file at the root of your repo. In release v8.10.0 Gitleaks added a `Fingerprint` value to the Gitleaks report. Each leak, or finding, has a Fingerprint that uniquely identifies a secret. Add this fingerprint to the `.gitleaksignore` file to ignore that specific secret. See Gitleaks' [.gitleaksignore](https://github.com/zricethezav/gitleaks/blob/master/.gitleaksignore) for an example. Note: this feature is experimental and is subject to change in the future. + +## Sponsorships + +<p align="left"> + <a href="https://www.tines.com/?utm_source=oss&utm_medium=sponsorship&utm_campaign=gitleaks"> + <img alt="Tines Sponsorship" src="https://user-images.githubusercontent.com/15034943/146411864-4878f936-b4f7-49a0-b625-f9f40c704bfa.png" width=200> + </a> + </p> + +## Exit Codes + +You can always set the exit code when leaks are encountered with the --exit-code flag. Default exit codes below: + +``` +0 - no leaks present +1 - leaks or error encountered +126 - unknown flag +``` diff --git a/components/resc-vcs-scanner/gitleaks_config/seco-gitleaks-linux-amd64 b/components/resc-vcs-scanner/gitleaks_config/seco-gitleaks-linux-amd64 index e0a5ede1..f9d14995 100644 Binary files a/components/resc-vcs-scanner/gitleaks_config/seco-gitleaks-linux-amd64 and b/components/resc-vcs-scanner/gitleaks_config/seco-gitleaks-linux-amd64 differ