jf pipenv install might prevent Github secrets masking

[sorekz]

Describe the bug

We run the jfrog cli in our Github workflows to setup a pipenv. There we noticed that the jf pipenv install command seems to internally create a shell with a column width of 80. This means that after 80 characters it adds a new line, which is also visible in the console and breaks the secret masking.
It's important to note that this special case only occurs when:

• verbose output is enabled (the jfrog cli does this by default on Github runners and I also couldn't find a way to disable the behaviour)
• for the authentication a username and token as password input is used (or a very long password). Otherwise the 80 characters are not reached.

See the reproduction steps for more

Current behavior

Gitub fails to mask secrets when additional characters like line breaks are added in the log output
The token DdJwCpuAVENmcStnCKpKZswcOmYjElVPhGabiTnCfGJKCZhMDBbnZsryixishQFzIycKSdCDnK is not valid but with the same length as a normal token, to showcase the issue.

https://***:DdJwCpuAVENmcStnCKpKZswcOmYjElVPhGabiTnCfGJKCZhMDBbnZsryixishQFzIycKSd
[email protected]/artifactory/api/pypi/remote-pypi/simple --no-input --upgrade

Reproduction steps

Working workflow given the following setup:

• there is an Artifactory and vars.JF_URL is set (as my url is private I replaced it with "acme.jfrog.io" in the logs)
• there is a pypi mirror with the remote name remote-pypi (name is not relevant for the test)
• secrets.JF_USER and secrets.JF_ACCESS_TOKEN are set

name: "jf pipenv install secret masking"

on:
  workflow_dispatch:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12' 
      - run: pip install pipenv
      - uses: jfrog/setup-jfrog-cli@v4
      
      # Just an example, the actual dependencies are not relevant for the test
      - run: |
          cat > Pipfile << EOF
          [[source]]
          url = "https://pypi.org/simple"
          verify_ssl = true
          name = "pypi"
          
          [dev-packages]
          parse = "*"
          importlib-metadata = "*"
          colorama= {version = "*", sys_platform = "== 'win32'"}
          invoke = "*"
          exceptiongroup = "==1.1.0"
          tomli = "*"
          pyyaml = "==6.0.1"
          build = "*"
          twine = "*"
          
      - run: jf --version # jfrog/setup-jfrog-cli installs version 2.67.0
      - run: jf c add serverId --url {{ vars.JF_URL }} --user ${{ secrets.JF_USER }} --password ${{ secrets.JF_ACCESS_TOKEN }}
      - run: jf pipenv-config --repo-resolve remote-pypi --global
      - run: jf pipc --repo-resolve remote-pypi --global
      - run: jf pipenv lock
      - run: jf pipenv install --dev

Relavant output in the jf pipenv install --dev step:

Run jf pipenv install --dev
##[debug]/usr/bin/bash -e /opt/app-root/src/_work/_temp/ccceb245-6bc3-48bd-b2db-d5f3cdd6f848.sh
08:36:25 [Info] Running pipenv install
08:36:25 [Warn] couldn't extract payload from Access Token.
The provided access token is not a valid JWT, probably a reference token.
Some package managers only support basic authentication which requires also a username.
If you plan to work with one of those package managers, please provide a username.
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
Installing dependencies from Pipfile.lock (0fa9f5)...
Installing dependencies from Pipfile.lock (0fa9f5)...
Writing supplied requirement line to temporary file: "backports.tarfile==1.2.0; 
python_version < '3.12' 
--hash=sha256:77e284d754527b01fb1e6fa8a1afe577858ebe4e9dad89[19](https://---.com/org/repo/actions/runs/4266318/job/9451381#step:9:20)e34c862cb399bc34 
--hash=sha256:d75e02c268746e1b8144c278978b6e98e85de6ad16f8e4b0844a154557eca991"
Writing supplied requirement line to temporary file: "build==1.2.2; 
python_version >= '3.8' 
--hash=sha256:119b2fb462adef986483438377a13b2f4[20](https://---.com/org/repo/actions/runs/4266318/job/9451381#step:9:21)64a2a3a4161f24a0cca698a07ac8c 
--hash=sha256:277ccc71619d98afdd841a0e96ac9fe1593b823af481d3b0cea748e8894e0613"
<< other downloads >>
Preparing Installation of "zipp==3.20.2; python_version >= '3.8' 
--hash=sha256:a817ac80d6cf4b23bf7f2828b7cabf326f15a001bea8b1f9b49631780ba28350 
--hash=sha256:bc9eb26f4506fda01b81bcde0ca78103b6e62f991b381fec825435c836edbc29"
$ /opt/app-root/src/.local/share/virtualenvs/test-daniel-v9v4-Lpl/bin/python 
/opt/app-root/src/.local/lib/python3.12/site-packages/pipenv/patched/pip/__pip-r
unner__.py install -i 
https://***:DdJwCpuAVENmcStnCKpKZswcOmYjElVPhGabiTnCfGJKCZhMDBbnZsryixishQFzIycKSd
[email protected]/artifactory/api/pypi/remote-pypi/simple --no-input --upgrade 
--no-deps -r /tmp/pipenv-t3ypdins-requirements/pipenv-bn1cgqjv-reqs.txt
Using source directory: 
'/opt/app-root/src/.local/share/virtualenvs/test-daniel-v9v4-Lpl/src'
Looking in indexes: 
https://***:***@acme.jfrog.io/artifactory/api/pypi/remote-pypi/simple

As you can see the lines

$ /opt/app-root/src/.local/share/virtualenvs/test-daniel-v9v4-Lpl/bin/python 
/opt/app-root/src/.local/lib/python3.12/site-packages/pipenv/patched/pip/__pip-r
unner__.py install -i 
https://***:DdJwCpuAVENmcStnCKpKZswcOmYjElVPhGabiTnCfGJKCZhMDBbnZsryixishQFzIycKSd
[email protected]/artifactory/api/pypi/remote-pypi/simple --no-input --upgrade 
--no-deps -r /tmp/pipenv-t3ypdins-requirements/pipenv-bn1cgqjv-reqs.txt

contain the secret in plain text because Github fails to mask it as the output contains a line break inbetween.
If we use the token without username it is masked because the line width of 80 is not exceeded.

$ /opt/app-root/src/.local/share/virtualenvs/test-daniel-v9v4-Lpl/bin/python 
/opt/app-root/src/.local/lib/python3.12/site-packages/pipenv/patched/pip/__pip-r
unner__.py install -i 
https://***@acme.
jfrog.io/artifactory/api/pypi/remote-pypi/simple --no-input --upgrade 
--no-deps -r /tmp/pipenv-t3ypdins-requirements/pipenv-bn1cgqjv-reqs.txt

Expected behavior

Expected behaviour is that the output does not add additional line breaks and therefore does not break the Github secret masking.
This probably also happens in other CICD systems and is dangerous as users assume that their secrets are masked.

JFrog CLI version

2.67.0

Operating system type and version

Windows, Linux

JFrog Artifactory version

No response

JFrog Xray version

No response

[RobiNino]

Hi @sorekz ,
JFrog CLI has an internal secrets masking for pipenv, that takes into account possible line breaks.
I couldn't reproduce this issue neither on GitHub actions nor on my machine.

Could you please try running the same commands outside GitHub and let us know if the issue still occurs? This could help identify the source of the issue.

Thanks

[sorekz]

When running locally it does some masking but not in all places. In the code below the Artifactory url, username and token was adjusted to other values of the same length.

Local test on Windows 10 using cmd and the latest jf cli 2.71.0. On Github we use a linux runner and see the same results.
Locally it's important to run jf pipenv install with the --verbose parameter. This is the default when running in CI and as far as I know can not be deactivated there. Of course also a clean pipenv is needed so that the requirements are downloaded.

jf --version # 2.71.0
jf c add --url mydomain1234.jfrog.io --user myuser1 --password kaMjLfnOKuiCYKHOehYnGxF7gP2BOWVaidpdiU7xcEJc8lDipeSbsw5ZqomMzzX0
jf pipenv-config --repo-resolve remote-pypi --global
jf pipc --repo-resolve remote-pypi --global
jf pipenv lock # Same Pipfile as in first comment
jf pipenv install --dev --verbose

<shortened>
--hash=sha256:ff3824dc5261f50c9b0dfb3be22b4567a6f938ccce4587b38952d85fd9e9afe4"
Preparing Installation of "requests==2.32.3; python_version >= '3.8'
--hash=sha256:55365417734eb18255590a9ff9eb97e9e1da868d4ccd6402399eaf68af20a760
--hash=sha256:70761cfe03c773ceb22aa2f671b4757976145175cdfca038c02654d061d6dcc6"
Preparing Installation of "urllib3==1.26.20; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4,
3.5' --hash=sha256:0ed14ccfbf1c30a9072c7ca157e4319b70d65f623e91e7b32fadb2853431016e
--hash=sha256:40c2dc0c681e47eb8f90e7e27bf6ff7df2e677421fd46756da1161c39ca70d32"
$ C:/temp/pipenv/.venv/Scripts/python.exe
'C:\Users\myuser1\AppData\Local\Programs\Python\Python312\Lib\site-packages\pipenv\patched\pip\__pip-runner__.py'
install -i
https://myuser1:kaMjLfnOKuiCYKHOehYnGxF7gP2BOWVaidpdiU7xcEJc8lDipeSbsw5ZqomMzzX0@mydomain1234.jfrog.io/artifactory/api/p
ypi/remote-pypi/simple --no-input --upgrade --no-deps -r
'c:\users\myuser1\appdata\local\temp\pipenv-96xsr5ri-requirements\pipenv-utfaoyk6-reqs.txt'
Using source directory: 'C:\\temp\\pipenv\\.venv\\src'
Looking in indexes: https://myuser1:****@mydomain1234.jfrog.io/artifactory/api/pypi/remote-pypi/simple

Password is masked in the last line
Looking in indexes: https://myuser1:****@mydomain1234.jfrog.io/artifactory/api/pypi/remote-pypi/simple,
but not in the one above.

'C:\Users\myuser1\AppData\Local\Programs\Python\Python312\Lib\site-packages\pipenv\patched\pip\__pip-runner__.py'
install -i
https://myuser1:kaMjLfnOKuiCYKHOehYnGxF7gP2BOWVaidpdiU7xcEJc8lDipeSbsw5ZqomMzzX0@mydomain1234.jfrog.io/artifactory/api/p
ypi/remote-pypi/simple --no-input --upgrade --no-deps -r

Edit:
Here Github would probably have masked it as the line does not contain a line break. Might also depend a bit on where the shell inserts the actual line breaks (on Windows my shell has 120 cols). However if the jf cli does password masking, it should probably do that in all places.