Private repos failing to install

[woodpav]

My serverless.yml is:

plugins:
  - serverless-python-requirements
custom:
    pythonRequirements:
        dockerizePip: true
        dockerSsh: true

I have my ssh key in ~/.ssh/id_rsa (it's my only one). Installing the requirements works when I do not use sls deploy.

Permission denied (publickey).
fatal: Could not read from remote repository.

Changing to dockerizePip: false fixes the error. I'd like to use dockerizePip: true.

[dschep]

Hmm that's odd. Could you provide some more environment details? IE OS, sls version, & plugin version.

cc @cgrimal, Any idea why this might be failing?

[woodpav]

I'm on MacOS 10.13.6, serverless-python-requirements: 4.2.4, sls: 1.30.3

[smartinspereira]

I get the same error if my ssh key is password protected, removing the password resolves the error.

There is a mention of the ssh-agent in the README.md:69-70 w.r.t a password protected ssh key @dschep @cgrimal, but I do not really grasp what

If your SSH key is password protected, you can use ssh-agent because $SSH_AUTH_SOCK is also mounted & the env var set.

means in this context, if it only tells me to add the ssh-key prior to sls deploy, what I think it does, that is not changing anything. :-(

edit: MacOS 10.14.1 (18B75), serverless-python-requirements: 4.2.5, sls: 1.33.2

[bsamuel-ui]

Try this:

eval `ssh-agent`           # Don't do this on Mac OS X
ssh-add                    # To add the default .ssh/id_* keys
ssh-add ~/.ssh/foobar.pem  # To add a specific key

The ssh-add command will ask for your password, and after that, the unencrypted ssh key is now living in that agent. You should be able to run the process normally after that point.

Mac OS X manages ssh-agent for you, but on linux the eval command is setting an environment variable (SSH_AUTH_SOCK) corresponding to a domain socket that an agent process is listening to in the background. When you run ssh, it will forward the request through that socket to the agent process which will negotiate the crypto. The point of that dance is so your unencrypted key never leaves the agent.

[smartinspereira]

This is what I did. It is not working. I get the same error as @woodpav but there is a longer stack trace I could post later.

[dschep]

just to clarify, your private package is in your requirements.txt as something like git+ssh://github.com/foo/bar, yes?

[smartinspereira]

@dschep yes, exactly.
See the output of the commands:

• echo "$(less requirements.txt)"
• SLS_DEBUG=* sls deploy
• cd && git clone ssh://git@github.com/foo/bar

$ echo "$(less requirements.txt)"
psycopg2
git+ssh://git@github.com/foo/bar

$ SLS_DEBUG=* sls deploy
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generated requirements from [...]/requirements.txt in [...]/.serverless/requirements.txt...
Serverless: Installing requirements from [...]/.serverless/requirements/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.6
Collecting git+ssh://git@github.com/foo/bar.git (from -r /var/task/requirements.txt (line 1))
  Cloning ssh://git@github.com/foo/bar.git to /tmp/pip-req-build-r_sjq8dd
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
 
  Error --------------------------------------------------
 
  null
 
     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
 
  Stack Trace --------------------------------------------
 
Error: null
    at installRequirements ([...]/node_modules/serverless-python-requirements/lib/pip.js:262:11)
    at installRequirementsIfNeeded ([...]/node_modules/serverless-python-requirements/lib/pip.js:451:3)
    at ServerlessPythonRequirements.installAllRequirements ([...]/node_modules/serverless-python-requirements/lib/pip.js:530:29)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:391:22)
    at PluginManager.spawn (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:409:17)
    at Deploy.BbPromise.bind.then (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:117:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:107:10)
    at BbPromise.reduce (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:391:55)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:391:22)
    at PluginManager.run (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:422:17)
    at variables.populateService.then.then (/usr/local/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at processImmediate (timers.js:632:19)
    at process.topLevelDomainCallback (domain.js:120:23)
From previous event:
    at Serverless.run (/usr/local/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/usr/local/lib/node_modules/serverless/bin/serverless:44:28)
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           11.2.0
     Serverless Version:     1.34.1

$ cd && git clone ssh://git@github.com/foo/bar
Cloning into 'bar'...
remote: Enumerating objects: ..., done.
remote: Counting objects: ..., done.
remote: Compressing objects: ..., done.
remote: Total ..., reused ..., pack-reused ...
Receiving objects: ..., done.
Resolving deltas: ..., done.

[dschep]

Issue confirmed. I think it's an issue with shell-quote turning /path/to/sock:/tmp/sock:z into /path/to/sock\:/tmp/sock\:z 😢

[smartinspereira]

Oh no 😞.

Just did a little testing, maybe it helps. Using v1.2.0 seem to work, no error on sls package. Starting with v2.0.0, sls package is giving me the same error.

fixing /path/to/sock\:/tmp/sock\:z into /path/to/sock:/tmp/sock:z and fixing all the other escaped / neither helps.

[dschep]

Looks like the typescript plugin is to blame: https://github.com/prisma/serverless-plugin-typescript/search?q=.build&unscoped_q=.build

commented on the wrong pr in the wrong repo

[smartinspereira]

See comment above, but i found:

1. $SSH_AUTH_SOCK is not being forwarded to docker docker/for-mac#410
2. Support for sharing unix sockets docker/for-mac#483

so maybe we can't easily fix that issue?

[dschep]

Hmm. you're right. I removed the quote_single to test and it also fails :(

[bsamuel-ui]

I saw this referenced, it's basically using a TTY channel to forward the socket into the container which is a total hack, but one of the less awful ways I can think of doing it.