Support dev containers through a Remote-SSH session

[szlend]

The original issue #20 was closed (as part of the April release), but it only implemented dev container support through WSL. The SSH part of the feature request is still unsupported as far as I could tell.

Feature request

When connected through SSH in a remote development session, the host’s Docker containers should be discoverable and VS Code should be able to use its Docker daemon for container remote development with devcontainer.json.

There are currently ways to make attaching to containers work described in the docs:

• https://code.visualstudio.com/docs/remote/containers-advanced#_developing-inside-a-container-on-a-remote-docker-host
• https://code.visualstudio.com/docs/containers/ssh

These do work to a point and I am able to attach to a container through SSH. However the devcontainer.json part does not work inside a Remote-SSH session. Visual Studio Code does not pick up the file and does not suggest restarting the session in a container like it does locally. Most Remote-Container options are not available in the Command Palette either.

As far as I can tell, you can get this working by skipping the Remote-SSH session, and configuring the Docker daemon to connect through SSH like in the docs. However this requires maintaining a separate devcontainer.json, and having two source trees (local and remote), which is far from ideal.

It's possible to configure the ms-vscode-remote.remote-containers to run as a workspace extension through remote.extensionKind, which is closer to what I would like. Though that does not work either because as far as I can tell, when reopening a session in a container, VS Code is unaware that the container was running behind Remote-SSH.

[SR-G]

I think i have the exact same situation :

1. a linux desktop (or a windows laptop), running only VSCode + (if needed) docker client (only) (and no local workspace / no local docker daemon)
2. a remote linux server, running :

• docker daemon with port 2375 exposed
• my workspace
• if needed (probably not), workspace may be mounted through SSHFS or exposed through CIFS

So from my local computers, i would like to Remote-SSH on the linux server (as i don't have / want the files in local), and from there, to be able to Remote-Container/Open the project, with the remote docker client / remote docker daemon (so i should even not need the local docker client).

At this time / as far as i can tell, this seems to not be working (with 1.45.1).

Maybe it's a corner case, but in my situation :

• it's way better to use a remote docker daemon (fast server with plenty of RAM versus slow desktop/laptop)
• it's way better to have my workspaces in a share location in my remote server, allowing me to retrieve the same workspaces from several computers acting as clients.

[guoquan]

@SR-G

At this time / as far as i can tell, this seems to not be working (with 1.45.1).

Maybe it's a corner case, but in my situation :

• it's way better to use a remote docker daemon (fast server with plenty of RAM versus slow desktop/laptop)
• it's way better to have my workspaces in a share location in my remote server, allowing me to retrieve the same workspaces from several computers acting as clients.

I think I have a similar situation and I was using remote.extensionKind as a workaround.
I added the following setting to the remote setting.json:

{
    "remote.extensionKind": {
        "ms-azuretools.vscode-docker": [ "ui" ],
    },
    "docker.host":"tcp://<remote_ip>:2375"
}

The case is that I will always open a window for the remote first, then attach to the container, and keep the window for remote open in the background.
Well, might be this is typical for WINDOWS because it's MS 🤣no offensive.

I have a "proxyjump" server in the middle and CIFS filesystem on remote and mounted to the container (might need some other workaround) and they all work well for me.

@szlend had the same solution I think, and as mentioned, the concern is that it is really a just a workaround and really not supported by the extension. Some features of the extension are not usable in this case.

[diricxbart]

Our setup, in short, is as following: our Docker daemon is running on a Linux server, we are establishing a tunnel to it using: ssh server_name -L localhost:2375:/var/run/docker.sock -N and then use the .devcontainer.json approach to create and launch the Docker container on the server and attach to it with VSCode installed on a Windows laptop. The Docker images are build on that server (using Jenkins).

However, we still have the following issue:

1. The build docker images contain a custom user 'my_user' with UID & GID both 2000
2. Using the postCreateCommand we change the UID & GID of 'my_user' to the UID & GID of the user starting the container. But for this we need to run this command as root user.

The issue is that we then also need to start the container as root user, which causes file permission issues on volume mounts. For example, all new created files are owned by root, not 'my_user'...

If we could specify a specific user to be used to execute the postCreateCommand, then we could run the postCreateCommand as 'root'. If we could then start the container as 'custom_user', which now will already have the correct UID & GID, then we wouldn't have the file permission issues on the volume mounts anymore.

Could this functionality please be added? See also here for a similar request.

Or is there something we can try using the existing functionality? I noticed that recently the postStartCommand and postAttachCommand commands are added, however I didn't find a way yet how to solve this issue with those new commands either.

[chrmarti]

@diricxbart Have you considered updating UID & GID by building an image from the original image with UID 2000? We do that when the local machine is Linux, unfortunately that feature cannot be configured to run when the local machine is Windows (something to think about).

[diricxbart]

@chrmarti My local machine is Windows... What I've did in the past was to, on the Linux server, launch the Docker container and add the user there. Next from VSCode attach to the running Docker container. But this is a bit of a cumbersome workflow.

Ideally I just open the .devcontainer.json based environment in VSCode from my Windows machine, it launches the Docker container on the Linux server and works as if all was installed locally. My current experience is nearly that, apart from the fact that my file permissions are messed-up all of the time...

I really don't mind adding more complexity in the Docker container and/or in the .devcontainer.json, if needed. This is a one-time effort, reusable across different projects, ...

To me this postCreateCommand is a nearly perfect solution, it allows to modify the 'base' Docker container. One of the use cases here is to change the user UID and GID. (We also use this to create derived Docker containers from a 'base' Docker image, each with different Python packages installed.) So we use this to perform some tweaks / changes on our 'base' Docker image. The downside is that changing UID and GID has to be done as root (you can't have the 'custom_user' change its UID and GID, even as sudo, it again screws up the file permissions).

So to me, it would be perfect if I could run the postCreateCommand as root user (via a "postCreateUser" setting?) and specify for example 'custom_user' via the "remoteUser" setting...

[diricxbart]

@chrmarti How do you suggest to continue here? What do you think about the option of adding a separate "postCreateUser" setting (next to the existing "remoteUser" setting? Should I create a separate issue for this, or is it OK to track this in this one?

[chrmarti]

@diricxbart A separate issue would be great. This issue will address the problem in a more general way.

[theasianpianist]

@guoquan

I have a "proxyjump" server in the middle and CIFS filesystem on remote and mounted to the container (might need some other workaround) and they all work well for me.

Can you elaborate on how you are able to connect to a container on a remote host via a proxyjump? I have an identical setup and cannot figure out how to incorporate the proxy jump information into the 'docker.host' setting.

[guoquan]

Hi @theasianpianist,
Basically, I set my docker extension to be working locally, and docker.host to a local sock in server-level configuration, and set up a default LocalForward in my ssh connection from the remote docker sock to the specific local sock.
Concretely, in server-level configuration, I added the following section:

{
    "remote.extensionKind": {
        "ms-azuretools.vscode-docker": [ "ui" ],
    },
    "docker.host": "unix:///tmp/foo_bar_.sock",
}

And in my local ~/.ssh/config, I added

Host myproxy
  User my_proxy_user
Host foo
  HostName foo.bar.com
  User my_remote_user
  ProxyJump myproxy
  LocalForward /tmp/foo_bar_.sock /var/run/docker.sock
  StreamLocalBindUnlink yes

In this way, the configurations are valid only for this server, and all the socks are activated automatically when you connect to foo server on vscode. You can even config for multiple servers at ones.

[BigPapa314]

Hi,

I have to develop on a remote servers docker instance. Therefore I setup a docker context and configured a devcontainer. The container comes up but the project folder is not mounted. It seams that docker is not able to mount from remote hosts. If I use the local docker context all works fine.

If I could connect to the remote server via Remote-SSH, checkout the project and open the devcontainer there, the project folder should be able to get mounted.

I think the only option currently is a vscode installation on the server with SSH / X forwarding.

It would be awesome if only an empty server with docker installed would be enough to utilize remote resources.

[PavelSosin-320]

Docker çan mount everything using the proper driver passed with run volume option but bind means local filesystem. בתאריך יום א׳, 14 במרץ 2021, 22:02, מאת Thomas ***@***.***>:

…

Hi, I have to develop on a remote servers docker instance. Therefore I setup a docker context and configured a devcontainer. The container comes up but the project folder is not mounted. It seams that docker is not able to mount from remote hosts. If I use the local docker context all works fine. If I could connect to the remote server via Remote-SSH, checkout the project and open the devcontainer there, the project folder should be able to get mounted. I think the only option currently is a vscode installation on the server with SSH / X forwarding. It would be awesome if only an empty server with docker installed would be enough to utilize remote resources. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2994 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM7YVKCNP7ZIOJKQYM3PX7TTDUI5RANCNFSM4NAO43IQ> .

[BigPapa314]

@PavelSosin-320 you are right! I'll check if its possible to share the project folder automatically with vscode and mount it in the container. (Maybe nfs or cifs, ...)

[chrmarti]

I will check, creating separate issues to keep track. 👍

[le-ortega]

Hi! We are really interested in this feature. We are wondering if you know when is this supposed to be released. At the same time is some kind of release plan available just to check it?
Thanks for your work!

[maresb]

@le-ortega please see #2994 (comment) for instructions on how to get started.

[felipecrs]

@le-ortega #5828

[phillipo6]

@chrmarti so while working with the insider release, i encountered another bug.
When in the startup of the devcontainer some error occures (such as docker cant bind a port, because it's already in use) i am unable to "Reopen in SSH", nothing happens when running that command

[chrmarti]

Published 0.209.0 with fixes for the environment variables, Git credentials and SSH/GPG agent forwarding. (Fixing #5976 and #5977.)

@phillipo6 Make sure you are on the latest VS Code Insiders version, there was a similar bug fixed recently.

@le-ortega This will be available with VS Code 1.63 which is expected to ship next week.

[cbandera]

One more finding: Not sure whether this is avoidable or not...
When I was opening VSCode on the remote before and this "local" instance had already created the devcontainer.json Image with the tag specified in the "name": tag, then I get a error message from the docker deamon while trying to setup the devenv from remote.
docker: Error response from daemon: Conflict. The container name "/<my_specified_name>" is already in use by container "410a23dd5f7641a44cb092bd9cf039bd7bd51f07664b910223ff3bce448b0228". You have to remove (or rename) that container to be able to reuse that name.
Since it is working locally to open a devcontainer and reuse the already existing but stopped container, I assume that this should also be the same behaviour when starting it from remote.
Maybe there is just a difference in expected container hashes because locally on the host I was using the "normal" VSCode while remotely I am using the insiders version?

[phillipo6]

Published 0.209.0 with fixes for the environment variables, Git credentials and SSH/GPG agent forwarding. (Fixing #5976 and #5977.)

@phillipo6 Make sure you are on the latest VS Code Insiders version, there was a similar bug fixed recently.

@le-ortega This will be available with VS Code 1.63 which is expected to ship next week.

No this did not fix my issue, when the creation failed and i do not directly choose the option reopen in ssh, instead i choose more actions. And then open the command palette and choose reopen in ssh, nothing happens.

Furthermore i pruned my whole docker system and now when i start the docker devcontainer, it is missing crucial extensions listed in the devcontainer.json. I have to install them manually (but not all!)
Okay that seems to be an issue with the c# extension which has a version mismatch with the marketplace

[chrmarti]

Published 0.209.1 with fixes for reconnection when the ssh connection drops (#5995) and the trust dialog showing twice (#5996).

Maybe there is just a difference in expected container hashes because locally on the host I was using the "normal" VSCode while remotely I am using the insiders version?

@cbandera I assume you are specifying a name in "runArgs"? Your guess is right then, insiders and stable create different containers.

No this did not fix my issue, when the creation failed and i do not directly choose the option reopen in ssh, instead i choose more actions. And then open the command palette and choose reopen in ssh, nothing happens.

@phillipo6 Could you open a bug report with details on your setup? (It worked when I tried your steps.)

[johnstairs]

This latest version is looking really solid. SSH agent forwarding was not working previously, but now it is.

[eitsupi]

This feature is really great. It works very smoothly for me now.

The only thing that bothered me was that the name of the "Recently Used Folders" that appears when we right-click on the Windows taskbar VSCode icon does not have the information "SSH" in it, so I may get a little lost.

[soumroy88]

Thanks @chrmarti really great work. Pending from long time.

I have below set-up,

• windows Host (no docker) + ssh key configured
• aws ec2 ubuntu instance + docker + repo with dev container.

When I reopen the container for some reason the Reopen stuck at "Installing Server" state.
Last log
[22496 ms] Start: Run in container: (dd iflag=fullblock bs=8192 count=6543 2>/dev/null; dd iflag=fullblock bs=3238 count=1 2>/dev/null) | tar --no-same-owner -xz -C /vscode/vscode-server-insiders/bin/linux-x64/7db1a2b88f7557e0a43fec75b6ba7e50b3e9f77e-insider_1638943187267

At this stage, all my ssh connection to the remote instance, e.g. via putty etc is almost dead. (extremely slow and unresponsive).
But, if I do Remote : Close remote connection then all ssh connections are normal again. This is strange!

PS: I am using a company internal tool for port forwarding and ssh is done at [email protected]:1337

devconnect forward --local-port 1337 --remote-port 22

Update: installation goes through after around 15-20 mins. and now everything is normal again! 👍

[dreoliv]

Not sure if this is an issue with this feature specifically, but I have two installed extensions that have a dependency between each other. Every time I connect to the container, I get the following message:

Cannot activate the 'SQLTools MySQL/MariaDB' extension because it depends on the 'SQLTools' extension, which is not loaded. Would you like to reload the window to load the extension?

The problem goes away if I reload the window.

My .devcontainer.json:

{
  "name": "Backend",
  "dockerComposeFile": ["../docker-compose.yml"],
  "service": "backend",
  "shutdownAction": "none",
  "extensions": ["prisma.prisma", "esbenp.prettier-vscode", "mtxr.sqltools", "mtxr.sqltools-driver-mysql"],
  "workspaceFolder": "/code",
  "remoteUser": "dev"
}

In this case, mtxr.sqltools-driver-mysql is dependent on mtxr.sqltools.

LMK if this isnt'related to this feature, I'll be happy to create a separate issue.

[chrmarti]

This is now available with VS Code 1.63. 🎉

I have opened new issues for the reports above and will follow up there.

Thanks everyone for the excellent feedback!

[alippai]

Should this work with remote Podman as well?

[chrmarti]

@alippai I haven't tested with Podman specifically, but it should work the same way it works locally. You will need Podman installed only on the SSH server (so remote Podman should not be needed).