Known issues with SSH/SFTP connections using SSH.NET in gh bbs2gh

[timrogers]

When running a migration from Bitbucket Server/Data Center, we use the Bitbucket instance's REST API to ask it to generate a migration archive.

Once that archive is generated, it is stored on the Bitbucket Server's local storage, and there is no way to access that file over HTTP(S).

In order to enable an end-to-end migration, the gh bbs2gh CLI can automatically connect to the Bitbucket Server using SFTP or SMB to grab the archive and continue with the migration.

At the moment, for SFTP connections (which are the most common case - anecdotally, relatively few users are running Bitbucket on Windows), we are using the SSH.NET library to programatically perform SFTP operations.

There are a number of known issues with this library - or at least our implementation - which affect users.

Fortunately, there is a workaround: doing the SSH step outside of the CLI, and then passing the obtained file to the CLI to continue the next steps of thee migration.

The goal of this discussion is to document known issues and outlined potential next steps.

Known issues with SSH.NET

• SSH.NET only supports a limited set of SSH key ciphers listed here. When a key with an unsupported cipher is used, an exception like cipher name aes256-ctr for openssh key file is not supported is thrown.
• SSH.NET throws a Subsystem 'sftp' could not be executed error, even when an SFTP operation can be performed using the sftp binary directly.

Potential next steps

• Contribute back to SSH.NET to address known gaps: Contributing back to the community would be great, but SSH.NET seems to be largely unmaintained, with very few changes merged in the last year, and lots of pull request awaiting review.
• Do nothing, and push customers to the workaround above: This allows customers to move forward, but it means that we lose the magic end-to-end migration flow in the CLI, which is a big win for customers.
• Shell out and run sftp operations manually: This will avoid the idiosyncracies of SSH.NET and will allow us to support a wider range of setups natively, but it will probably bring other problems, and it will be hard to support Windows where the sftp binary is not available, most likely forcing us to support SSH.NET and shelling-out simultaneously.
• Add an option to generate-script which uses sftp directly in the generated script: This will allow us to offer more flexible and debuggable SFTP support, whilst not shelling out directly in the code.

[dylan-smith]

Another alternative approach is to have an option with generate-script that will generate a migration script that includes using sftp directly in the script. This avoids us having to support shelling out from within the CLI code, and gives customers more flexibility/debuggability IMO.

So for each repo in the script it would do something like:

gh bbs2gh migrate-repo ... # the combination of args to trigger archive generation only
sftp ... # download the archive locally
gh bbs2gh migrate-repo --archive-path ... # uploads the archive to cloud, kicks off migration

[timrogers]

Good point! Added to the main body of the discussion.

[dvalleit]

hello @timrogers @dylan-smith
Not sure if my question is related to this thread. I'm currently running a migration from BBS to GH using the CLI but the output shows the following:
[2023-09-27 18:19:33] [INFO] Export completed. Your migration archive should be ready on your instance at $BITBUCKET_SHARED_HOME/data/migration/export/Bitbucket_export_14.tar [2023-09-27 18:19:33] [INFO] Uploading Archive to Azure... [2023-09-27 18:19:33] [ERROR] System.IO.DirectoryNotFoundException: Could not find a part of the path '/var/atlassian/application-data/bitbucket/shared/data/migration/export/Bitbucket_export_14.tar'.

As you may see the file is successfully created (validated directly on the server). I tried using the generate-script scope but I cannot see any commands related to SFTP in it. Also checked the files permissions and all seems good. I know I can go manually but just want to know if there's any workaround to do it automated.

So I'm not sure if my issue is related to this, or maybe you have another scope to check this. Any help would be really appreciated.
Thanks

[dylan-smith]

I'm guessing you're missing the --ssh-user and --ssh-private-key args when calling the command.

@timrogers this is the 2nd time recently I've seen somebody make this mistake and get a confusing error message, I'm pretty sure we're missing an arg validation scenario here.

[timrogers]

@dvalleit Can you let us know if @dylan-smith's suggestion helps?

I'll add it to my to do list to take a look at this validations.

[dvalleit]

Sure thing! I'm waiting on the client to get the keys and let you know, but it definitely makes sense what @dylan-smith is saying. As Partner we have worked with this kind of migrations many times and I've never payed attention to the statement in the docs since it appear really hidden. Maybe put this highlighted would be awesome for people with the same issue.

Thanks both for the quick response

[dvalleit]

hello @timrogers and @dylan-smith, just passing by to let you know that finally ssh didn't since the server was on a VPC and insted trying to get credentials and stuff we generated the file, downloaded using gcloud commands and used bbs2gh with --archive-path param, so everything worked smoothly. We will use this approach but certainly I'll try with ssh first as you recommended.
thanks for all the help!

[Najzer0]

Thank you very much for your discussion here: SSH.NET throws a Subsystem 'sftp' could not be executed error, even when an SFTP operation can be performed using the sftp binary directly.
... the exact error was driving me insane and I tried to fix in ssh.net while it works fine with cli. Then I will stop my efforts on making this work