Unable to clone repositories in Windows Containers

[atrauzzi]

Cross posting from here: microsoft/dotnet-framework-docker#61

Not sure what happened, but up until last week, I was able to clone git repositories via SSH-based URIs in my Windows containers. Now, suddenly without warning, when I go to clone - even a public repository, I get this error:

C:\>git clone git@github.com:atrauzzi/praxis.git
git clone git@github.com:atrauzzi/praxis.git
Cloning into 'praxis'...
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Here's a simple Dockerfile that can be used to reproduce the issue:

FROM microsoft/dotnet-framework:4.7.1-windowsservercore-1709
SHELL ["powershell"]

RUN Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
RUN choco install --yes git

RUN refreshenv

I'm not sure at this point what might be the issue - whether it's Windows or git. The only suspicious thing is that if I try to run the mingw based ssh.exe, it prints a blank line and then exits. My gut is telling me something about the SSH support in git for Windows is having a hard time?

[atrauzzi]

So. upon doing some more digging, it looks like the issue is to do with known_hosts not having the key for github.com. If I manually enter the container and run this command, I'm able to clone repositories:

./ssh-keyscan -v -H github.com >> /Users/ContainerAdministrator/.ssh/known_hosts

Unfortunately, as part of my container automation, I'm faced with some difficulties in resolving this correctly. My initial instinct was to add this to my Dockerfile:

WORKDIR '/Program Files/Git/usr/bin'
RUN ./ssh-keyscan -v -H github.com >> /Users/ContainerAdministrator/.ssh/known_hosts

Which gives me this error message during my container build:

Step 8/11 : RUN ./ssh-keyscan -v -H github.com >> /Users/ContainerAdministrator/.ssh/known_hosts
 ---> Running in 5132a8ee3d63
The command 'powershell ./ssh-keyscan -v -H github.com >> /Users/ContainerAdministrator/.ssh/known_hosts' returned a non-zero code: 1

So now I'm left trying to figure out what about being run in a container build seems to be upsetting the SSH tools bundled with git for Windows. I'm also a bit frustrated that I've had no error feedback at any step along the way.

If I try to run it via cmd rather than powershell, I get this error:

The command 'cmd /C "ssh-keyscan -v -H github.com" > /known_hosts' returned a non-zero code: 3221225794

[atrauzzi]

cc. docker/for-win#1351

[dscho]

it looks like the issue is to do with known_hosts not having the key for github.com

Not that I claim to know how to resolve this properly, but maybe you want to imitate what I do in my VSTS jobs:

mkdir -p .ssh &&
echo 'github.com,192.30.252.128 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==' >>.ssh/known_hosts

Sure, this would fail if the host key changed. But that's exactly what I would want it to do: to fail, so that I can figure out whether it was really GitHub that changed the key, or whether it was my cute buddy playin' a trick on me.

[atrauzzi]

@dscho - Indeed, I think I'm going to have to do something like that in the interim. It would be nice if I could have my Dockerfile pull that information as per the keyscan.

Thanks for the snippet, I'll probably end up using it until this can be fixed properly.

[thecloudtaylor]

Two part solution...
Part 1: Use the OpenSSH port for Windows (which is going in box!)
Part 2: Pre-create the .ssh directory

Part 1

FROM microsoft/windowsservercore:1709

ENV OPENSSH_VERSION 0.0.24.0
ENV OPENSSH_DOWNLOAD_URL "https://github.com/PowerShell/Win32-OpenSSH/releases/download/$OPENSSH_VERSION/OpenSSH-Win64.zip"

RUN powershell -Command
$ErrorActionPreference = 'Stop';
(New-Object System.Net.WebClient).DownloadFile('%OPENSSH_DOWNLOAD_URL%', 'openssh.zip') ;
Expand-Archive openssh.zip -DestinationPath c:\ ;
Remove-Item openssh.zip -Force

RUN setx PATH %PATH%;c:\OpenSSH-Win64

Part 2

FROM openssh

RUN mkdir C:\Users\ContainerAdministrator\.ssh
RUN ssh-keyscan.exe -v -H github.com >> /Users/ContainerAdministrator/.ssh/known_hosts

[Sean-Brown]

@taylorb-microsoft
I'm using a Bitbucket server, not Github, but the principal should be the same right? It doesn't seem to work for me though.

After installing openssh and adding the server's key to the known_hosts file (tested by successfully opening an ssh session on the Bitbucket server), I copied my Bitbucket ssh key to C:\Users\ContainerAdministrator\.ssh\, but now I still can't get the container to clone my repo.

Running ssh-agent doesn't output anything, but I believe it is running because ssh-add works:

Identity added: C:\Users\ContainerAdministrator.ssh\id_rsa (C:\Users\ContainerAdministrator.ssh\id_rsa)

but still cloning does not work:

Cloning into 'repo'...
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I'm curious if there are additional steps required to get this to work? Have you actually been able to successfully clone a Git repo in a Windows container?

[dscho]

Have you actually been able to successfully clone a Git repo in a Windows container?

Yes ;-) Almost all of Git for Windows' continuous testing and integration is performed in Docker containers itself.

BTW did you eval the output of ssh-agent? Otherwise, subsequent ssh sessions won't know how to connect to the agent.

To diagnose your woes further, I would suggest running ssh directly with the host name, with a liberal amount of -v thrown in: ssh -v -v -v -v -v -v bitbucket.org.

[Sean-Brown]

BTW did you eval the output of ssh-agent?

Is there an add-in that brings this command to powershell? Otherwise I don't seem to have eval as a recognized cmdlet.

Thanks for the -v pointer, it gives me a good starting point for debugging.

[Sean-Brown]

Thanks for the assistance, using that verbose flag helped pinpoint the issue: I was using a config file in my user's .ssh directory but apparently I needed to remove some permissions for a couple of users, as indicated by an error message:

debug3: Bad permissions. Try removing permissions for user: S-1-5-11 on file C:\Users\ContainerAdministrator/.ssh/config.

[dscho]

@atrauzzi can this ticket be closed?

[Custodia]

I'm having a similar problem but I haven't been able to pinpoint the problem yet. I just started looking into this and I don't seem to have the same Bad permissions error message. I do however have multiple errors like this: debug3: Failed to open file:C:\\Users\\ContainerAdministrator/.ssh/id_dsa error:2 for all the ssh files I've copied over to the container.

Did you use git clone without ssh private/public keys? And what user where you able to run the clone commands with? @atrauzzi @Sean-Brown

[Custodia]

Scratch what I said above, with ssh -vvvvvv mybitbucket.com I'm able to get Authenticated to mybitbucket.com ([192.168.70.1]:22). however git clone still fails with the same non descriptive error.