panic: runtime error: index out of range

[ChrisKnaf]

Hi guys,

sas support sent me here and asked if I could submit a new issue.

Building a --type single image and running it works fine.
When trying to build a --type multiple or --type full, I always get the following error:

2019/10/22 09:42:05 httpproxy:
Step 19/19 : LABEL sas.recipe="true" sas.recipe.version="19.06.1" sas.recipe.image="httpproxy" sas.layer.httpproxy="true"
2019/10/22 09:42:05 httpproxy:
---> Using cache
2019/10/22 09:42:05 httpproxy:
---> c066fd352551
2019/10/22 09:42:05 httpproxy:
---> c066fd352551
2019/10/22 09:42:05 httpproxy:
[Warning] One or more build-args [BASE SAS_RPM_REPO_URL] were not consumed
2019/10/22 09:42:05 httpproxy:
Successfully built c066fd352551
2019/10/22 09:42:05 httpproxy:
Successfully tagged anaxima/sas-viya-httpproxy:19.06.1-20191022094035-57c3c36
2019/10/22 09:42:05 index.docker.io/anaxima/sas-viya-httpproxy:19.06.1-20191022094035-57c3c36
panic: runtime error: index out of range

goroutine 73 [running]:
main.buildWorker(0x1, 0xc00030c0c0, 0xc000024360, 0xc000024420, 0xc000024300)
/sas-container-recipes/order.go:695 +0x8f4
created by main.(*SoftwareOrder).Build
/sas-container-recipes/order.go:843 +0x3a1
exit status 2
sas-container-recipes-builder-19.06.1-20191022094035-57c3c36

Not sure, what information to add... sas support told me, he would link this newly created issue to the internal support ticket.

I tried this with CentOS 7.5 in a VirtualBox on my laptop, with CentOS 7.6 in Azure and with RHEL 7.5 in Azure.

I added the following to line 679 in file order.go:

progress <- container.GetWholeImageName()

Output was

2019/10/22 09:42:05 index.docker.io/anaxima/sas-viya-httpproxy:19.06.1-20191022094035-57c3c36

The image is there, yet not pushed to the docker registry.
A simple

docker push anaxima/sas-viya-httpproxy

works fine

Tried this with docker hub (--docker-registry-url https://index.docker.io/v1/, also tried index.docker.io and https://index.docker.io) as well as a local registry running in a container.

I'm no go expert so debugging is kinda hard for me... besides, I don't really have a clue what I should check.

This is my build last command:

./build.sh --type multiple --zip ../SAS_Viya_deployment_data.zip --mirror-url http://chrismaschine/mirror --docker-registry-url https://index.docker.io/v1/ --docker-namespace anaxima --addons "auth-demo" --workers 1 --skip-docker-registry-push --verbose 2>&1 | tee /tmp/2019_10_22_11_40_build_httpd.log

Tried it with yum repo mirrored by SimpleHTTPServer as well as httpd with disabled selinux.
Tried it without mirror leading to a overflowing sas download counter, because - how sas support explained it - every image being build increments the counter... hence the mirror.

Any thoughts?

Thanks!

Chris

[g8sman]

Hi Chris,

I think the issue is that the registry url that is passed in is not being used by Docker when it tags the image. You can try the '--skip-registry-push' option and see if that gets you farther. I believe you will get all the images generated and the manifests as well. You can then do a manual push to get the images into the registry and then update the manifests to have the correct image location. In this case, there should only be three image locations to update.

Cheers!

Jon

[ChrisKnaf]

Hi Jon,

thanks for your help.
I now tried the following:

./build.sh 
--type multiple 
--zip ../SAS_Viya_deployment_data.zip 
--mirror-url http://`hostname`/mirror 
--docker-registry-url https://index.docker.io/v1/ 
--docker-namespace anaxima 
--addons "auth-demo" 
--skip-docker-url-validation 
--skip-docker-registry-push 
--workers 1 
--verbose

With this call, it built image "sas-viya-httpproxy"... didn't push it to the docker registry, so I did that manually with no problems.

Ran the command above a second time and it built the image "sas-viya-sas-casserver-primary".
Again, pushed it to docker manually.

Successfully built 2 of 3 images by now.

Ran the command above a third time then and expected the third image "sas-viya-programming" to be built.... didn't happen though, it built "sas-viya-sas-casserver-primary" a second time in spite of the fact that I followed the same steps as with the first image.

Now I'm stuck with multiple "sas-viya-sas-casserver-primary" images and it won't progress with the build of the last image.

Any ideas?

Thanks and best regards,

Chris

[ChrisKnaf]

I just checked the created manifests:

$ ll builds/multiple/manifests/kubernetes/services/
total 12
-rw-r--r-- 1 cknaf docker 393 Oct 31 12:23 cas.yml
-rw-r--r-- 1 cknaf docker 320 Oct 31 12:23 httpproxy.yml
-rw-r--r-- 1 cknaf docker 251 Oct 31 12:23 programming.yml

and found an inconsistency.

httpproxy.yml:

apiVersion: v1
kind: Service
metadata:
name: sas-viya-httpproxy

spec:
selector:
app: sas-viya-httpproxy
ports:
- name: "80"
protocol: TCP

programming.yml:

apiVersion: v1
kind: Service
metadata:
name: sas-viya-programming

spec:
selector:
app: sas-viya-programming
ports:
- name: "7080"
protocol: TCP

cas.yml:

apiVersion: v1
kind: Service
metadata:
name: sas-viya-cas

spec:
selector:
app: sas-viya-cas
ports:
- name: "5570"
protocol: TCP

Don't know if that's for any use, but I expected "app:" and "name" being "sas-viya-sas-casserver-primary", not only "sas-viya-cas".

Thanks again!

Chris

[g8sman]

Hi Chris,

I am a little confused by what you have provided. You should not have had to run the build three times. A one time run should have created the three images. I am glad that you got the three images though!

In regards to the unexpected name of the "app" for CAS, this was done on purpose. The Docker image created is sas-casserver-primary and it is used as a controller and worker. The name of the pods is sas-viya-cas because there was an issue in the "full" deployment when sas-viya-sas-casserver-primary was used. We shortened it for the full deployment and to keep things in sync, we did the same for the multiple deployment. If you run a MPP mode you will see "sas_viya"cas" and "sas-viya-cas-worker" as the pod names.

Apologies for the confusion.

Cheers,

Jon

[ChrisKnaf]

We did some serious go debugging today and we found something, which helps us to workaround the error.

As mentioned in the log, the error occurs in order.go line 688.
Here are lines 683 to 688:

  imageInfo, err := container.SoftwareOrder.DockerClient.ImageList(container.SoftwareOrder.BuildContext,
  	types.ImageListOptions{Filters: filterArgs})
  if err != nil {
  	container.SoftwareOrder.WriteLog(true, "Unable to connect to Docker client for image build sizes")
  }
  imageSize := imageInfo[0].Size

In our case, imageInfo is empty while err is not (683 & 684).
You do check err for being nil but you don't check, if len(imageInfo) > 0 before trying to access it.
Since imageInfo is empty, accessing it in 688 throws the panic error.

What we did next was to remove the filter in ImageListOptions, so in 684 we went from types.ImageListOptions{Filters: filterArgs}) to
types.ImageListOptions{}).

First run: all three images were created.

Than I removed all the images and did the build without the skipping.

./build.sh
--type multiple
--zip ../SAS_Viya_deployment_data.zip
--mirror-url http://hostname/mirror
--docker-registry-url https://index.docker.io/v1/
--docker-namespace anaxima
--addons "auth-demo"
--verbose 2>&1 | tee /tmp/hostname_date +"%Y-%m-%d"_4.log

Went through without any error, also pushed the three images to docker hub.
For now, removing the filter in Line 684 works for us.

But why does the filter not work / why is imageInfo empty?

When checking the images with a grep, all images were displayed:

$ docker image ls -a | grep sas
anaxima/sas-viya-programming 19.06.1-201911041155[...]
anaxima/sas-viya-sas-casserver-primary 19.06.1-201911041155[...]
anaxima/sas-viya-httpproxy 19.06.1-201911041155[...]
sas-container-recipes-builder 19.06.1-201911041155[...]

When doing this with the filter, it does only show one image:

$ docker image ls -a --filter=reference='sas'
REPOSITORY TAG IMAGE ID CREATED SIZE
sas-container-recipes-builder 19.06.1-201911041155[....]

Even with a completely wildcarded filter, it doesn't show all containers:

$ docker image ls -a --filter=reference='*'
REPOSITORY TAG IMAGE ID CREATED SIZE
sas-container-recipes-builder 19.06.1-201911041155[...]
centos 7 67fa590cfc1c 2 months ago 202MB
golang 1.12.0 cee68f119e19 8 months ago 772MB

The images

anaxima/sas-viya-programming
anaxima/sas-viya-sas-casserver-primary
anaxima/sas-viya-httpproxy

are not listed.

We think, that might be a bug in dockers cli...

What I really can't tell is, why nobody else is facing that problem...

Best regards,

Chris

[g8sman]

Hi Chris,

Thanks for the feedback. We can take a look and see about adjusting the code to what you found. The issue I think is using DockerHub as the registry. It looks like the Docker API drops the registry when Docker Hub is used which is what messes up the code. The image created does not contain the registry and we are expecting the registry to be a part of the image name. We have kept out of pushing to DockerHub as the image is publicly available at that point, unless you have you have a private account. All of our testing is with private Docker registries.

Thank you again for the code and we will look and see if we can add some logic in to help with interacting with DockerHub.

You may want to check your manifests as I suspect that the image they will try to pull down is not correct.

Cheers,

Jon

[ChrisKnaf]

Hi Jon,

I was not aware of the API problems when using DockerHub, although I did set my registry to "private" already...
Anyway, when using a local docker registry and using the build-options

--skip-docker-registry-push
--skip-docker-url-validation

the build process completes without an error.
Pushing the images manually works fine as well, I'm only facing some trouble deploying the new environment with the post-build kubctl-commands but I think I'll manage that with the kubernetes documentation.

I'm very happy about finally being able to build the images :-)

Thanks for your help!

Chris