Unable to install a custom pack in docker image for K8 setup ! #5794

NitinNahal · 2022-11-02T09:23:30Z

NitinNahal
Nov 2, 2022

In one of my projects (moving the stackstorm setup from standalone to K8), I am trying to follow the documentation to create a docker image for the custom app notifier as mentioned in - https://github.com/StackStorm/st2packs-dockerfiles#building-the-st2packs-image

Here is my directory structure first -

nnahal$ ls -lrth
-rw-r--r-- 1 nnahal staff 324B Oct 30 06:24 Makefile
-rw-r--r-- 1 nnahal staff 2.0K Oct 30 06:24 README.md
drwxr-xr-x 4 nnahal staff 128B Oct 30 06:24 st2packs-builder
drwxr-xr-x 4 nnahal staff 128B Oct 30 06:24 st2packs-runtime
drwxr-xr-x 6 nnahal staff 192B Nov 1 12:13 st2packs-image
nnahal$ cd st2packs-image/
nnahal$ ls
Dockerfile README.md hooks packs
nnahal$ cd packs/
nnahal$ ls
notifier
nnahal$ ls -lrth notifier/
total 24
-rw-r--r-- 1 nnahal staff 10B Nov 1 14:18 README.md
drwxr-xr-x 4 nnahal staff 128B Nov 1 14:19 actions
drwxr-xr-x 3 nnahal staff 96B Nov 1 14:19 config
-rw-r--r-- 1 nnahal staff 150B Nov 1 14:19 config.schema.yaml
-rw-r--r-- 1 nnahal staff 212B Nov 1 14:19 pack.yaml
drwxr-xr-x 3 nnahal staff 96B Nov 1 14:19 rules

My pwd and Dockerfile are as follows -

nnahal$ pwd
/Users/nnahal/st2packs-dockerfiles

nnahal$ ls -lrth
total 16
-rw-r--r-- 1 nnahal staff 324B Oct 30 06:24 Makefile
-rw-r--r-- 1 nnahal staff 2.0K Oct 30 06:24 README.md
drwxr-xr-x 4 nnahal staff 128B Oct 30 06:24 st2packs-builder
drwxr-xr-x 4 nnahal staff 128B Oct 30 06:24 st2packs-runtime
drwxr-xr-x 6 nnahal staff 192B Nov 1 12:13 st2packs-image

nnahal$ cat st2packs-image/Dockerfile
ARG PACKS=“file:///tmp/stackstorm-st2”
FROM stackstorm/st2packs:builder AS builder
COPY packs/notifier /tmp/stackstorm-st2/
# Install custom packs
RUN /opt/stackstorm/st2/bin/st2-pack-install ${PACKS}
###########################
# Minimize the image size. Start with alpine:3.8,
# and add only packs and virtualenvs from builder.
FROM stackstorm/st2packs:runtime

Now when I try to build this image , I am able to see some steps but do not see my pack being taken into consideration nor do I see the other steps mentioned in Docker file (ls etc).

Below is the output of my build step -

st2packs-dockerfiles nnahal$ docker build -t nnahal/notifier:1.0 st2packs-image
[+] Building 2.0s (9/9) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 37B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/stackstorm/st2packs:runtime 1.7s
=> [auth] stackstorm/st2packs:pull token for registry-1.docker.io 0.0s
=> [stage-1 1/1] FROM docker.io/stackstorm/st2packs:runtime@sha256:33f4e77bdd2e942750d8631668ccc9973f12719dac96dc4081e99d30934da69c 0.0s
=> [builder 1/3] FROM docker.io/stackstorm/st2packs:builder 0.2s
=> => resolve docker.io/stackstorm/st2packs:builder 0.2s
=> CACHED [stage-1 2/1] COPY --from=builder /opt/stackstorm/packs /opt/stackstorm/packs 0.0s
=> CACHED [stage-1 3/1] COPY --from=builder /opt/stackstorm/virtualenvs /opt/stackstorm/virtualenvs 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:a73aab89e80875efa15debca79b573268d94c8249f814e373e5a6b6c86661b05 0.0s
=> => naming to docker.io/nnahal/notifier:1.0

Can someone please help me to understand if I am doing something wrong here ? When I see this image and check its contents, it does not have my pack installed, which is visible from above build logs.

Answered by NitinNahal

Dec 1, 2022

Thanks @armab for your support,

So I finally got it working today !

Problem : I forgot that in my config.schema.yaml I had a parameter to be required which was not being passed into the config file in the values.yaml.
Solution : Please remember to put required params in the configs which is part of the values.yaml.

i.e.

  packs:
    # Custom StackStorm pack configs. Each record creates a file in '/opt/stackstorm/configs/'
    # https://docs.stackstorm.com/reference/pack_configs.html#configuration-file
    # NOTE: This takes precedence over the contents of st2.packs.volumes.configs (if defined) on helm upgrade.
    configs:
      core.yaml: |
        ---
        # example core pack config …

View full answer

amanda11 · 2022-11-03T17:08:32Z

amanda11
Nov 3, 2022
Maintainer

Could you try with the names matching e.g. in the Dockerfile use:

ARG PACKS=“file:///tmp/notifier”
FROM stackstorm/st2packs:builder AS builder
COPY packs/notifier /tmp/notifier/

1 reply

NitinNahal Nov 4, 2022
Author

Thank you for replying.

I tried with this file as you suggested -

ARG PACKS=“file:///tmp/notifier”
FROM stackstorm/st2packs:builder AS builder
COPY packs/notifier /tmp/notifier/

# Install custom packs
RUN /opt/stackstorm/st2/bin/st2-pack-install ${PACKS}

###########################
# Minimize the image size. Start with alpine:3.8,
# and add only packs and virtualenvs from builder.
FROM stackstorm/st2packs:runtime

But when I build -

nnahal$ docker build -t nnahal/notifier:1.0 st2packs-image/
[+] Building 2.0s (9/9) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 401B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/stackstorm/st2packs:runtime 1.7s
=> [auth] stackstorm/st2packs:pull token for registry-1.docker.io 0.0s
=> [builder 1/3] FROM docker.io/stackstorm/st2packs:builder 0.2s
=> => resolve docker.io/stackstorm/st2packs:builder 0.2s
=> [stage-1 1/1] FROM docker.io/stackstorm/st2packs:runtime@sha256:33f4e77bdd2e942750d8631668ccc9973f12719dac96dc4081e99d3 0.0s
=> CACHED [stage-1 2/1] COPY --from=builder /opt/stackstorm/packs /opt/stackstorm/packs 0.0s
=> CACHED [stage-1 3/1] COPY --from=builder /opt/stackstorm/virtualenvs /opt/stackstorm/virtualenvs 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:a73aab89e80875efa15debca79b573268d94c8249f814e373e5a6b6c86661b05 0.0s
=> => naming to docker.io/nnahal/notifier:1.0

But still notifier pack is not installed in the image !

NitinNahal · 2022-11-04T08:39:59Z

NitinNahal
Nov 4, 2022
Author

Another strange thing is I added one additional line RUN ls -lrt packs , just to see if even the build process is able to reach this part and prints the content. But when I build the image this ls output is not there so I feel strange that is it even taking these changes into consideration.

Kindly help/suggest.

0 replies

NitinNahal · 2022-11-11T09:15:57Z

NitinNahal
Nov 11, 2022
Author

Hello,

So I have an update on this,

With some help of colleagues we noticed that image build up somehow was not working on mac os and also there is a missing path to certificates which we follow internally. So the update is that with ubuntu os we are now able to build the image and I have pushed the image in docker.hub:

https://hub.docker.com/repository/docker/nnahal/st2packs

But still trying to install the image with changes in values.yaml as follows -

    # Custom packs images settings.
    #
    # For each given st2packs container you can define repository, name, tag and pullPolicy for this image below.
    # Multiple pack images can help when dealing with frequent updates by only rebuilding smaller images for desired packs
    # E.g. having all desired StackStorm-Exchange packs in one image and several custom packs in additional images
    images:
      - repository: nnahal
        name: st2packs
        tag: "1"
        pullPolicy: Always

I run the command -

nnahal$ helm install st2 stackstorm/stackstorm-ha -f stackstorm-k8s/values.yaml
Error: INSTALLATION FAILED: failed post-install: timed out waiting for the condition

I re tried the command with increasing timeout and debug mode -

nnahal$ sudo helm install st2 stackstorm/stackstorm-ha -f stackstorm-k8s/values.yaml --debug --timeout 900s
install.go:178: [debug] Original chart version: ""
install.go:195: [debug] CHART PATH: /Users/nnahal/Library/Caches/helm/repository/stackstorm-ha-0.100.0.tgz

client.go:128: [debug] creating 49 resource(s)
client.go:310: [debug] Starting delete for "st2-job-st2-apikey-load" Job
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job st2-job-st2-apikey-load with timeout of 15m0s
client.go:568: [debug] Add/Modify event for st2-job-st2-apikey-load: ADDED
client.go:607: [debug] st2-job-st2-apikey-load: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for st2-job-st2-apikey-load: MODIFIED
client.go:310: [debug] Starting delete for "st2-job-st2-key-load" Job
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job st2-job-st2-key-load with timeout of 15m0s
client.go:568: [debug] Add/Modify event for st2-job-st2-key-load: ADDED
client.go:607: [debug] st2-job-st2-key-load: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for st2-job-st2-key-load: MODIFIED
client.go:607: [debug] st2-job-st2-key-load: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for st2-job-st2-key-load: MODIFIED
client.go:310: [debug] Starting delete for "st2-job-st2-register-content" Job
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job st2-job-st2-register-content with timeout of 15m0s
client.go:568: [debug] Add/Modify event for st2-job-st2-register-content: ADDED
client.go:607: [debug] st2-job-st2-register-content: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for st2-job-st2-register-content: MODIFIED
client.go:607: [debug] st2-job-st2-register-content: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for st2-job-st2-register-content: MODIFIED
Error: INSTALLATION FAILED: failed post-install: job failed: BackoffLimitExceeded
helm.go:84: [debug] failed post-install: job failed: BackoffLimitExceeded
INSTALLATION FAILED
main.newInstallCmd.func2
	helm.sh/helm/v3/cmd/helm/install.go:127
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/[email protected]/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/[email protected]/command.go:974
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/[email protected]/command.go:902
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	runtime/proc.go:255
runtime.goexit
	runtime/asm_amd64.s:1581

helm version -

nnahal$ helm version
version.BuildInfo{Version:"v3.9.4", GitCommit:"dbc6d8e20fe1d58d50e6ed30f09a04a77e4c68db", GitTreeState:"clean", GoVersion:"go1.17.13"}

Any suggestions ? @amanda11 , Thank you in advance.

22 replies

NitinNahal Nov 30, 2022
Author

Thanks @armab for your support,

So I finally got it working today !

Problem : I forgot that in my config.schema.yaml I had a parameter to be required which was not being passed into the config file in the values.yaml.
Solution : Please remember to put required params in the configs which is part of the values.yaml.

i.e.

  packs:
    # Custom StackStorm pack configs. Each record creates a file in '/opt/stackstorm/configs/'
    # https://docs.stackstorm.com/reference/pack_configs.html#configuration-file
    # NOTE: This takes precedence over the contents of st2.packs.volumes.configs (if defined) on helm upgrade.
    configs:
      core.yaml: |
        ---
        # example core pack config yaml
      notifier.yaml: |
        ---
        # example notifier pack config yaml
        certificate_path: "/usr/local/share/ca-certificates/"          # <----- was missing this line
    # Custom packs images settings.
    #
    # For each given st2packs container you can define repository, name, tag and pullPolicy for this image below.
    # Multiple pack images can help when dealing with frequent updates by only rebuilding smaller images for desired packs
    # E.g. having all desired StackStorm-Exchange packs in one image and several custom packs in additional images
    images:
      - repository: docker.io/nnahal
        name: st2packs
        tag: 1
        pullPolicy: Always

This was a small problem but took a lot of time.

Summarising our discussion : It is clear that the only way to proceed in such case is to observe which job is failing. If everything works well, pack installation last job ideally is to do st2-job-st2-register-content. In case you do not see it, chances are something is wrong internally.

As soon as you start install of helm chart with -f values.yaml, open another terminal,
run : kubectl get pods
Observe which pod results in crash consistently and without coming to status RUNNING or COMPLETED
Once identified it will be good to run kubectl logs -f -l job-name=<job-name> and see the last message.
To get this job name, run : kubectl describe pods and look for failed job, under it you will find the Labels which is basically the container identification forking these jobs. In labels there is job-name, use it in last command to get errors.

In my case error was this variable not being found in the values.yaml which the pack expects.

arm4b Nov 30, 2022

Nice, @NitinNahal! Glad things eventually worked out.

Also thanks for wrapping up example steps to debug the issue 👍

NitinNahal Dec 1, 2022
Author

Thanks @armab , I just want to mark this discussion answered for now but I do not see any option where I can do so. Is it possible for you ?

arm4b Dec 1, 2022

@NitinNahal Looks like it's not possible to mark the reply in a sub-thread as an answer.

Could you perhaps copy-paste the same details you wrote here (#5794 (reply in thread)) and post it outside of the thread?

We could mark it as an answer then. That sounds like a perfect detailed answer that will help others in the future.

NitinNahal Dec 1, 2022
Author

Thanks @armab , I added it below 👍

NitinNahal · 2022-12-01T15:32:17Z

NitinNahal
Dec 1, 2022
Author

Thanks @armab for your support,

So I finally got it working today !

Problem : I forgot that in my config.schema.yaml I had a parameter to be required which was not being passed into the config file in the values.yaml.
Solution : Please remember to put required params in the configs which is part of the values.yaml.

i.e.

  packs:
    # Custom StackStorm pack configs. Each record creates a file in '/opt/stackstorm/configs/'
    # https://docs.stackstorm.com/reference/pack_configs.html#configuration-file
    # NOTE: This takes precedence over the contents of st2.packs.volumes.configs (if defined) on helm upgrade.
    configs:
      core.yaml: |
        ---
        # example core pack config yaml
      notifier.yaml: |
        ---
        # example notifier pack config yaml
        certificate_path: "/usr/local/share/ca-certificates/"          # <----- was missing this line
    # Custom packs images settings.
    #
    # For each given st2packs container you can define repository, name, tag and pullPolicy for this image below.
    # Multiple pack images can help when dealing with frequent updates by only rebuilding smaller images for desired packs
    # E.g. having all desired StackStorm-Exchange packs in one image and several custom packs in additional images
    images:
      - repository: docker.io/nnahal
        name: st2packs
        tag: 1
        pullPolicy: Always

This was a small problem but took a lot of time.

Summarising our discussion : It is clear that the only way to proceed in such case is to observe which job is failing. If everything works well, pack installation last job ideally is to do st2-job-st2-register-content. In case you do not see it, chances are something is wrong internally.

As soon as you start install of helm chart with -f values.yaml, open another terminal,
run : kubectl get pods
Observe which pod results in crash consistently and without coming to status RUNNING or COMPLETED
Once identified it will be good to run kubectl logs -f -l job-name=<job-name> and see the last message.
To get this job name, run : kubectl describe pods and look for failed job, under it you will find the Labels which is basically the container identification forking these jobs. In labels there is job-name, use it in last command to get errors.

In my case error was this variable not being found in the values.yaml which the pack expects.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to install a custom pack in docker image for K8 setup ! #5794

{{title}}

Replies: 4 comments 23 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Unable to install a custom pack in docker image for K8 setup ! #5794

NitinNahal Nov 2, 2022

Replies: 4 comments · 23 replies

amanda11 Nov 3, 2022 Maintainer

NitinNahal Nov 4, 2022 Author

NitinNahal Nov 4, 2022 Author

NitinNahal Nov 11, 2022 Author

NitinNahal Nov 30, 2022 Author

arm4b Nov 30, 2022

NitinNahal Dec 1, 2022 Author

arm4b Dec 1, 2022

NitinNahal Dec 1, 2022 Author

NitinNahal Dec 1, 2022 Author

NitinNahal
Nov 2, 2022

Replies: 4 comments 23 replies

amanda11
Nov 3, 2022
Maintainer

NitinNahal Nov 4, 2022
Author

NitinNahal
Nov 4, 2022
Author

NitinNahal
Nov 11, 2022
Author

NitinNahal Nov 30, 2022
Author

NitinNahal Dec 1, 2022
Author

NitinNahal Dec 1, 2022
Author

NitinNahal
Dec 1, 2022
Author