Not working with Jupyter docker stacks

[abin-tiger]

I'm trying to build a custom image using the jupyter docker stacks.

# Using jupyter/pyspark-notebook:spark-3.1.2
FROM jupyter/pyspark-notebook@sha256:22908e014eacdbb86d4cda87d4c215d0b2354d88f29a5fbc510d7c642da10851

USER root

RUN pip install jupyterlab-sparkmonitor

RUN mkdir /home/ipython \
    && ipython profile create --ipython-dir /home/ipython \
    && chown -R jovyan:users /home/ipython \
    && echo "c.InteractiveShellApp.extensions.append('sparkmonitor.kernelextension')" >> /home/ipython/profile_default/ipython_kernel_config.py

# RUN jupyter nbextension install sparkmonitor --py
RUN jupyter serverextension enable --py --system sparkmonitor 

ENV IPYTHONDIR /home/ipython/ipython

USER ${NB_UID}

When I run it I'm getting the error

WARN: Jupyter Notebook deprecation notice https://github.com/jupyter/docker-stacks#jupyter-notebook-deprecation-notice.
/usr/local/bin/start-notebook.sh: running hooks in /usr/local/bin/before-notebook.d
/usr/local/bin/start-notebook.sh: running /usr/local/bin/before-notebook.d/spark-config.sh
/usr/local/bin/start-notebook.sh: done running hooks in /usr/local/bin/before-notebook.d
Executing the command: jupyter notebook
[I 17:39:36.585 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
[W 2021-11-19 17:39:37.661 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-11-19 17:39:37.661 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-11-19 17:39:37.661 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-11-19 17:39:37.661 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2021-11-19 17:39:37.675 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.9/site-packages/jupyterlab
[I 2021-11-19 17:39:37.675 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 17:39:37.800 NotebookApp] Serving notebooks from local directory: /home/jovyan
[I 17:39:37.800 NotebookApp] Jupyter Notebook 6.4.0 is running at:
[I 17:39:37.800 NotebookApp] http://0a683f496ccb:8888/?token=2feb3628fe68b399bf2dc2e69b01ccba3007f2e6cb45168b
[I 17:39:37.800 NotebookApp]  or http://127.0.0.1:8888/?token=2feb3628fe68b399bf2dc2e69b01ccba3007f2e6cb45168b
[I 17:39:37.800 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 17:39:37.805 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///home/jovyan/.local/share/jupyter/runtime/nbserver-8-open.html
    Or copy and paste one of these URLs:
        http://0a683f496ccb:8888/?token=2feb3628fe68b399bf2dc2e69b01ccba3007f2e6cb45168b
     or http://127.0.0.1:8888/?token=2feb3628fe68b399bf2dc2e69b01ccba3007f2e6cb45168b
[I 17:39:40.570 NotebookApp] 302 GET /?token=2feb3628fe68b399bf2dc2e69b01ccba3007f2e6cb45168b (172.17.0.1) 1.340000ms
[I 2021-11-19 17:39:48.215 LabApp] Build is up to date
[I 17:39:51.086 NotebookApp] Creating new notebook in 
[I 17:39:51.123 NotebookApp] Writing notebook-signing key to /home/jovyan/.local/share/jupyter/notebook_secret
[I 17:39:51.473 NotebookApp] Kernel started: 074d5f0e-e798-4106-bc1a-5f9251674ac5, name: python3
[W 17:39:55.156 NotebookApp] Got events for closed stream None
[IPKernelApp] ERROR | No such comm target registered: SparkMonitor
[IPKernelApp] ERROR | No such comm target registered: SparkMonitor
[I 17:40:11.661 NotebookApp] Starting buffering for 074d5f0e-e798-4106-bc1a-5f9251674ac5:e9836e60-7c01-42dd-9c4d-c28798e5d9ba
[I 17:40:41.307 NotebookApp] Shutting down on /api/shutdown request.
[I 17:40:41.308 NotebookApp] Shutting down 1 kernel
[I 17:40:41.510 NotebookApp] Kernel shutdown: 074d5f0e-e798-4106-bc1a-5f9251674ac5
[I 17:40:41.512 NotebookApp] Shutting down 0 terminals
SPARKMONITOR_SERVER: Loading Server Extension

[sfantangare-acn]

Following code worked for me. No need to execute in root mode, no need to set IPYTHONDIR explicitly

# Create ipython profile
RUN ipython profile create && \
    fix-permissions "${CONDA_DIR}" && \
    fix-permissions "/home/${NB_USER}"
# Add spark monitor kernel extention
RUN echo "c.InteractiveShellApp.extensions.append('sparkmonitor.kernelextension')" >>  "/home/${NB_USER}/.ipython/profile_default/ipython_config.py"