Changes required to run SIMX on HPCAC #71

ztiffany · 2021-06-15T22:08:15Z

Signed-off-by: Zach Tiffany [email protected]

This is a dirty set of changes that were made to set up MKT for run on HPCAI. Do not merge.

Signed-off-by: Zach Tiffany <[email protected]>

artpol84 · 2021-06-15T22:10:29Z

docker/fc34/kvm.Dockerfile

+    vim \
+    iperf \
+    crash \
+    zstd \


this is the development convenience.

Maybe it'd be cool to allow user the ability to provide his own docker file that will incrementally append needed things to the already existing image,

It was always in back of my mind, but didn't investigate how to do it without rebuilding all images.

Can you just apply something on top of the existing image?
Like running one additional docker file over the existing image extending it and making the new one the “current “?

artpol84 · 2021-06-15T22:10:41Z

docker/fc34/kvm.Dockerfile

-RUN /root/basic-setup.sh && /root/kvm-setup.sh
+RUN /root/basic-setup.sh
+
+RUN /root/kvm-setup.sh


can be ignored

ztiffany · 2021-06-15T22:10:52Z

plugins/do-build.py

@@ -42,6 +42,12 @@ def make_simx(args):

    subprocess.call(cmd + ['-j%d' %(args.num_jobs)])

+def make_rdmo_app(args):


I started throwing in some stuff to make MKT build my rdmo app. I abandoned that, though. Ignore references to rdmo-app and the packages added to support.Dockerfile.

I added packages to the VM image to build rdmo-app inside my VM instead.

Building inside VM looks simpler, but it misses the MKT concept. We wanted to separate build environment from run environment. It allows us to enjoy from specific optimizations and makes run fast.

artpol84 · 2021-06-15T22:11:12Z

docker/fc34/support-simx.sh

-# git_url: http://l-gerrit.mtl.labs.mlnx:8080/simx
-# git_commit: 41f602dc05b3c115b176ac3f7869e8bd390cbd92
+# git_url: /global/home/users/ztiffany/test/simx
+# git_commit: 3f3c2c9338f3bbb73cf3bd298152e020e394086f


can be ignored

artpol84 · 2021-06-15T22:11:54Z

docker/fc34/support-simx.sh

@@ -18,7 +18,7 @@ From simx.git
 %build
 ./mlnx_infra/config.status.mlnx --target=x86 --prefix=/opt/simx
 make %{?_smp_mflags}
-make %{?_smp_mflags} -C mellanox/
+make %{?_smp_mflags} -C mellanox/ SIMX_PROJECT=mlx5


This tells SimX to only build the NIC part. I think it makes sense unless the switch part is planned to be used.

Yeah, it makes sense for now. A long time ago, I pitched this project to switch team, they even tried it, but decided to stick with VMs because of differences in technical level expertise between development team and verification team.

artpol84 · 2021-06-15T22:13:16Z

docker/fc34/support-smatch.sh

-# git_url: git://repo.or.cz/smatch.git
-# git_commit: 9bb66fa2d7c73b3338a27fd6b38d7d509b2a1c1b
+# git_url: /global/home/users/artemp/scratch/.cache/mellanox/mkt/smatch.git
+# git_commit: 72c21a144a812cadbe349801da1b24bc331af256


For some reason, the site where we were building this can't access the original URL.
This is specific to that site and shouldn't be considered, probably.
Especially given that "mkt images" is not a must.

IIRC if you preload the normal cache directory it doesn't require network access so long as the commit_id is already present. So these weird disconnected cases are solved by transfering the cache directory from some network connected machine

What is normal cache directory?

➜ kernel git:(master) ls ~/.cache/mellanox/mkt
iproute2-next.git rdma-core.git simx.git smatch.git sparse.git tc-build.git

That’s the issue, it fails to download there from got saying connection refused.
Again, I don’t think we should consider it in MKT. this is obviously the issue on that site.

we just haven’t cleaned the version we ended up with for the sake of time consumption.

artpol84 · 2021-06-15T22:13:46Z

docker/fc34/support-sparse.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 # ---
-# git_url: git://git.kernel.org/pub/scm/devel/sparse/sparse.git
-# git_commit: 8af2432923486c753ab52cae70b94ee684121080
+# git_url: /global/home/users/artemp/scratch/.cache/mellanox/mkt/sparse.git


same as above

artpol84 · 2021-06-15T22:15:10Z

plugins/do-build.py

@@ -79,9 +85,13 @@ def setup_from_pickle(args, pickle_params):
    subprocess.check_output(['make', 'headers_install',
        'INSTALL_HDR_PATH=/usr'], cwd=args.kernel)

+if not os.path.isdir('/images/ztiffany/ccache'):


This is likely not needed based on my later experience

ztiffany · 2021-06-15T22:57:43Z

plugins/do-kvm.py

@@ -64,11 +64,17 @@ def remove_mounts():


 def is_passable_mount(v):
+    print ("Checking mount: {}".format(v))
    if v[2] == "nfs" or v[2] == "nfs4":


qemu-system-x86_64: -fw_cfg etc/sercon-port,string=2: warning: externally provided fw_cfg item names should be prefixed with "opt/" qemu-system-x86_64: -device virtio-9p-pci,fsdev=host_bind_fs0,mount_tag=bind0: cannot initialize fsdev 'host_bind_fs0': failed to open '<snip>': Permission denied

"Permission denied" - let's debug, it shouldn't

If I am root on the node, I cannot LS my users home directory

Ls fails with Permission denied as well

ztiffany · 2021-06-15T23:09:44Z

plugins/do-kvm.py

+    if v[1].startswith("/images/"):
+        print ("YES!!!")
+        return True
+    if v[1].startswith("/plugins"):


HPCAC nodes are diskless. Here is how plugins are mounted:

Evaluating: /plugins v is: ['tmpfs', '/plugins', 'tmpfs', 'ro,relatime,mode=555', '0', '0'] Passing: /plugins

Here is from a working system:

['/dev/sda5', '/plugins', 'ext3', 'ro,relatime', '0', '0']

AFAIK, docker can't mount tmpfs, need to think about workaround

It does work if we add the above

So I think it should work as is

ztiffany · 2021-06-15T23:11:33Z

plugins/do-kvm.py

@@ -64,11 +64,17 @@ def remove_mounts():


 def is_passable_mount(v):


On HPCAC, this was needed to get the rdma-core directory passed through:

mkt run --dir /images/ztiffany/src/rdma-core/

Is this expected?

no, it means your config file is incomplete or another bug, we mount whole src directory

It could be because /images is on tmpfs, but I don’t think we saw even an attempt to mount it

ztiffany · 2021-06-15T23:12:52Z

plugins/do-build.py

@@ -97,3 +107,5 @@ def setup_from_pickle(args, pickle_params):
    make_rdma_core(args)
 if args.project == "simx":
    make_simx(args)
+if args.project == "rdmo-app":


Ignore this.

ztiffany · 2021-06-15T23:13:05Z

utils/build.py

@@ -67,6 +67,7 @@ def run_ci_cmd(self, supos):
        "rdma": "iproute2",
        "kernel": "kernel",
        "mlnx_infra": "simx",
+        "rdmo-app": "rdmo-app",


artpol84 · 2021-06-15T23:13:28Z

utils/cmdline.py

@@ -27,7 +27,7 @@ def get_cache_fn(fn):
    an impact on the operation of mkt - at worst it will run slower."""
    global cache_dir
    if cache_dir is None:
-        cache_dir = os.path.expanduser("~/.cache/mellanox/mkt/")
+        cache_dir = '/images/ztiffany/.cache/mellanox/mkt/'


the home dir is insufficient to hold these caches,
Is there a way to point it somewhere else?

.cache is general mechanism, it is worth to make symlink

Makes sense

jgunthorpe · 2021-06-16T00:46:19Z

@ztiffany please fix your git config to use your nvidia address, thanks

artpol84 · 2021-06-16T01:12:10Z

@jgunthorpe this is not intended for merge. It’s an FYI to indicate what we had to hack to make it work on the particular system.

we agreed with @rleon that we will open this one

rleon · 2021-06-16T05:53:31Z

docker/fc34/basic-setup.sh

@@ -69,4 +69,5 @@ cat <<EOF > /etc/sysctl.d/hugepages.conf
 vm.nr_hugepages=2
 EOF

-rpm -U /opt/rpms/*.rpm
+#rpm -U /opt/rpms/*.rpm
+rpm -U --force /opt/rpms/*.rpm


Fixed in c2f86ca

Changes required to run SIMX on HPCAI

bc3a267

Signed-off-by: Zach Tiffany <[email protected]>

artpol84 reviewed Jun 15, 2021

View reviewed changes

docker/fc34/kvm.Dockerfile

RUN /root/basic-setup.sh && /root/kvm-setup.sh

RUN /root/basic-setup.sh

RUN /root/kvm-setup.sh

Copy link

artpol84 Jun 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can be ignored

ztiffany commented Jun 15, 2021

View reviewed changes

artpol84 reviewed Jun 15, 2021

View reviewed changes

ztiffany commented Jun 15, 2021

View reviewed changes

ztiffany changed the title ~~Changes required to run SIMX on HPCAI~~ Changes required to run SIMX on HPCAC Jun 15, 2021

ztiffany commented Jun 15, 2021

View reviewed changes

artpol84 reviewed Jun 15, 2021

View reviewed changes

rleon reviewed Jun 16, 2021

View reviewed changes

		@@ -42,6 +42,12 @@ def make_simx(args):

		subprocess.call(cmd + ['-j%d' %(args.num_jobs)])

		def make_rdmo_app(args):

		@@ -64,11 +64,17 @@ def remove_mounts():


		def is_passable_mount(v):

Changes required to run SIMX on HPCAC #71

Are you sure you want to change the base?

Changes required to run SIMX on HPCAC #71

Conversation

ztiffany commented Jun 15, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ztiffany Jun 15, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

artpol84 Jun 15, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

artpol84 Jun 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rleon Jun 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgunthorpe commented Jun 16, 2021

artpol84 commented Jun 16, 2021

Choose a reason for hiding this comment

ztiffany Jun 15, 2021 •

edited

Loading

artpol84 Jun 15, 2021 •

edited

Loading

artpol84 Jun 16, 2021 •

edited

Loading

rleon Jun 16, 2021 •

edited

Loading