bug: robot's filesystem is out of space,make push sometimes fails

[SyntaxColoring]

Overview

Our make push commands, used in development to test changes, sometimes fail—apparently because there's no space left on the robot's filesystem.

Steps to reproduce

1. Grab a fresh dev build (ot2-system.zip).
2. Push it to your OT-2 via the Opentrons App.
3. Run make push from this repo.
4. Look for out-of-space errors during the parts where files are unzipped.

[SyntaxColoring]

It feels like there’s something else going on here than just us having too many files. Especially since I keep running into this problem when no one else on the team seems to be. It feels like there’s some secondary environmental factor that’s making the difference, like me updating from the Opentrons App beforehand, or not updating from the Opentrons App beforehand, or something?

Notes from @sfoster1 on the root cause:

Here's the problem: when buildroot crystallizes its rootfs artifact, it crushes the partition down to the minimum size for the content. so if you do an app install, you can't add a lot to it

Also i'm pretty sure that with adding a bunch of gigantic python packages like pandas we're running low on space on the rootfs in general.

there's two long-term solutions to this:

• make sure the stupid filesystem and partition takes up all the available space in the partition table
• figure out how to make python packages that require native extensions installable, which they currently are not because there's no compiler, because buildroot is really against allowing it, because "having a compiler" also implies "having development headers for everything" and it causes your space usage to balloon

And from @sanni-t:

Seth and I spent a good hour a few weeks ago trying to figure out how to expand the filesystem and were running into weird errors that made it sound like the fs was already expanded but we were still missing about 18MB of space.
Then another thing is that buildroot doesn’t compress the rootfs down perfectly to the size used on rootfs; I don’t remember ever running into that space issue when rootfs size was about 285MB last year. So I think it’s worth testing whether freeing up some space before packaging rootfs mitigates the problem.
The ‘something else’ problem which makes your robot run into space issues could be related to inodes. We think there’s about 4MB of space being taken up by inodes right now and that could vary a bit from system to system.

[SyntaxColoring]

Note from @mcous: As of Opentrons/buildroot#150 / robot software v5.1, the instructions below won't work (because the test files are no longer present on the robot). Happily, the workaround below shouldn't be necessary as make push is expected to work once your robot has been updated to v5.1.

---

I've successfully used this workaround:

1. Install a clean ot2-system.zip through the Opentrons App.
2. SSH in
3. mount -o remount,rw /
4. rm -rf /usr/lib/python3.7/site-packages/pandas/tests /usr/lib/python3.7/site-packages/numpy/core/tests /usr/lib/python3.7/site-packages/tornado/test
5. df / shows 0 available blocks still, which is weird, but whatever
6. mount -o remount,ro /
7. Exit SSH
8. make push

This removes some big directories that we don't need. The removed directories were chosen based on this investigation from @sanni-t and @amitlissack:

Size  | Path
-------------------
116.4M	/usr/lib/python3.7/site-packages
Largest python packages:
24.4M	/usr/lib/python3.7/site-packages/pandas
11.7M	/usr/lib/python3.7/site-packages/numpy
30.9M	/usr/lib/python3.7/site-packages/notebook
4.7M	/usr/lib/python3.7/site-packages/pydantic
Largest test folders:
7.0M	/usr/lib/python3.7/site-packages/pandas/tests

1.1M	/usr/lib/python3.7/site-packages/numpy/core/tests

773.0K	/usr/lib/python3.7/site-packages/tornado/test
Smaller test folders:
104.0K	/usr/lib/python3.7/site-packages/jsonschema/tests

369.0K	/usr/lib/python3.7/site-packages/nbconvert/tests

111.0K	/usr/lib/python3.7/site-packages/traitlets/tests

132.0K	/usr/lib/python3.7/site-packages/urwid/tests

534.0K	/usr/lib/python3.7/site-packages/numpy/lib/tests

266.0K	/usr/lib/python3.7/site-packages/numpy/ma/tests

121.0K	/usr/lib/python3.7/site-packages/numpy/f2py/tests

134.0K	/usr/lib/python3.7/site-packages/numpy/polynomial/tests

313.0K	/usr/lib/python3.7/site-packages/IPython/core/tests

156.0K	/usr/lib/python3.7/site-packages/nbformat/tests

291.0K	/usr/lib/python3.7/site-packages/notebook/tests

206.0K	/usr/lib/python3.7/site-packages/zmq/tests

[SyntaxColoring]

Some further investigation.

Here is my robot's output for fdisk -l /dev/mmcblk0:

Disk /dev/mmcblk0: 14.9 GiB, 15931539456 bytes, 31116288 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *       8192    90111    81920   40M  c W95 FAT32 (LBA)
/dev/mmcblk0p2        90112   729087   638976  312M 83 Linux
/dev/mmcblk0p3       729088  1368063   638976  312M 83 Linux
/dev/mmcblk0p4      1368064 30785535 29417472   14G  f W95 Ext'd (LBA)
/dev/mmcblk0p5      1376256  1417215    40960   20M 83 Linux
/dev/mmcblk0p6      1425408 30785535 29360128   14G 83 Linux

Since p2 and p3 perfectly fill the space between p1 and p4, I am assuming that 638976 fdisk sectors (327155712 bytes) is the correct size for each of those partitions.

Now, if I do dumpe2fs /dev/mmcblk0p2, I get Block count: 319488 and Block size: 1024, which multiply to 327155712 bytes—exactly the same as the partition size.

So, with regards to:

Seth and I spent a good hour a few weeks ago trying to figure out how to expand the filesystem and were running into weird errors that made it sound like the fs was already expanded but we were still missing about 18MB of space.

I worry that, at some level, the filesystem really is already as big as it possibly can be. (I also tried resize2fs and got The filesystem is already 319488 (1k) blocks long. Nothing to do!, which I assume is the same message that @sanni-t and @sfoster1 got.)

I'm yet sure how to interpret the rest of the dumpe2fs output, or how to reconcile this 327155712 bytes value with what df / reports for me, which is ~18 MiB lower: 308438016 bytes.

[mcous]

The immediate issue of make push not working seems to have been resolved by Opentrons/buildroot#150 (and others).

Going to leave Opentrons/buildroot#144 open, since there's probably more stuff we can remove.