[Tutorial] Permanently Setting Your ULIMIT System Value

[TippyFlitsUK]

The Linux ulimit command output can be rather misleading when it comes to successfully running a Lotus process.
If the Max open files limit is not set correctly for Lotus you are very likely to run in to numerous random errors like the example shown below of a failed Window Post.

2020-11-04T13:42:00.666Z    INFO    storageminer    storage/wdpost_run.go:573    running window post    {"chain-random": "MkkfXf777PJMtpmQw97++Cnl08M8qSMcul/H4P+m+hI=", "deadline": {"CurrentEpoch":206364,"PeriodStart":206370,"Index":0,"Open":206370,"Close":206430,"Challenge":206350,"FaultCutoff":206300,"WPoStPeriodDeadlines":48,"WPoStProvingPeriod":2880,"WPoStChallengeWindow":60,"WPoStChallengeLookback":20,"FaultDeclarationCutoff":70}, "height": "206364", "skipped": 0}
2020-11-04T13:42:00.674 INFO filcrypto::proofs::api > generate_window_post: start
2020-11-04T13:42:00.681 INFO filecoin_proofs::api::post > generate_window_post:start
2020-11-04T13:42:00.681 INFO filecoin_proofs::caches > trying parameters memory cache for: Window_POST[34359738368]
2020-11-04T13:42:00.681 INFO filecoin_proofs::caches > found params in memory cache for Window_POST[34359738368]
2020-11-04T13:42:00.724 INFO filcrypto::proofs::api > generate_window_post: finish
2020-11-04T13:42:00.725Z    INFO    storageminer    storage/wdpost_run.go:590    computing window post    {"batch": 0, "elapsed": 0.0589228}
2020-11-04T13:42:00.725Z    ERROR    storageminer    storage/wdpost_run.go:101    runPost failed: running window post failed:
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPost
        /home/username/lotus/storage/wdpost_run.go:602
  - Too many open files (os error 24)
    github.com/filecoin-project/filecoin-ffi.GenerateWindowPoSt
        /home/username/lotus/extern/filecoin-ffi/proofs.go:586
    github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper.(*Sealer).GenerateWindowPoSt
        /home/username/lotus/extern/sector-storage/ffiwrapper/verifier_cgo.go:47
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPost
        /home/username/lotus/storage/wdpost_run.go:587
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runGeneratePoST
        /home/username/lotus/storage/wdpost_run.go:99
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).startGeneratePoST.func1
        /home/username/lotus/storage/wdpost_run.go:83
    runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1373
2020-11-04T13:42:00.725Z    ERROR    storageminer    storage/wdpost_run.go:45    Got err %!w(*xerrors.wrapError=&{running window post failed 0xc309986620 {[7536560 20713574 20696062]}}) - TODO handle errors
2020-11-04T13:42:00.725Z    WARN    storageminer    storage/wdpost_changehandler.go:249    Aborted window post Proving (Deadline: &{CurrentEpoch:206364 PeriodStart:206370 Index:0 Open:206370 Close:206430 Challenge:206350 FaultCutoff:206300 WPoStPeriodDeadlines:48 WPoStProvingPeriod:2880 WPoStChallengeWindow:60 WPoStChallengeLookback:20 FaultDeclarationCutoff:70})

Even if you increase the max open files value using the ulimit -n command it doesn't actually guarantee that your Lotus process will use the new setting.

The following steps help you check the max open files setting that Lotus sees and then permanently set a higher value so you never have to worry about it again.

CHECK YOUR CURRENT LOTUS MAX OPEN FILES SETTING

1. Start your Lotus daemon in the usual way.
2. Run the following command ps -ef | grep lotus. This will produce and output screen that looks something like below (you may have more or less information reported depending on how many Lotus processes you are running).

UID        PID  PPID  C STIME TTY          TIME CMD
UserName+  5590  5377 99 16:37 pts/2    00:35:03 lotus daemon
UserName+  6435  5423 99 16:40 pts/8    00:35:21 lotus-miner run

3. You are looking for the PPID value of the Lotus daemon. In the above example that is 5377
4. Run the following command with your Lotus PPID value: cat /proc/5377/limits. This will output a screen that looks like this:

Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             2063251              2063251              processes 
Max open files            1024                 1048576              files     
Max locked memory         16777216             16777216             bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       2063251              2063251              signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

4. In the above example the Max open files value is 1024. This is much too low to run Lotus and you will need to follow the steps in the next section to permanently set a correct limit. If your value is high then you are all set and do not need to read any further. I personally set a maximum value of 1048576 and would recommend using the same.

PERMANENTLY SETTING YOUR LOTUS MAX OPEN FILES SETTING TO A NEW VALUE

1. Run the following command: sudo nano /etc/sysctl.conf
2. Add the following line to the opened file: fs.file-max = 1048576
3. Then run the following command to apply the above changes: sudo sysctl -p
4. Reboot your system and check your new limits have been applied using the same steps above.

[f8-ptrk]

fs.nr_open is worth considering too for /etc/sysctl.conf

setting the values in /etc/sysctl.conf is the only way to make the file limit work correctly with systemd controlled processes!

use ps ax | grep lotus to find the process id of a running lotus/lotus-miner/lotus-worker and then use cat /proc/PROCESS_ID/limits to evaluate that the limits set in the service file via LimitNOFILE=32000000:128000000 are working!