Question About Dependency Conflict Between Size Parameter and Target Type in FIO Testing

[sk93mo]

In my tests, I noticed a dependency conflict between the size parameter and the target type (directory/filename), which resulted in different test outcomes. The specific situation is as follows:

When I used the following commands, the test results were consistent and normal:

fio --name=Q32T16_4K_RAN_W **--filename=/dev/nvme0n1** --group_reporting --direct=1 --numjobs=16 --iodepth=32 --thread=1 --ioengine=libaio --rw=randwrite --bs=4k **--size=1g**

fio --name=Q32T16_4K_RAN_W **--directory=/media/mydriver** --group_reporting --direct=1 --numjobs=16 --iodepth=32 --thread=1 --ioengine=libaio --rw=randwrite --bs=4k **--size=64m**

However, when I used the following commands, the test results were abnormal:

fio --name=Q32T16_4K_RAN_W **--filename=/dev/nvme0n1** --group_reporting --direct=1 --numjobs=16 --iodepth=32 --thread=1 --ioengine=libaio --rw=randwrite --bs=4k **--size=64m**

fio --name=Q32T16_4K_RAN_W **--directory=/media/mydriver** --group_reporting --direct=1 --numjobs=16 --iodepth=32 --thread=1 --ioengine=libaio --rw=randwrite --bs=4k **--size=1g**

From my observations, when using a filename, the size parameter must be set to 1g (target capacity), rather than 1024/16=64m; whereas when using a directory, the size parameter must be set to 1024/16=64m, rather than 1g (target capacity).

I would like to understand the reason for this dependency conflict and whether there are any solutions or recommendations. I look forward to your response.

Thank you for your assistance!

[sitsofe]

Hi @sk93mo,

I think the help for the size option explains this:

size=int
[...]
Fio will divide this size between the available files determined by options such as nrfiles, filename, unless filesize is specified by the job.

As you're using numjobs=16 normally each of the 16 jobs is going to create a seperate file of size size but when you set filename all 16 jobs now refer to same single file so a single file of size size will be made. However, in your case filename refers to a block device so no new file needs to be made and I/O will be restricted to the first size region of the existing "file"...

...but be careful! That "single file, multiple jobs" infers that you will be running 16 jobs over the same 1g region which means they can interfere with each other and because you're doing writes that means certain writes may end up being thrown away. For example, job1 writes the first 4k but then job2 immediately writes the same 4k before the job1's 4k gets all the way to non-volatile storage. Due to something in your storage stack doing an optimisation, job1's 4k may be thrown away and success returned because it's already been replaced! To sidestep this issue you may want to arrange for different jobs to write to different regions of the same file by using offset_increment.

[sk93mo]

Hi @sitsofe,

I would like to express my sincere gratitude for your previous response and assistance. I have encountered another issue during my testing process and hope to receive your guidance.

When conducting a Q1T1 4k random read test, I found taht the bandwidth obtanined is always correct when using a directory. However, when I use filename for the test, the results are sometimes correct, but most of the time they are incorect, showing around 200MB/S. This issue is always evident after performing a Q1T1 4K random write, where the subsequent random read performance consistently remains at 200MB/S.

fio --name=Q1T1_4k_SEQ_R --filename=/dev/$device --direct=1 --numjobs=1 --iodepth=1 --thread=1 --ioengine=libaio --rw=randread --bs=4k --size=1GiB --write_bw_log=Q1T1_1M_SEQ_R --log_avg_msec=1000 > ${log_dir_1}/Q1T1_1M_SEQ_R.txt

fio --name=Q1T1_4k_SEQ_R --directory=/media/mydriver1 --direct=1 --numjobs=1 --iodepth=1 --thread=1 --ioengine=libaio --rw=randread --bs=4k --size=1GiB --write_bw_log=Q1T1_1M_SEQ_R --log_avg_msec=1000 > ${log_dir_1}/Q1T1_1M_SEQ_R.txt

[sitsofe]

Hi @sk93mo,
(Note your latest question is a rather issue to the question you asked in the title so it might have been better to ask it in a new discussion but let's continue here for now)

TLDR; I'm guessing what you are seeing is some sort of overhead when you are going through a filesystem but it's hard to know for sure.

At first glace your both of your jobs look fine. However, it's hard to diagnose why you're getting such varied results because you haven't shared information about the environment that you're running these jobs in (see https://github.com/axboe/fio/blob/master/REPORTING-BUGS ). From your previous question we can guess that the first job is going to an NVMe block device and we can see that the second job is going into a filesystem but it would help to know the following:

• What version of fio are you running?
• What kernel are you using?
• What filesystem are you using?
• Is the filesystem directly on an NVMe partition or are there more things in the storage stack between it and the underlying disk (e.g. a docker filesystem, disk encryption etc)?
• Are you using the same underlying block device for both tests?
• Are you sure your disk can't read at 200MiB/s at 4k depth 1?

Also you left out most of the summary that fio produces which contains useful information! If you can attach it as markdown formatted text that can help the reader. You can use iostat to see what is happening while your fio jobs are running and get a second opinion.

Any time you get strange results with fio, I strongly recommend cutting the fio job down to the smallest number of options that still reproduce the problem to help reduce the scope of things to look at. I will offer some suggestions this time so can you check whether removing all of the following still allows the issue to occur:

• --numjobs=1 --iodepth=1 --thread=1
• --ioengine=libaio
• --write_bw_log=Q1T1_1M_SEQ_R --log_avg_msec=1000

My wild guess is that both results that you see are "right" (but as previously state I can't know for sure). Using the following jobs:

./fio --readonly --name=filesytem --filename bigfile --direct=1 --bs=4k --size=30G
sudo ./fio --readonly --name=blockdev --filename /dev/nvme0n1 --bs=4k --direct=1 --offset=1M

my laptop here gives a speed of 53MiB/s for the "filesystem" job and 142MiB/s for the "blockdev" job. My filesystem is ext4 on top of a LUKS encrypted block device and doing the same job against /dev/dm-0 shows a speed of 55MiB/s.

[sitsofe]

Can you add --end_fsync=1 to your jobs and see if the total time/bandwidth changes for your write jobs? I'm also expecting this to result in disk stats for your filesystem showing that I/O was seen. BTW: you only need to include one of the separate filesystem results - they are going to be the same as each other.

[sitsofe]

Also can you rerun your read tests with a size that is at least twice as large as your memory?

[sk93mo]

Can you add --end_fsync=1 to your jobs and see if the total time/bandwidth changes for your write jobs? I'm also expecting this to result in disk stats for your filesystem showing that I/O was seen. BTW: you only need to include one of the separate filesystem results - they are going to be the same as each other.

Here are the results after adding the end_fsync：

The test duration was too short, resulting in insufficient time to record the number of IO operations on the disk. When I set the runtime for Q1T1_1M_SEQ_W to 30 seconds, there were recorded IO operations.

##filename=/dev/nvme0n1p1

Run status group 0 (all jobs):
  WRITE: bw=7566MiB/s (7934MB/s), 7566MiB/s-7566MiB/s (7934MB/s-7934MB/s), io=222GiB (238GB), run=30001-30001msec
Disk stats (read/write):
  nvme0n1: ios=56/452578, merge=0/6, ticks=8/43667, in_queue=43690, util=99.81%

##filename=/dev/nvme0n1p1 --end_fsync=1

Run status group 0 (all jobs):
  WRITE: bw=7770MiB/s (8148MB/s), 7770MiB/s-7770MiB/s (8148MB/s-8148MB/s), io=228GiB (245GB), run=30093-30093msec
Disk stats (read/write):
  nvme0n1: ios=0/467826, merge=0/6, ticks=0/45204, in_queue=45225, util=98.73%

##directory=/media/mydriver1

Run status group 0 (all jobs):
  WRITE: bw=8237MiB/s (8637MB/s), 8237MiB/s-8237MiB/s (8637MB/s-8637MB/s), io=241GiB (259GB), run=30001-30001msec
Disk stats (read/write):
  nvme0n1: ios=0/492838, merge=0/33, ticks=0/41378, in_queue=41396, util=99.70%

##directory=/media/mydriver1 --end_fsync=1

Run status group 0 (all jobs):
  WRITE: bw=8454MiB/s (8865MB/s), 8454MiB/s-8454MiB/s (8865MB/s-8865MB/s), io=248GiB (266GB), run=30003-30003msec
Disk stats (read/write):
  nvme0n1: ios=0/505645, merge=0/33, ticks=0/42986, in_queue=43000, util=99.75%

[sk93mo]

Also can you rerun your read tests with a size that is at least twice as large as your memory?

##filename=/dev/nvme0n1p1

##directory=/media/mydriver

[sitsofe]

I'm still unsure why you're seeing faster write speeds in a filesystem compared to going to the block device but at this point I'm convinced that the issue is not in fio. There's a small possibility you are triggering reads during your block device writes because you are overwriting the start the of the disk where the partition table/filesystem superblock lives but I wouldn't expect the impact to be great if you're working over a large size.