Choosing Linux RAID5 chunk size (part 3) -- Ext4 / CDNjs raw data

Followup

This is the second part on this topic preceeded by:

• Choosing Linux RAID5 chunk size (part 1) -- block-device level benchmarks;
• Choosing Linux RAID5 chunk size (part 2) -- the internet's wisdom;

Summary

The next phase in my RAID5 chunk size benchmark tries to take into account the file-system, namely Ext4.

Therefore in this part I quickly describe the methodology and the raw data. (An in-depth analysis and conclusions would follow in a separate part.)

The methodology...

• mdadm --create a RAID5 md block device from 3x 4 TiB drives (Western Digital Gold (WD WD4002FYYZ)); the chunk size ranges from 4 KiB up to 64 MiB (and all the 2's powers in between);

• use only the first 512 MiB from each drive (thus resulting in an 1 TiB md block device); (the bandwidth for this region, for each individual drive, is almost ~200 MiB/s;)

• mke2fs an Ext4 file-system by using proper -E stride=<...>,strip_width=<...> values;

• git clone (without checkout) the cdnjs.com repository; (the clone is done shallowly, with a depth of 128 commits, which at the time of the execution contained the commit 344797430da2d5...;) (this first clone is not benchmarked, as its performance depends mainly on the network;)

• git repack this first clone; (this operation is mostly CPU intensive;)

• git clone (also without checkout) the first repository to a second one (on the same file-system); (this second clone is benchmarked;)

• git checkout (from the second repository) the commit 344797430da2d5...;

• git status the checked-out work-tree;

• du the checked-out work-tree; (around 158 GiB;)

• find all the files in the checked-out work-tree; (around 4.9 M files;)

• cat all the files (sorted by inode);

• sample ~1/64 (~1.5%) of all the files; (around 80 K files;)

• md5sum the sampled files (sorted by path, then by inode, and then randomly) with a level of parallelism from 1 up to 512 (and all the 2's powers in between);

• before each of these steps are executed, an sync and sysctl vm.drop_caches=3 are issued;

• in total ~4.5 TiB of IO (mostly reads) was generated in ~41 hours;

The raw data (metrics)...

The following metrics are reported:

• elapsed -- the "real-world" ("wall-clock") time for the given operation;
• CPU -- the time that the CPU was actually used (both user-space and kernel-space); (this shouldn't vary much;)
• IO/s -- the real IO bandwidth; (as the ratio between IO and the real time;)
• IO/s (CPU) -- the bandwidth as the ratio between the IO and the CPU used time; (this also shouldn't vary much;)
• IO/s (OS) -- the bandwidth as the ratio between the IO and the OS (kernel-space) used time; (this would indicate any OS overhead;)
• waits -- the context-switches as a consequence (mainly) of IO;
• (there are in total 630 datapoints for about 7 metrics;)

The raw data (graphs)...

File-system, Git, and cat graphs

md5sum summary graphs

md5sum heatmaps

md5sum detailed graphs

To be continued...