Storage Ruminations

Background:

I’ve been working with a customer whose data set exceeds 100TB, and they were concerned in particular with the NFS write performance for their data ingests, which typically included over 100GB of data. The typical data ingest included files that averaged 35k files with roughly 5000 files per directory. The customer noticed differing performance depending on the number of files per directory. Below we test the following scenarios:

 

Test 1: Baseline Bandwidth Test, 1x 3418MB file

Test 2: 10000x 35KB files, 5000x files per directory

Test 3: 10000x 35KB files, 1000x files per directory

Test 4: 10000x 35KB files, 100x files per directory

Test 5: 10000x 35KB files, 10x files per directory

 

Physical Infrastructure:

Host OS: Ubuntu 2.6.12-9-686 (VM)

Physical Server: SuperMicro X7DVL, 8x CPU @ 1.6 GHz, 8GB memory, 2x GigE NICs

Storage Array: NS20

RAID Configuration: clar_r5_economy NAS pool (R5 8+1, 146GB 10k drives)

Protocol: NFSv3

Transfer Size: v3xfersize parameter left at the 32768 byte default

LAN Switch: Netgear GS116 GigE 16-port switch

Negotiation Settings: Auto-negotiation on both DM and GigE switch

Note: In a future blog post, I will do the same tests but hard coding 1000/full to demonstrate the differences.

 

Test Parameters:

I wrote a perl script that used the dd command to generate various test cases.

Usage: perf.pl BLOCKSIZE BLOCKSPERFILE TOTALFILES FILESPERDIR ITERATIONS

 

The perl script takes the blocksize and blocksperfile parameters and inputs them into the dd command:

dd if=/dev/zero of=mkfile.dat bs=$BLOCKSIZE count=$BLOCKSPERFILE 2>/dev/null

 

So for example, suppose I run:

perf.pl 35840 1 10000 5000 1

This run would generate 1x 35840 byte block files, 10000x files total, 5000x files per directory (hence only 2x directories) and running the whole test 1x iterations.

 

Test 1: Baseline Bandwidth Test, 1x 3418MB file

First, I’d just like to demonstrate that we get reasonable overall bandwidth to the backend storage. Here we’re doing a test generating a single 3125MB file. All tests will be run with 10 iterations to get a reasonable average.

 

Command: ./perf.pl 35840 100000 1 1 10

Running Parameters:

basedir = /mnt/nfs3/testing

dd blocksize = 35840

dd blocksperfile = 100000

total files = 1

files per directory = 1

loop iterations = 10

dd command: dd if=/dev/zero of=mkfile.dat bs=32768 count=100000 2>/dev/null

 

 

The block count column is calculated from: BLOCKSPERFILE * TOTALFILES. Hence Capacity (MB) is calculated: BLOCKSIZE * BLOCKSPERFILE * TOTALFILES / 1024 / 1024.

 

The Transfer Rate (MB/s) is calculated as: Capacity (MB) / Total Seconds. The Transfer Rate (Mb/s) is calculated as Transfer Rate (MB/s) * 8.

 

So you see we got an average of 58.1 second per transfer with roughly 480.4 Mb/s. You’ll also notice that we started at 73 seconds total and slowly stabilized at 50-51 seconds. It’s likely that cache started playing a bigger role in these tests.

 

Test 2: 10000x 35KB files, 5000x files per directory

Command: ./perf.pl 35840 1 10000 5000 10

Running Parameters:

basedir = /mnt/nfs3/testing

dd blocksize = 35840

dd blocksperfile = 1

total files = 10000

files per directory = 5000

loop iterations = 10

 

So you see we got an average of 131.1 seconds per transfer with roughly 20.8 Mb/s. The overall transfer rate drops significantly because we are now writing 10000x individual 35KB files instead of a single 3418MB file.

 

Test 3: 10000x 35KB files, 1000x files per directory

Command: ./perf.pl 35840 1 10000 1000 10

Running Parameters:

basedir = /mnt/nfs3/testing

dd blocksize = 35840

dd blocksperfile = 1

total files = 10000

files per directory = 1000

loop iterations = 10

 

 

So you see that we got roughly equivalent performance as the 5000 files per directory case, 130.6 seconds per transfer with roughly 20.9 Mb/s.

 

Test 4: 10000x 35KB files, 100x files per directory

Command: ./perf.pl 35840 1 10000 100 10

Running Parameters:

basedir = /mnt/nfs3/testing

dd blocksize = 35840

dd blocksperfile = 1

total files = 10000

files per directory = 100

loop iterations = 10

 

Again, it looks like we’re getting roughly equivalent performance as the 5000 and 1000 files per directory cases, 131.1 seconds per transfer with roughly 20.8 Mb/s.

 

Test 5: 10000x 35KB files, 10x files per directory

Command: ./perf.pl 35840 1 10000 10 10

Running Parameters:

basedir = /mnt/nfs3/testing

dd blocksize = 35840

dd blocksperfile = 1

total files = 10000

files per directory = 10

loop iterations = 10

 

Last but not least, here you see that it is slightly slower but only nominally so, 132.8 seconds per transfer with roughly 20.5 Mb/s.

 

Conclusion:

My initial guess was that the differential in times would show a more significant differential, but the tests show that the directory changing causes nominal overhead at most.

 

Two additional tests I’d like to perform are:

1) Comparing Test 1 above with 35840 block sizes to having a 32768 block sizes matching the v3xfersize for NFSv3

2) Comparing the tests conducted in this blog post using 1000/full hard coding versus the auto-negotiation settings used here.