NS20 NFS Performance

Background:

I’ve been working with a customer whose data set exceeds 100TB, and they were concerned in particular with the NFS write performance for their data ingests, which typically included over 100GB of data. The typical data ingest included files that averaged 35k files with roughly 5000 files per directory. The customer noticed differing performance depending on the number of files per directory. Below we test the following scenarios:

 

Test 1: Baseline Bandwidth Test, 1x 3418MB file

Test 2: 10000x 35KB files, 5000x files per directory

Test 3: 10000x 35KB files, 1000x files per directory

Test 4: 10000x 35KB files, 100x files per directory

Test 5: 10000x 35KB files, 10x files per directory

 

Physical Infrastructure:

Host OS: Ubuntu 2.6.12-9-686 (VM)

Physical Server: SuperMicro X7DVL, 8x CPU @ 1.6 GHz, 8GB memory, 2x GigE NICs

Storage Array: NS20

RAID Configuration: clar_r5_economy NAS pool (R5 8+1, 146GB 10k drives)

Protocol: NFSv3

Transfer Size: v3xfersize parameter left at the 32768 byte default

LAN Switch: Netgear GS116 GigE 16-port switch

Negotiation Settings: Auto-negotiation on both DM and GigE switch

Note: In a future blog post, I will do the same tests but hard coding 1000/full to demonstrate the differences.

 

Test Parameters:

I wrote a perl script that used the dd command to generate various test cases.

Usage: perf.pl BLOCKSIZE BLOCKSPERFILE TOTALFILES FILESPERDIR ITERATIONS

 

The perl script takes the blocksize and blocksperfile parameters and inputs them into the dd command:

dd if=/dev/zero of=mkfile.dat bs=$BLOCKSIZE count=$BLOCKSPERFILE 2>/dev/null

 

So for example, suppose I run:

perf.pl 35840 1 10000 5000 1

This run would generate 1x 35840 byte block files, 10000x files total, 5000x files per directory (hence only 2x directories) and running the whole test 1x iterations.

 

Test 1: Baseline Bandwidth Test, 1x 3418MB file

First, I’d just like to demonstrate that we get reasonable overall bandwidth to the backend storage. Here we’re doing a test generating a single 3125MB file. All tests will be run with 10 iterations to get a reasonable average.

 

Command: ./perf.pl 35840 100000 1 1 10

Running Parameters:

basedir = /mnt/nfs3/testing

dd blocksize = 35840

dd blocksperfile = 100000

total files = 1

files per directory = 1

loop iterations = 10

dd command: dd if=/dev/zero of=mkfile.dat bs=32768 count=100000 2>/dev/null

 

Block Size	Block Count	Capacity (MB)	Iteration	Total Seconds	Transfer Rate (MB/s)	Transfer Rate (Mb/s)
35840	100000	3417.96875	1	73	46.82148973	374.5719178
35840	100000	3417.96875	2	70	48.828125	390.625
35840	100000	3417.96875	3	67	51.01445896	408.1156716
35840	100000	3417.96875	4	63	54.25347222	434.0277778
35840	100000	3417.96875	5	52	65.73016827	525.8413462
35840	100000	3417.96875	6	53	64.48997642	515.9198113
35840	100000	3417.96875	7	51	67.0189951	536.1519608
35840	100000	3417.96875	8	51	67.0189951	536.1519608
35840	100000	3417.96875	9	50	68.359375	546.875
35840	100000	3417.96875	10	51	67.0189951	536.1519608
			Average	58.1	60.05540509	480.4432407

 

The block count column is calculated from: BLOCKSPERFILE * TOTALFILES. Hence Capacity (MB) is calculated: BLOCKSIZE * BLOCKSPERFILE * TOTALFILES / 1024 / 1024.

 

The Transfer Rate (MB/s) is calculated as: Capacity (MB) / Total Seconds. The Transfer Rate (Mb/s) is calculated as Transfer Rate (MB/s) * 8.

 

So you see we got an average of 58.1 second per transfer with roughly 480.4 Mb/s. You’ll also notice that we started at 73 seconds total and slowly stabilized at 50-51 seconds. It’s likely that cache started playing a bigger role in these tests.

 

Test 2: 10000x 35KB files, 5000x files per directory

Command: ./perf.pl 35840 1 10000 5000 10

Running Parameters:

basedir = /mnt/nfs3/testing

dd blocksize = 35840

dd blocksperfile = 1

total files = 10000

files per directory = 5000

loop iterations = 10

 

Block Size	Block Count	Capacity (MB)	Iteration	Total Seconds	Transfer Rate (MB/s)	Transfer Rate (Mb/s)
35840	10000	341.797	1	131	2.60913645	20.8730916
35840	10000	341.797	2	130	2.629206731	21.03365385
35840	10000	341.797	3	132	2.589370265	20.71496212
35840	10000	341.797	4	130	2.629206731	21.03365385
35840	10000	341.797	5	132	2.589370265	20.71496212
35840	10000	341.797	6	130	2.629206731	21.03365385
35840	10000	341.797	7	132	2.589370265	20.71496212
35840	10000	341.797	8	131	2.60913645	20.8730916
35840	10000	341.797	9	132	2.589370265	20.71496212
35840	10000	341.797	10	131	2.60913645	20.8730916
			Average	131.1	2.60725106	20.85800848

So you see we got an average of 131.1 seconds per transfer with roughly 20.8 Mb/s. The overall transfer rate drops significantly because we are now writing 10000x individual 35KB files instead of a single 3418MB file.

 

Test 3: 10000x 35KB files, 1000x files per directory

Command: ./perf.pl 35840 1 10000 1000 10

Running Parameters:

basedir = /mnt/nfs3/testing

dd blocksize = 35840

dd blocksperfile = 1

total files = 10000

files per directory = 1000

loop iterations = 10

 

Block Size	Block Count	Capacity (MB)	Iteration	Total Seconds	Transfer Rate (MB/s)	Transfer Rate (Mb/s)
35840	10000	341.797	1	130	2.629206731	21.03365385
35840	10000	341.797	2	131	2.60913645	20.8730916
35840	10000	341.797	3	130	2.629206731	21.03365385
35840	10000	341.797	4	131	2.60913645	20.8730916
35840	10000	341.797	5	132	2.589370265	20.71496212
35840	10000	341.797	6	131	2.60913645	20.8730916
35840	10000	341.797	7	130	2.629206731	21.03365385
35840	10000	341.797	8	130	2.629206731	21.03365385
35840	10000	341.797	9	131	2.60913645	20.8730916
35840	10000	341.797	10	130	2.629206731	21.03365385
			Average	130.6	2.617194972	20.93755978

 

So you see that we got roughly equivalent performance as the 5000 files per directory case, 130.6 seconds per transfer with roughly 20.9 Mb/s.

 

Test 4: 10000x 35KB files, 100x files per directory

Command: ./perf.pl 35840 1 10000 100 10

Running Parameters:

basedir = /mnt/nfs3/testing

dd blocksize = 35840

dd blocksperfile = 1

total files = 10000

files per directory = 100

loop iterations = 10

 

Block Size	Block Count	Capacity (MB)	Iteration	Total Seconds	Transfer Rate (MB/s)	Transfer Rate (Mb/s)
35840	10000	341.797	1	130	2.629206731	21.03365385
35840	10000	341.797	2	132	2.589370265	20.71496212
35840	10000	341.797	3	130	2.629206731	21.03365385
35840	10000	341.797	4	132	2.589370265	20.71496212
35840	10000	341.797	5	130	2.629206731	21.03365385
35840	10000	341.797	6	132	2.589370265	20.71496212
35840	10000	341.797	7	131	2.60913645	20.8730916
35840	10000	341.797	8	132	2.589370265	20.71496212
35840	10000	341.797	9	130	2.629206731	21.03365385
35840	10000	341.797	10	132	2.589370265	20.71496212
			Average	131.1	2.60728147	20.85825176

Again, it looks like we’re getting roughly equivalent performance as the 5000 and 1000 files per directory cases, 131.1 seconds per transfer with roughly 20.8 Mb/s.

 

Test 5: 10000x 35KB files, 10x files per directory

Command: ./perf.pl 35840 1 10000 10 10

Running Parameters:

basedir = /mnt/nfs3/testing

dd blocksize = 35840

dd blocksperfile = 1

total files = 10000

files per directory = 10

loop iterations = 10

 

Block Size	Block Count	Capacity (MB)	Iteration	Total Seconds	Transfer Rate (MB/s)	Transfer Rate (Mb/s)
35840	10000	341.797	1	133	2.569901316	20.55921053
35840	10000	341.797	2	132	2.589370265	20.71496212
35840	10000	341.797	3	134	2.550722948	20.40578358
35840	10000	341.797	4	132	2.589370265	20.71496212
35840	10000	341.797	5	134	2.550722948	20.40578358
35840	10000	341.797	6	132	2.589370265	20.71496212
35840	10000	341.797	7	135	2.531828704	20.25462963
35840	10000	341.797	8	132	2.589370265	20.71496212
35840	10000	341.797	9	133	2.569901316	20.55921053
35840	10000	341.797	10	131	2.60913645	20.8730916
			Average	132.8	2.573969474	20.59175579

Last but not least, here you see that it is slightly slower but only nominally so, 132.8 seconds per transfer with roughly 20.5 Mb/s.

 

Conclusion:

My initial guess was that the differential in times would show a more significant differential, but the tests show that the directory changing causes nominal overhead at most.

 

Two additional tests I’d like to perform are:

1) Comparing Test 1 above with 35840 block sizes to having a 32768 block sizes matching the v3xfersize for NFSv3

2) Comparing the tests conducted in this blog post using 1000/full hard coding versus the auto-negotiation settings used here.