NS20 NFS Performance

Background:

I’ve been working with a customer whose data set exceeds 100TB, and they were concerned in particular with the NFS write performance for their data ingests, which typically included over 100GB of data. The typical data ingest included files that averaged 35k files with roughly 5000 files per directory. The customer noticed differing performance depending on the number of files per directory. Below we test the following scenarios:

 

Test 1: Baseline Bandwidth Test, 1x 3418MB file

Test 2: 10000x 35KB files, 5000x files per directory

Test 3: 10000x 35KB files, 1000x files per directory

Test 4: 10000x 35KB files, 100x files per directory

Test 5: 10000x 35KB files, 10x files per directory

 

Physical Infrastructure:

Host OS: Ubuntu 2.6.12-9-686 (VM)

Physical Server: SuperMicro X7DVL, 8x CPU @ 1.6 GHz, 8GB memory, 2x GigE NICs

Storage Array: NS20

RAID Configuration: clar_r5_economy NAS pool (R5 8+1, 146GB 10k drives)

Protocol: NFSv3

Transfer Size: v3xfersize parameter left at the 32768 byte default

LAN Switch: Netgear GS116 GigE 16-port switch

Negotiation Settings: Auto-negotiation on both DM and GigE switch

Note: In a future blog post, I will do the same tests but hard coding 1000/full to demonstrate the differences.

 

Test Parameters:

I wrote a perl script that used the dd command to generate various test cases.

Usage: perf.pl BLOCKSIZE BLOCKSPERFILE TOTALFILES FILESPERDIR ITERATIONS

 

The perl script takes the blocksize and blocksperfile parameters and inputs them into the dd command:

dd if=/dev/zero of=mkfile.dat bs=$BLOCKSIZE count=$BLOCKSPERFILE 2>/dev/null

 

So for example, suppose I run:

perf.pl 35840 1 10000 5000 1

This run would generate 1x 35840 byte block files, 10000x files total, 5000x files per directory (hence only 2x directories) and running the whole test 1x iterations.

 

Test 1: Baseline Bandwidth Test, 1x 3418MB file

First, I’d just like to demonstrate that we get reasonable overall bandwidth to the backend storage. Here we’re doing a test generating a single 3125MB file. All tests will be run with 10 iterations to get a reasonable average.

 

Command: ./perf.pl 35840 100000 1 1 10

Running Parameters:

basedir = /mnt/nfs3/testing

dd blocksize = 35840

dd blocksperfile = 100000

total files = 1

files per directory = 1

loop iterations = 10

dd command: dd if=/dev/zero of=mkfile.dat bs=32768 count=100000 2>/dev/null

 

Block Size

Block Count

Capacity (MB)

Iteration

Total Seconds

Transfer Rate (MB/s)

Transfer Rate (Mb/s)

35840

100000

3417.96875

1

73

46.82148973

374.5719178

35840

100000

3417.96875

2

70

48.828125

390.625

35840

100000

3417.96875

3

67

51.01445896

408.1156716

35840

100000

3417.96875

4

63

54.25347222

434.0277778

35840

100000

3417.96875

5

52

65.73016827

525.8413462

35840

100000

3417.96875

6

53

64.48997642

515.9198113

35840

100000

3417.96875

7

51

67.0189951

536.1519608

35840

100000

3417.96875

8

51

67.0189951

536.1519608

35840

100000

3417.96875

9

50

68.359375

546.875

35840

100000

3417.96875

10

51

67.0189951

536.1519608

 

 

 

Average

58.1

60.05540509

480.4432407

 

The block count column is calculated from: BLOCKSPERFILE * TOTALFILES. Hence Capacity (MB) is calculated: BLOCKSIZE * BLOCKSPERFILE * TOTALFILES / 1024 / 1024.

 

The Transfer Rate (MB/s) is calculated as: Capacity (MB) / Total Seconds. The Transfer Rate (Mb/s) is calculated as Transfer Rate (MB/s) * 8.

 

So you see we got an average of 58.1 second per transfer with roughly 480.4 Mb/s. You’ll also notice that we started at 73 seconds total and slowly stabilized at 50-51 seconds. It’s likely that cache started playing a bigger role in these tests.

 

Test 2: 10000x 35KB files, 5000x files per directory

Command: ./perf.pl 35840 1 10000 5000 10

Running Parameters:

basedir = /mnt/nfs3/testing

dd blocksize = 35840

dd blocksperfile = 1

total files = 10000

files per directory = 5000

loop iterations = 10

 

Block Size

Block Count

Capacity (MB)

Iteration

Total Seconds

Transfer Rate (MB/s)

Transfer Rate (Mb/s)

35840

10000

341.797

1

131

2.60913645

20.8730916

35840

10000

341.797

2

130

2.629206731

21.03365385

35840

10000

341.797

3

132

2.589370265

20.71496212

35840

10000

341.797

4

130

2.629206731

21.03365385

35840

10000

341.797

5

132

2.589370265

20.71496212

35840

10000

341.797

6

130

2.629206731

21.03365385

35840

10000

341.797

7

132

2.589370265

20.71496212

35840

10000

341.797

8

131

2.60913645

20.8730916

35840

10000

341.797

9

132

2.589370265

20.71496212

35840

10000

341.797

10

131

2.60913645

20.8730916

 

 

 

Average

131.1

2.60725106

20.85800848

So you see we got an average of 131.1 seconds per transfer with roughly 20.8 Mb/s. The overall transfer rate drops significantly because we are now writing 10000x individual 35KB files instead of a single 3418MB file.

 

Test 3: 10000x 35KB files, 1000x files per directory

Command: ./perf.pl 35840 1 10000 1000 10

Running Parameters:

basedir = /mnt/nfs3/testing

dd blocksize = 35840

dd blocksperfile = 1

total files = 10000

files per directory = 1000

loop iterations = 10

 

Block Size

Block Count

Capacity (MB)

Iteration

Total Seconds

Transfer Rate (MB/s)

Transfer Rate (Mb/s)

35840

10000

341.797

1

130

2.629206731

21.03365385

35840

10000

341.797

2

131

2.60913645

20.8730916

35840

10000

341.797

3

130

2.629206731

21.03365385

35840

10000

341.797

4

131

2.60913645

20.8730916

35840

10000

341.797

5

132

2.589370265

20.71496212

35840

10000

341.797

6

131

2.60913645

20.8730916

35840

10000

341.797

7

130

2.629206731

21.03365385

35840

10000

341.797

8

130

2.629206731

21.03365385

35840

10000

341.797

9

131

2.60913645

20.8730916

35840

10000

341.797

10

130

2.629206731

21.03365385

 

 

 

Average

130.6

2.617194972

20.93755978

 

So you see that we got roughly equivalent performance as the 5000 files per directory case, 130.6 seconds per transfer with roughly 20.9 Mb/s.

 

Test 4: 10000x 35KB files, 100x files per directory

Command: ./perf.pl 35840 1 10000 100 10

Running Parameters:

basedir = /mnt/nfs3/testing

dd blocksize = 35840

dd blocksperfile = 1

total files = 10000

files per directory = 100

loop iterations = 10

 

Block Size

Block Count

Capacity (MB)

Iteration

Total Seconds

Transfer Rate (MB/s)

Transfer Rate (Mb/s)

35840

10000

341.797

1

130

2.629206731

21.03365385

35840

10000

341.797

2

132

2.589370265

20.71496212

35840

10000

341.797

3

130

2.629206731

21.03365385

35840

10000

341.797

4

132

2.589370265

20.71496212

35840

10000

341.797

5

130

2.629206731

21.03365385

35840

10000

341.797

6

132

2.589370265

20.71496212

35840

10000

341.797

7

131

2.60913645

20.8730916

35840

10000

341.797

8

132

2.589370265

20.71496212

35840

10000

341.797

9

130

2.629206731

21.03365385

35840

10000

341.797

10

132

2.589370265

20.71496212

 

 

 

Average

131.1

2.60728147

20.85825176

Again, it looks like we’re getting roughly equivalent performance as the 5000 and 1000 files per directory cases, 131.1 seconds per transfer with roughly 20.8 Mb/s.

 

Test 5: 10000x 35KB files, 10x files per directory

Command: ./perf.pl 35840 1 10000 10 10

Running Parameters:

basedir = /mnt/nfs3/testing

dd blocksize = 35840

dd blocksperfile = 1

total files = 10000

files per directory = 10

loop iterations = 10

 

Block Size

Block Count

Capacity (MB)

Iteration

Total Seconds

Transfer Rate (MB/s)

Transfer Rate (Mb/s)

35840

10000

341.797

1

133

2.569901316

20.55921053

35840

10000

341.797

2

132

2.589370265

20.71496212

35840

10000

341.797

3

134

2.550722948

20.40578358

35840

10000

341.797

4

132

2.589370265

20.71496212

35840

10000

341.797

5

134

2.550722948

20.40578358

35840

10000

341.797

6

132

2.589370265

20.71496212

35840

10000

341.797

7

135

2.531828704

20.25462963

35840

10000

341.797

8

132

2.589370265

20.71496212

35840

10000

341.797

9

133

2.569901316

20.55921053

35840

10000

341.797

10

131

2.60913645

20.8730916

 

 

 

Average

132.8

2.573969474

20.59175579

Last but not least, here you see that it is slightly slower but only nominally so, 132.8 seconds per transfer with roughly 20.5 Mb/s.

 

Conclusion:

My initial guess was that the differential in times would show a more significant differential, but the tests show that the directory changing causes nominal overhead at most.

 

Two additional tests I’d like to perform are:

1) Comparing Test 1 above with 35840 block sizes to having a 32768 block sizes matching the v3xfersize for NFSv3

2) Comparing the tests conducted in this blog post using 1000/full hard coding versus the auto-negotiation settings used here.

Advertisements

One thought on “NS20 NFS Performance

  1. Pingback: NS20 NFS Performance, Part Deux « Storage Ruminations

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s