[ydl-gen] horrible NFS write performance compared to OS X Server

[Nick Jones]

Does anyone have any advice for doing I/O with small files ( say
30,000 files making up 290MB or an average of 9.8KB per file, dataset
is tarred up on anon ftp at neuron.neurology.uiowa.edu ) over NFS with
Yellow Dog Linux.  One of our reasons for switching to YDL is
filesystem performance.  It performs quite a bit faster than OS X
Server with HFS+CS filesystem when working with our types of small
files (locally).

However, when doing NFS writes it is extremely slow (see benchmarks
below).  So the NFS + Filesystem interaction (or the low congestion 1
hop gige ethernet network, doubtful) is introducing some serious
delays.  Is there another RAID config that works better for NFS or is
this just a problem with NFS?  With NFS I've tried async, various
rwsizes, noac, nocts, and even UDP (dangerous) to see if it was my
configuration (on the server it's exported async).   Perhaps it is yet
the configuration that is causing the issue, but I doubt it.

------ LOCAL TO RAID TRANSFERS -------
--Transfer of 30,000 files (290MB) from YDL 4.1 XServe local disk to
FC attached RAID 5 ext 2
[root at alienraid /]# rm -rf /mnt/right/ppg/test2/*
[root at alienraid /]# date; cp -R /test/extracted_data.dir
/mnt/right/ppg/test2/; date;
Thu Aug 17 13:12:07 CDT 2006
Thu Aug 17 13:12:57 CDT 2006
[root at alienraid /]# rm -rf /mnt/right/ppg/test2/*
[root at alienraid /]# date; cp -R /test /mnt/right/ppg/; date;
Thu Aug 17 15:39:06 CDT 2006
Thu Aug 17 15:39:56 CDT 2006

--Transfer of 30,000 files (290MB) from OS X Server XServe local disk
to FC attached RAID 5 HFS+CS
tcsh% date ; sudo cp -R /test/extracted_data.dir /ppg/test2 ; date ;
Thu Aug 17 13:16:44 CDT 2006
Thu Aug 17 13:28:26 CDT 2006
tcsh% sudo rm -rf /ppg/test2/*
-- double check for sanity
tcsh% date ; sudo cp -R /test/extracted_data.dir /ppg/test2 ; date ;
Thu Aug 17 13:35:56 CDT 2006
Thu Aug 17 13:47:33 CDT 2006

So, we're talking about 50 seconds to write 290MB (5.8 MB/sec) under
YDL local to raid.   Not terrible I guess for RAID 5, or is it?  Would
RAID 3 be better here? (I imagine so, and am willing to sacrifice read
rates, which would still be good)   Does NFS play nicer with RAID 3?



OK, now for NFS, for this I used a subset dataset so I'm not waiting
for 20 minutes on the benchmark.  The data set is roughly 7000 files
(56MB) and is also on anon ftp at neuron.neurology.uiowa.edu.  If you
wanna see something interesting, check these out then check out the
same transfers using ssh protocol (Chris, I may have told you this
wasn't significant, but I must not have tested correctly, please trust
these results however as I am being quite careful so as to give myself
a very clear picture of what is going on).

------ LOCAL CLIENT TO NFS MOUNTED RAID TRANSFERS (Dell P 360 SUSE
10.1 client)-----
--Transfer of 7000 files from SUSE Linux 10.1 local reiser3 to NFS
mounted OS X Server HFS+CS RAID5 volume (osx-ppg is an nfs mount with
options rwsize=8192, vers3)
/# date; cp -r /test/extracted_data.dir /osx-ppg/test2/; date;
Thu Aug 17 15:43:14 CDT 2006
Thu Aug 17 15:44:52 CDT 2006

--Transfer of 7000 files from SUSE Linux 10.1 local reiser3 to NFS
mounted YDL 4.1 RAID5 ext2 volume.
/# date; cp -r /test/extracted_data.dir /ppg/test2/; date;
Thu Aug 17 15:53:49 CDT 2006
Thu Aug 17 16:00:01 CDT 2006

WHOA THERE!!
Looking at the two above benchmarks using NFS, I can only scratch my
head.   Anybody got any ideas??  Why would the write to the YDL server
take longer than the OS X server, if the same operation done locally
is 12 times faster under YDL??   Heres the same thing from a Powermac
G5 dual 2.7 Tiger client.

------ MORE LOCAL CLIENT TO NFS MOUNTED RAID TRANSFERS (PMac G5 10.4
Client) -----
--Transfer of 7000 files from OS X 10.4 local disk to NFS mounted OS X
Server HFS+CS RAID5
tcsh% date; sudo cp -r /test/ /Volumes/yoda.gige.uiowa.edu/test2; date;
Thu Aug 17 16:13:12 CDT 2006
Thu Aug 17 16:15:28 CDT 2006

--Transfer of 7000 files from OS X 10.4 local disk to NFS mounted YDL ext2 RAID5
tcsh% date ; sudo cp -r /test/ /ppg/test2 ; date ;
Thu Aug 17 16:37:56 CDT 2006
Thu Aug 17 16:46:46 CDT 2006

Once again, writing to the YDL server takes way too long for some
reason even from a Mac client, which removes the client as a cause.
I'm using async in exports but....that should make moving small files
faster.


Here are some rsync transfers of the 7000 file dataset that show where
the problem may lie, and that is with NFS' interaction with the ext2
filesystem or the Linux 2.6 kernel in general.   I don't claim to know
because the second I do the problem seems to change :-)  Does anyone
know why this is happening?

----- LOCAL CLIENT TO RAID OVER SSH TRANSFERS ------
--rsync transfer of 7000 files from SUSE 10.1 local reiser3 to OS X
Server HFS+CS RAID5
/# date ; rsync -a /test/extracted_data.dir
setup at yoda:/Volumes/Left/ppg-backup/test2/ ; date ;
Thu Aug 17 16:53:01 CDT 2006
Thu Aug 17 16:54:42 CDT 2006

--rsync transfer of 7000 files from SUSE 10.1 local reiser3 to YDL ext2 RAID 5
/# date ; rsync -a /test/extracted_data.dir
root at alienraid:/mnt/right/ppg/test2/ ; date ;
Thu Aug 17 16:57:09 CDT 2006
Thu Aug 17 16:57:14 CDT 2006


I'm concerned about some ratios I'm seeing here, like the 12:1 ratio
of OSXS:YDL where YDL is 12 times faster on the local to RAID
transfer, and then the 1:4 ratio of OSXS:YDL where YDL is 4 times
SLOWER over NFS from the same linux client (tests using a Mac are
below and show a similar ratio).  But using rsync the results show a
whopping 16:1 ratio similar to the local transfers.

So, what could cause this (very bad) problem with writing small files
over NFS when using YDL 4.1 NFSD?  Are there implications with data
integrity?

Thank you everyone for any comments or clues as to what's happening in
the guts, or any suggestions on how to resolve this?   What about
rsyncd, can it be used to statically "mount" filesystems on a client?
 I'm reachin here.  Not sure about SMB either (can't have ugo
permissions with SMB I don't believe)  FTP?  Not transparantly
available like NFS.

Thanks again.

Nick


-- 
Nick Jones
University of Iowa
Dept of Neurology
Systems Analyst
319-356-0451