Jump to: navigation, search

Bbcp - fast files transfer over net. New rpm for Fedora. Compare speed with rsync

About the Authors

Other recent contributors

Make this page better by editing it.
Hubbitus
0

Problem of fast transfer big amount of data

In searching fast way to move my files and backups between servers. Awesome rsync good, traditional and really good… but slow. You may find tons of complains and tuning advices like that and this. Problem on the cut you may read in well article on lwn and as conclusion the most problems of rsync are:

  • single process by design on network transfer. Many not so trivial and reliable solutions present.
  • ssh overhead (which can't be turned off but may be used faster ciphers like arcfour) and you may try use even rsyncd (by rsync:// protocol), but it also not so helpful

Candidates

In my many hours searching I found few solutions which may helps if you want fast transfer large amount of data. There many proprietary and cloud solutions, but not so many simple OpenSource utilities. For example you may refer to that overview and that which seems very helpful for me.

Two of that looks promising:

  • bbcp - written by Andy Hanushevsky at SLAC and intended to be fery fast.
  • udr - by Laboratory for Advanced Computing. Is just rsync wrapper for address its some mentioned limitations and use over UDP.

Today I package bbcp for Fedora to try

Tests, comparison, observations

It is not so easy say what faster and how. So, I wrote simple test hosted on github (so, anyone may rerun it).

First attempt

Commit with results

All tests made 3 times to look at min/max/average value and basically exclude random error. No big deviation found.

Full text results

####################################################################################
#### BBCP:  Transfer many files locally (files: 15947, dirs: 1063)
#### Pass #1/3
#### CMD: bbcp -AeOprkz -P 2 'localhost:/home/pasha/temp/Speed.test/bbcp/linux/arch' 'test.transfer' 2>&1 | egrep -v 'bbcp: Creating' --line-buffered
#### Date: 2015-04-07_15:52:38
####################################################################################
Sync
Drop caches
bbcp: Indexing files to be copied...
bbcp: Copying 15947 files and 0 links in 1063 directories.

real	4m10.396s
user	1m0.366s
sys	0m58.577s
####################################################################################
#### RSYNC: Transfer many files locally (files: 15947, dirs: 1063)
#### Pass #1/3
#### CMD: rsync -aHAErlDz 'linux/arch' 'test.transfer'
#### Date: 2015-04-07_16:04:58
####################################################################################
Sync
Drop caches

real	0m14.879s
user	0m7.219s
sys	0m2.768s
####################################################################################
#### RSYNC: Transfer many files REMOTELY(localhost:) (files: 15947, dirs: 1063)
#### Pass #1/3
#### CMD: rsync -aHAErlDz 'localhost:/home/pasha/temp/Speed.test/bbcp/linux/arch' 'test.transfer'
#### Date: 2015-04-07_16:05:45
####################################################################################
Sync
Drop caches

real	0m24.246s
user	0m3.102s
sys	0m2.423s
####################################################################################
#### BBCP:  Transfer 1 big file 1GB locally: -rw-rw-r-- 1 pasha pasha 1073741824 апр  7 11:52 1gb.test
#### Pass #1/3
#### CMD: bbcp -AeOprkz -P 2 'localhost:/home/pasha/temp/Speed.test/bbcp/1gb.test' 'test.transfer' 2>&1 | egrep -v '^bbcp: Creating' --line-buffered
#### Date: 2015-04-07_16:07:01
####################################################################################
Sync
Drop caches
bbcp: Indexing files to be copied...
bbcp: Copying 0 files and 0 links in 0 directories.
bbcp: 150407 16:07:04  7% done; 54.1 MB/s
bbcp: 150407 16:07:06  15% done; 46.4 MB/s
bbcp: 150407 16:07:08  28% done; 53.1 MB/s
bbcp: 150407 16:07:10  36% done; 50.0 MB/s

real	0m25.885s
user	0m3.398s
sys	0m1.709s
####################################################################################
#### RSYNC: Transfer 1 big file 1GB locally: -rw-rw-r-- 1 pasha pasha 1073741824 апр  7 11:52 1gb.test
#### Pass #1/3
#### CMD: rsync -aHAErlDz '1gb.test' 'test.transfer'
#### Date: 2015-04-07_16:08:11
####################################################################################
Sync
Drop caches

real	1m40.064s
user	1m11.830s
sys	0m7.619s

Observations

bbcp in ten times faster rsync on transfer big file and equally slower on set of many small files!

Parameters tuning

It looks so terrible for me. I found many documentations how to tune bbcp and official faq also seems helpfull.

I made second test attempt with different TCP window size and number of simulation TCP streams. Test script enhanced.

Test results (colorized in html) now much bigger, so

Conclusion

bbcp looks very good but optimized for network transfer operations much more than in local filesystem. At least on transfer big files where incredibly wins by speed. Speed working with many small files is really worst. And disabling/enabling md5 checking transfer has no significant changes. At first look made synthetic tests also show it is filesystem bound nor CPU nor network. In some cases it may be compensated by using tar or dump & restore. Also compression and encryption was not tested (it is not so interesting for me), but some people also report troubles.

I plan made more tests, including real transfer between servers, but in any case now I plan use it for transfer at least big files!

And I also hope udr may address some issues of rsync leave well known syntax.

And I hope I help to someone who also want try bbcp.


avatar

Hubbitus

40 months ago
Score 0++

@Luka Rahne, in linkedin comments ( https://www....824975097856 ) point out what tets use compression (-z) for rsync where bbcp do not. Good catch!

I made additional test without it: https://gith...256d237f9424

Report itself in html: https://cdn....8:00:26.html
Add your comment
Pavel-Alexeev (aka Pahan-Hubbitus) site. Mostly Fedora related welcomes all comments. If you do not want to be anonymous, register or log in. It is free.



Share your opinion