[OpenIndiana-discuss] rsync problems: out of memory

[Frank Van Damme]

2011/7/15 Richard L. Hamilton <rlhamil at smart.net>:
> I've seen that happen to other people too.
>
> If your directory is really "flat" (millions of files in one directory, no subdirectories), you're out of luck unless you rearrange it.  If your directory is not flat, then break up the job into multiple rsyncs that don't overlap.  As I recall, their situation was relatively easy in that there was a level at which they could break it up into workable pieces and only be dealing with subdirectories, not files at that leve; so they could script their way out of their problem.
>
> Compiling the latest rsync source for yourself (for both ends) doesn't hurt.  I had to do that for my Mac, where of all things the rsync version was too old to support extended attributes (strange, since the Mac's HFS+ filesystem has supported them pretty much forever).  But scaling difficulties is a different situation than missing features, so I don't mean to suggest that the newest version will be magically ok even at the level of multimillion file runs; it probably won't be.

It's not too flat. Actually I managed to rsync the same dataset to
Nexenta (well, I also had a failure, but later in the copy and not
consistant) which has an rsync version 3.0.7 while OI has 3.0.6 - I
now compiled 3.0.8 on OI so we'll see if 2 minor versions will make a
difference (quicker to try than to read the changelog ;) ).

-- 
Frank Van Damme
No part of this copyright message may be reproduced, read or seen,
dead or alive or by any means, including but not limited to telepathy
without the benevolence of the author.