[OpenIndiana-discuss] rsync problems: out of memory

[Richard L. Hamilton]

I've seen that happen to other people too.

If your directory is really "flat" (millions of files in one directory, no subdirectories), you're out of luck unless you rearrange it.  If your directory is not flat, then break up the job into multiple rsyncs that don't overlap.  As I recall, their situation was relatively easy in that there was a level at which they could break it up into workable pieces and only be dealing with subdirectories, not files at that leve; so they could script their way out of their problem.

Compiling the latest rsync source for yourself (for both ends) doesn't hurt.  I had to do that for my Mac, where of all things the rsync version was too old to support extended attributes (strange, since the Mac's HFS+ filesystem has supported them pretty much forever).  But scaling difficulties is a different situation than missing features, so I don't mean to suggest that the newest version will be magically ok even at the level of multimillion file runs; it probably won't be.

On Jul 15, 2011, at 9:44 AM, Frank Van Damme wrote:

> I have a problem when rsyncing a directory with a *lot* of files in it
> to an openindiana system (it's an 151-dev installation). It failed when
> I issued the command on the sender side (linux box), this is the error
> when you try the command on the receiver side:
> 
> ERROR: out of memory in flist_expand [receiver]
> ERROR: out of memory in flist_expand [generator]
> rsync error: error allocating core memory buffers (code 22) at
> util.c(117) [receiver=3.0.6]
> rsync error: error allocating core memory buffers (code 22) at
> util.c(117) [generator=3.0.6]
> 
> The machine has 2 GB of memory, plus swap... it stops after just over an
> hour.
> 
> -- 
> No part of this copyright message may be reproduced, read or seen,
> dead or alive or by any means, including but not limited to telepathy
> without the benevolence of the author.
> 
> _______________________________________________
> OpenIndiana-discuss mailing list
> OpenIndiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss
> 

-- 
The waitress asked, "Do you want lemon or no lemon with that iced tea?"
Naturally, I said "yes", and then burst out laughing, because there simply
wasn't any other answer in Boolean logic.  She didn't get it, but I got
the lemon, which I wanted anyway.  Later, I realized a quantum computer
could have offered another answer: Schroedinger's Lemon!