[OpenIndiana-discuss] Crossbow performance & bit rot

[James Carlson]

Jason Matthews wrote:
> 
> So I have determined the trigger event for the "resets" in the response
> times. It appears that there is some sort of memory leak in the kernel. When
> memory utilization gets very high, I am not sure how high but it is around a
> few hundred bytes on the freelists with zfs data near zero, the reset cycles
> start on the vnics.

If it really is a memory leak (as opposed to just a lot of memory
legitimately in use), then it should be possible (albeit slightly
painful) to find out where it's going.  Putting "set kmem_flags=0xf" in
/etc/system will cause the kernel memory allocator to keep track of
allocations.  You can then panic the system (as in "reboot -d"), get the
kernel dump up in mdb, and use "::findleaks" to discover what's leaking
memory.

It's possible that this is related to zero-copy issues that cause dblks
to get held down for a long time.  I think there are tunables that
disable the zero-copy optimizations, but I don't remember what they are.

At least with some of the bugs I've seen historically in this area, the
toxic behavior is to have some application that has a socket or DLPI
stream open, but that doesn't bother reading from the socket.  A large
number of buffers can accumulate on an unread stream, and if enough of
them get stuck there, you can end up with a pretty sick system.

It's also possible that the causation runs the other way -- that
whatever causes the problem to happen (a packet storm of some sort?)
also happens to cause a large amount of kernel memory to be allocated.

> I suspect excessive memory allocation and eventual starvation have a lot to
> do with the weird symptoms. I wonder if the memory leak is an independent
> process from the network issue or if they are directly related in some way. 

It's possible that they're related.  It sounds to me like this is a
pretty good clue.

-- 
James Carlson         42.703N 71.076W         <carlsonj at workingcode.com>