[OpenIndiana-discuss] Crossbow performance & bit rot

[James Carlson]

On 12/31/2011 1:27 AM, Jason Matthews wrote:
> As a follow up, I have determined that crossbow is not at fault for the
> degradation of the network stack performance. As it turns out, the shared
> stack fails much faster than multiple zones with exclusive-ip.

For what it's worth, although VNICs are a feature of Crossbow, there's
no actual way to turn Crossbow on or off.  It's a design and locking
architectural change to the stack.  You can't really exclude it except
by returning to an older (Solaris 10) stack and trying the test there.

> The symptoms are slow out bound connection times, high ping times between
> systems on a local subnet  (1ms to 500ms on gigE), and general network
> flakiness. 

I don't know what is causing the problem you're seeing, but I do have
some guesses.

Assuming it affects all packets, and not just those of a particular
type, the slow packet delivery sounds a bit like either a driver problem
or a problem with the (Crossbow-introduced) "polling" mode that drivers
can use during periods of high traffic.

Another fairly likely possibility would be the hardware offload features
that some drivers can use -- checksum and LSO.  If the driver you're
using can turn off these features, I strongly recommend trying that.
I'd quickly run out of appendages before I could count the number of
times that "hardware acceleration" has thrown performance in the toilet.

I realize this is a production system, and that really hurts.  However,
since nobody else seems to have the problem, which makes solutions out
of the blue less likely, it would probably be instructive to do some
investigation.  I suggest:

 - using dtrace to find out what the code paths are when traffic is
working fine and when it's slow; the difference may reveal something,

 - installing a network interface card that uses a different driver
(e.g., if you have Intel, install Broadcom) and see if that interface
behaves differently.

Note that you'll probably need to know a fair bit about the driver to
use dtrace effectively.

-- 
James Carlson         42.703N 71.076W         <carlsonj at workingcode.com>