[OpenIndiana-discuss] Crossbow performance & bit rot

[James Carlson]

Jason Matthews wrote:
>> Have you tried isolating other components?  Is the behavior the same on
>> all switch ports?  Does it differ if you're connected to a different
>> brand of switch?
> 
> So I have four servers performing this particular role. Three of them,
> slowly and quietly march to their death. One, marches to its death but
> periodically gets a 'reset'  where http response time suddenly drops and the
> "clock" on the march to death starts over.
> 
> Yesterday, the 'resets' started happening on a second box.
> 
> I current switch is a Junper EX4200 configured as a virtual switch with
> reasonable size stack chassis configured as line cards. It seems to work
> just dandy for the other systems connected to it. I haven't tried another
> switch yet, but there are no link related issues being reported on either
> side. I have used this configuration before with e1000 based nics on
> OSOL2009.6 and OI148.

OK.  Still, if it's possible in your hardware environment, I suggest
trying different connections.  There are several reasons for this:

  1. It will help rule out another possible source of trouble, namely
     some kind of compatibility problem between these two systems.  This
     is valid regardless of what else may be attached to that switch.

  2. We've observed the same bad behavior from two different network
     interface cards with two completely different drivers, which makes
     the odds of some kind of low-level kernel driver problem a good bit
     lower.

  3. I don't know of any reports of similar behavior from anyone else,
     which means that either others are ignoring the problem, or that
     there's something that is somehow special about your environment.

>> Is there anything special going on here?  For example, do you have
>> standard IEEE link negotiation turned off -- forcing duplex or link speed?
> 
> Nothing particularly special. The gigE spec says to leave those in autoneg,
> and they are in autoneg. Autoneg appears to work as advertised.

OK; good.

> The servers are connected via a LACP with 2 members in the bundle. I have
> tested with LACP and with straight up Ethernet switching, it makes no
> difference.

That's an interesting bit.  I'm not positive what you mean by "straight
up Ethernet switching."  Does that mean a single physical link from the
server to the switch, or does it mean two links using 802.3ad link
aggregation but with LACP turned off?

If it means the latter, then that sounds like another variable to check
out.  I would configure a plain vanilla Ethernet link -- no 802.3ad --
and see if the problem can be seen there.

(Traditionally, on Solaris, IPMP has been used for link facility
protection.  In some cases, it can detect errors that LACP cannot.  I'm
not saying that LACP is a bad idea at all, but rather that since it has
far fewer miles on it, at least in terms of the Solaris kernel, the ice
is almost certainly thinner below.)

> Here is the switch config, it is pretty plain vanilla.

Yes, except for the 802.3ad link aggregation, it does look pretty plain.

> I havent seen the APIC issues but I have been turning the APIC knobs. I gave
> apic_timer_preferred_mode a whirl. It made no impact. I also tried
> 
> setprop acpi-user-options=0x8 and setprop acpi-user-options=0x2
> individually. These also brought me no joy.

OK.  That eliminates a few possibilities.

> I gave disabling hw acceleration a whirl.
> 
> tx_hcksum_enable = 0;
> rx_hcksum_enable = 0;
> lso_enable = 0;
> 
> disabling the hardware acceleration earned me about 40% jump in cpu
> utilization but no relief on the original problem.

... as does that.

> I haven't tried connecting them to another switch, but I see no link related
> issues. No discards, errors, etc.

I would consider excessive packet delivery delays to be a potential
link-related issue.  I realize that the problem isn't causing the
counters to pop, but that by itself doesn't mean that there's no issue
there.

Another thing to look at would be the kstats for the Ethernet drivers.
It would be interesting to see what changes before and after one of
these "reset" events, to see if the driver is noticing something or if
the problem is actually elsewhere.

> I did notice one interesting item. The server that has had the 'resets'
> where performance returns to normal but then begins the march of death has a
> link state of 'unknown' on its two vnics.

That is interesting.  Unfortunately, I'm not sure what it means.

At this point I think the prime suspect would have to be 802.3ad, though
I can't point at anything definitive.

-- 
James Carlson         42.703N 71.076W         <carlsonj at workingcode.com>