[OpenIndiana-discuss] CIFS performance issues

[James Carlson]

Robin Axelsson wrote:
> On 2012-01-24 21:59, James Carlson wrote:
>> Well, unless you get into playing tricks with IP Filter.  And if you do
>> that, then you're in a much deeper world of hurt, at least in terms of
>> performance.
> Here's what the virtualbox manul says about bridged networking:
> 
> "*Bridged networking*:
> 
> This is for more advanced networking needs such as network simulations
> and running servers in a guest. When enabled, VirtualBox connects to one
> of your installed network cards and exchanges network packets directly,
> circumventing your host operating system's network stack.

Note what it says above.  It says nothing about plumbing that interface
for IP on the host operating system.

I'm suggesting that you should _not_ do that, because you (apparently)
want to have separate interfaces for both host and the VirtualBox guests.

If that's not what you want, then I think you should clarify.

Perhaps the right answer is to put the host and guests on different
subnets, so that you have two interfaces with different subnets
configured on the same physical network.  That can have some risks with
respect to multicast, but at least it works far better than duplicating
a subnet.

>> I suspect that the right answer is to plumb only *ONE* of them in the
>> zone, and then use the other by name inside the VM when creating the
>> virtual hub.  That second interface should not be plumbed or configured
>> to use IP inside the regular OpenIndiana environment.  That way, you'll
>> have two independent paths to the network.
> Perhaps the way to do it is to create a dedicated jail/zone for
> VIrtualBox to run in and "plumb the e1000g2" to that zone. I'm a little
> curious as to how this would affect the performance I'm not sure if you
> have to split up the CPU cores etc between zones or if that is taken
> care of as the zones pretty much share the same kernel (and its task
> scheduler).

I'm confused.  If VirtualBox is just going to talk to the physical
interface itself, why is plumbing IP necessary at all?  It shouldn't be
needed.

>> It's possible to have DHCP generate multiple addresses per interface.
>> And it's possible to use IPMP with just one IP address per interface (in
>> fact, you can use it with as little as one IP address per *group*).  And
>> it's possible to configure an IPMP group with some static addresses and
>> some DHCP.
> In order to make DHCP generate more IP addresses I guess I have to
> generate a few (virtual) MAC addresses. Maybe ifconfig hadles this
> internally.

You don't have to work that hard.  You can configure individual IPv4
interfaces to use DHCP, and the system will automatically generate a
random DHCPv4 "ClientID" value for those interfaces.

For example, you can do this:

	ifconfig e1000g0:1 plumb
	ifconfig e1000g0:2 plumb
	ifconfig e1000g0:1 dhcp
	ifconfig e1000g0:2 dhcp

Using the old-style configuration interfaces, you can do "touch
/etc/dhcp.e1000g0:1" to set the system to plumb up and run DHCP on
e1000g0:1.

There's probably a way to do this with ipadm, but I'm too lazy to read
the man page for it.  I suggest it, though, as a worthwhile thing to do
on a lazy Sunday afternoon.

>> But those are just two small ways in which multiple interfaces
>> configured in this manner are a Bad Thing.  A more fundamental issue is
>> that it was just never designed to be used that way, and if you do so,
>> you're a test pilot.
> This was very interesting and insightful. I've always wondered how
> Windows tell the difference between two network connections in a
> machine, now I see that it doesn't. Sometimes this can get corrupted in
> Windows and sever the internet connection completely. If I understand
> correctly, the TCP stack in Windows is borrowed from Sun. I guess this
> is a little OT, it's just a reflection.

No, I don't think they're related in any significant way.  The TCP/IP
stack that Sun acquired long, long ago came from Mentat, and greatly
modified since then.  I suspect that Windows derives from of the BSD
code, but I don't have access to the Windows internals to make sure.

In any event, they all come from the basic constraints of the protocol
design itself, particularly RFC 791, and the "weak ES" model.

>>> I will follow these instructions if I choose to configure IPMP:
>>> http://www.sunsolarisadmin.com/networking/configure-ipmp-load-balancing-resilience-in-sun-solaris/
>>>
>> Wow, that's old.  You might want to dig up something a little more
>> modern.  Before OpenIndiana branched off of OpenSolaris (or before
>> Oracle slammed the door shut), a lot of work went into IPMP to make it
>> much more flexible.
> I'll see if there is something more up-to-date. There are no man entries
> for 'ipmp' in OI and 'apropos' doesn't work for me.

Try "man ifconfig", "man ipmpstat", "man if_mpadm".  Those should be
reasonable starting points.

>> In terms of getting the kernel's IRE entries correct, it doesn't matter
>> so much where the physical wires go.  It matters a whole lot what you do
>> with "ifconfig."
> Ok, but when it is not connected it has no IP address (as it is
> configured over DHCP) that can interfere with the multicast and the IP
> setup. Maybe this is a problem when the address is static.

Or if you allow it to get the address first and then yank the cable.  I
had no idea what your actual test case looked like, so I had to guess.

>>> I looked at the time stamps
>>> of the entries in the /var/adm/messages and they do not match the
>>> freeze-ups by the minute.
>> I assume that refers to the NWAM messages previously reported.  No, I
>> don't think those are the proximate cause of your problem.
>>
> 
> I've been playing around with 'ifconfig <interface> unplumb' (as a
> superuser of course) but it doesn't appear to do anything on e1000g2. As
> a memory refresher, here's the setup:
> 
> e1000g1: 10.40.137.185, DHCP (the computer name is associated with this
> address in the /etc/hosts file)
> e1000g2: 10.40.137.171, DHCP (bridged network of the VM is attached to
> this port)
> rge0: <no IP address>, DHCP (no cable attached)
> 
> No error message comes after that command and when issuing 'ifconfig -a'
> everything looks the same,

It's unclear what that means.  My first guess would be that the command
was somehow misused, but I don't understand how.  Perhaps you've
unplumbed IPv4 and were looking at IPv6 in the "ifconfig -a" output.

Or perhaps you're running NWAM, and that daemon is undoing your work
behind your back.  You probably don't want to use NWAM with a reasonably
complex system configuration like this.

> and the '/sbin route monitor' doesn't yield
> any messages.

That makes no sense.  You should at least get a flurry of messages when
you unplumb a working interface.

Something's deeply amiss with your system, but I don't know what it
might be.

> If I do the same on rge0 it will disappear in the IPv4
> section of the 'ifconfig -a' output but remain in the IPv6 section. I
> also see messages coming out of the route monitor; RTM_IFINFO,
> RTM_DELETE and RTM_DELADDR... as a result of the unplumb command on rge0
> which I think should be expected.

Yes; exactly.

> I can see that e1000g2 is operating in Promiscuous state (Whatever that
> means) which the other ethernet connections are not.

That's driven by applications.  If you have an application (such as,
probably, VirtualBox) that sets the interface into "promiscuous mode"
(receive all unicast messages), then that's what you'll see.

Configuring IEEE 802 bridging will set the flag if that's used.  (I
don't think you're using that here, though.  It's configured using
"dladm create-bridge".)

Snoop/ethereal/wireshark will also put the interface into promiscuous
mode when capturing traffic.

IP itself will not.

> I tried with 'ifconfig e1000g2 down' and the "UP" flag disappeared from
> the port in 'ifconfig -a'.

This is probably confusing here, but those commands probably don't do
what you may think they do.

The "down" and "up" commands control the IP-layer IFF_UP flag.  This is
an indication to IP itself and to higher-level applications that the IP
address itself is administratively available for use.  It says nothing
about whether the interface is usable or connected, but rather that the
administrator wants applications to be able to use the address.

That's quite independent from plumbing -- which is the connection
between IP (or any network layer) and the datalink layer (Ethernet
driver).  And, for that matter, it's also quite independent from the
IFF_RUNNING flag, which is set when the datalink layer tells the network
layer about status changes (such as a physical cable
connection/disconnection).

ifconfig(1M), unfortunately, comes from BSD which blurred the lines
between datalink and network layers.  They're really quite distinct in
OpenIndiana (and Solaris before it), but the confused user interface in
ifconfig sadly leads users astray.

> Once again this only applies to IPv4, IPv6
> remains unaffected. The route monitor yielded messages this time. The IP
> address 10.40.137.171 is still there however and the bridged network
> connection of the VM seems to be unaffected by this command (which would
> be desired if that command really severed the IFP from the host).

It may well be going through the other interface in this case.  I don't
think I have a good grasp on what you're actually trying to accomplish
here.  It seems overly complex ...

> If I try to unplumb it again (ifconfig e1000g2 unplumb) I get the error:
> "ifconfig: cannot unplumb e1000g2: Invalid argument provided"
> which is a bit strange.

Indeed.

> The route monitor yields the message (in
> digested form):
> RTM_LOSING (Kernel Suspects partitioning) <DST,GATEWAY,NETMASK,IFA>
> default 10.40.137.1 default default
> right after this command.

Yikes!  That's a good symptom that the IREs are broken due to the
misconfiguration.  Nothing good can come of that message.

> As a comparison; if I do the same on the rge0 which is already unplumbed
> I get:
> "ifconfig: cannot unplumb rge0: Interface does not exist"
> and no messages on the route monitor.
> 
> Maybe I'm using a bad monkey wrench...

That seems likely.  I think the whole thing was set in doubt when two
separate interfaces were configured up on the same subnet.  Everything
past that point sounds like the experience of a test pilot.  Some of the
control surfaces departed the craft in flight, but, hey, if that weren't
one of the possibilities, then we wouldn't need these brave souls.

-- 
James Carlson         42.703N 71.076W         <carlsonj at workingcode.com>