[OpenIndiana-discuss] OpenIndiana server networking problem.

[Jim Klimov]

Hello Peter,

I see that this has already been a long thread, and you've tried to
reconfigure and to explain some things in various posts. Can you
outline the logical configuration of what you're trying to build
and what you have as of now? Namely, it is indeed suspicious that
plugging in your OI server breaks the networking for other hosts
in the segment; do you still have this problem?

One way to achieve that inadvertently is to have this server announce
itself as a best router (better than your "modem" in priority) and to
not have any routes that it can use in practice, and with other nodes
being at the same time clients of a dynamic routing protocol (listening
to your server's RIPv2/OSPF/whatever announcements and trusting them).

Another way is to have your OI host register on the net with the same
IP address as your default gateway (192.168.0.1) and thus intercept
communications between the other hosts and the real router, and lead
those packets into nowhere. A similar in practice actively malicious
way would also involve "ARP spoofing" or "ARP poisoning", where your
server sends out fake ARP replies to (unsolicited) queries for IP to
MAC address resolution, thus registering itself as the router's IP
address number in the attacked client hosts. If those hosts don't
use the static ARP, such entries should time out after 5 minutes.

Both of this might be detectable without special tools by having the
network connectivity for other hosts repair itself (as in "linux box
can see internet through the modem") after a timeout specified by the
particular "broken" protocol, typically ranging from 30 to 300 seconds
after you unplug the offending (OI) host. Or after the router's and/or
attacked host's reboot to clear those caches, as you do :)

Of course, there are tools like sniffers (snoop, tcpdump) and simple
networking inspection (arp, netstat) to verify the particular fine
points - though email is not a convenient way to teach a course on
networking and write a book on it. At least, you know what to google
for now ;)

In the context of your problems, it might be important to turn off
name resolution for the tests (with sniffers and arp), which would
likely time out anyway, and have the tools display numeric IP addresses
instantly (typically this involves a -n flag, maybe -r for snoop).
Also if the earlier suggested "traceroute -n" fails for you, try
the "traceroute -nI" to use ICMP (ping-like queries) instead of
a default UDP-trace request protocol.

My understanding of your desired setup is thus:

(internet)
      |
    (ISP)
      |
(ISP's range of public IP addresses provided to you, 5 of 8 usable and
currently configured on the external interface of their cisco appliance)
      |
(cisco box you call "modem", internal interface is 192.168.0.1)
      |
(routert3 zone with vnic0 = 192.168.0.3)+(OI 192.168.0.4)+(linux)
      |
(etherstub with local zones in subnet 192.168.3.0/24, including vnic1)

This data path, if correctly assumed, requires a number of routers
and NAT translations and/or firewalls; optionally it can also be
simplified. As James wrote, for example, if you do have the 5-of-8
public ("routable", not 192.168.*) IP addresses from the provider,
you could configure OI to do the routing/translation instead of the
Cisco box - which would be demoted in functionality to a switch,
connecting your OI box with the ISP's networking infrastructure.
Support for the public IP addresses would be directly configured
(and firewalled) on the hosts connected to the cisco, such as the
router zone, maybe - not necessarily - the global zone, the linux
box and so on.

Otherwise, you have one NAT translation configurable on the Cisco,
which converts packets between public addresses and private ones
(192.168.*). Generally NAT can do many-to-one translations (by
multiplexing port numbers where they can be varied), but from your
mail and my practice with some ISPs it seems reasonable to assume
that they preconfigured about five one-to-one translations of all
ports (unchanged and unfirewalled) from a certain public IP address
to a certain private address - the 192.168.0.* numbers you must use.

The second translation you do happens in the routert3 zone, which
is a many-to-one port multiplexing. It takes packets incoming to
192.168.0.3 particular ports and outputs them via vnic1 to certain
destination hosts and ports on the 192.168.3.0/24 subnet (on the
etherstub with other local zones). For packets incoming to other
ports (undefined in NAT "rdr" mappings), the NAT subsystem tries
to direct them to listening daemons (like, say, an apache running
in the routert3 zone), or translate them into conversations earlier
started from hosts in 192.168.3.* zones to the internet, or drops
them if it can't determine a target.

Likewise, if your ipnat.conf includes "map" rules, then your hosts
from specified internal interfaces (vnic1) would be translated as
accesses to the internet from the external interface (vnic0) and
an IP address registered on it (192.168.0.3).

Note that attempts from any outside hosts to access 192.168.3.0
addresses directly should fail, and the NAT host (routert3) should
not be specially announced as a route to this subnet - it is hidden
by the address-translation anyway.

Furthermore, I believe that all of your OI zones use separate
IP stacks - the local zones, including routert3, would be
"ip-type=exclusive", and the global zone would be the only user
of a default shared IP stack.

The routing table you showed before for the global zone seems
reasonable (including that last line which caused a separate
discussion). The "netstat -nr" output for the routert3 zone
would likely include four lines, with two such subnet routes
to access the two subnets that it is directly connected to.

Just to clarify: routing tables are built in baseaddress/netmask
form, with both components being 32-bit numbers for IPv4.
The netmask is a contiguous string of ones at the start (and the
amount of such ones is the "prefix length" in short recording,
such as "/24"), followed by only zeroes at the end - a string
like 111...10...000 in binary representation. The netmask can be
ANDed with an IP address to calculate its subnet's "base address"
(i.e. "192.168.3.0"). A XORed netmask ANDed with an IP address
number chops off its "local" IP address in the subnet (i.e. the
".3" for your routert3). These are simple operations which can be
used quickly to compare and match destination hosts to routing
table entries (known routes to some subnets).

Each routing table entry includes an explicitly known network's
base address, a netmask (in four-number form or as prefix length),
and the IP address which is the known gateway to this network,
and optionally the source interface to use for outgoing packets.
Several routes can match the target IP address (i.e. the default
route with mask length 0 matches anything), so the matching route
with the longest prefix length - the "specific route" - is selected
to send a packet (or one of the routes with the same longest lengths -
usually done for failover routers).

The entry which puzzled you (aka "third line") was the most-specific
route from your system's kernel to the 192.168.0 subnet, and specified
that the 192.168.0.4 IP address on the bnx0 interface should be used
as origin for packets destined to hosts on 192.168.0.0/24 (somewhat
unfortunately, Solaris "netstat -nr" does not show netmask lengths
for more clarity). For localhost (127.0.0.0/8) there is a route via
loopback interface in the kernel. Other networks are reachable with
the default handling by some host holding the 192.168.0.1 address.

The routing table itself may be maintained by some daemon (i.e.
listening for announcements from other routers) or is entered
statically via /etc/defaultrouter file (for default route) and/or
/etc/inet/static_routes (for subnet routes). This is implemented
by the "route add/route delete" command which you can use manually.
Also one can see /etc/gateways (typically initializes the dynamic 
routing daemon, though the latter is not required to obsolete entries
without seen announcements from other routers nor is required to
announce anything itself). Addresses attached to active interfaces
are managed into the routing table by, AFAIK, the ifconfig command
which "up/down"'s the interfaces; these typically are most specific
routes to their subnets.

HTH,
//Jim Klimov