[OpenIndiana-discuss] Migrating root zpool from old disks to new one. only one system available

[J. V.]

I did check the ashift after the upgrade and both 3TB HDs report ashift=12.

I am actually using an HP N35L (1.3 Ghz vs. 1.5 GHz in the N40L) and I have
8 GB of non-ECC DDR3 so I could not answer the memory question.

These little machines are really great (If they came with SATA 3, they
would be the ultimate small server):
I do have an N40L with ESXi 5 running my firewall/router (Untangle 9.2) and
Windows Home Server 2011 (For the Media Server, Backup and WSUS services)..

The OpenIndiana server has 6 HDs (loaded modified BIOS so the extra
internal and external SATA ports run at full speeds):
4 x 3TB Seagate ST3000DM001 (Newer Seagates: Fastest 7200 rpm HD right now
... ~140 MB on SATA 2, ~160 on SATA 3).
2 x OCZ 60 GB SSD (Agility 3). Mounted where the CD-ROM would go with a Vantec
5 1/4, 2 x 2.5 in. (Used an eSATA to SATA cable for one of the SSDs).

Config:
Partitioned the SSDs in 3 slices:
rpool (16GB), zil1 (4GB), arc1 (~40GB)
rpool (16GB), zil2 (4GB), arc2 (~40GB)

rpool = 2 x 16GB mirrored
Storage pool 1 = 2 x 3TB mirrored + zil1 + arc1
Storage pool 2 = 2 x 3TB mirrored + zil2 + arc2

The mirrored rpool saved my butt last month when I was playing around and
over-wrote the boot rpool (gets confusing between slices and partitions
with my SSD config). Took me whole 20 min. to fix it:
Exchange SSD places, boot from good rpool, wait 5 min. for resilver,
re-install grub in damaged rpool, poweroff, exchange SSDs again).

No other dual cpu machine that I have uses less power:
The whole server only uses 23, 24 Watts when idle (2 SSDs active, 4 HDs
powered-down).
35 W with 1 pool active (2 SSDs, 2 HDs active)
~50 W with all HDs active and getting hammered.

Once I added zil + arc to the pools, the little thing runs at wire speeds:
Could not get any better!

It took me months to start gathering the hardware and fully learn ZFS,
but I am really glad to be back running a Solaris derivative: I started in
college administering Ultrix, Sun-OS, AIX, NeXTStep and others. Then later
I went to the dark side to learn all about MS, and finally feel back at
home with OpenIndiana!

Thanks to the IllumOS people and contributors here and everywhere else!

(Sorry for the long post. Hope my full config can help/inspire others).

Jose V.

On Tue, Jun 12, 2012 at 8:00 AM, Jim Klimov <jimklimov at cos.ru> wrote:

> 2012-06-12 15:40, J. V. пишет:
>
>  On 12/06/12 09:16 AM, Rich wrote:
>>
>>> Won't zpool replace fail b/c the new disks require ashift=12 and his
>>> existing pool devices have ashift=9?
>>>
>>
>> This should work fine:
>>
>> HP Microserver, upgrading a mirrored pool from 2TB HDs to 3TB HDs. At one
>> point, the pool had one 2TB HD with 512 sectors and the other with 4k
>> sectors (both in a mirror). Seems there was a bit of a performance hit,
>> but
>> everything worked fine.
>>
>> When I upgraded from 2TB to 3TB, the 3TB HDs were both 4k sectors:
>>
>
> Did the 3TB disk also report to the OS that it uses 4k sectors?
> What ashift value is used by the pool, ultimately? Example:
> # zdb -l /dev/rdsk/c2t0d0s0 | grep ashift
>        ashift: 9
>        ashift: 9
>        ashift: 9
>        ashift: 9
>
> If your 4KB disks use ashift=9 it is possible to get problems
> worse than decreased performance ;)
>
> Note that the ashift is set on a top-level vdev (the second-tier
> component of a pool) and the value is saved into/read from its
> vdevs (edge-tier components), and a pool can mix top-level vdevs
> with different ashifts, i.e. if it is a raid10 type of setup.
>
> PS: Do you use an HP N40L? Did you test if more than 8Gb RAM
> can fit into it (using DDR3 ECC sticks larger than 4Gb), or
> does the server's BIOS forbid that?
>
> HTH,
> //Jim Klimov
>