[OpenIndiana-discuss] RAIDZ issue

Fri Feb 19 05:34:35 UTC 2021

If I create a one vdev ( raidz2 with 6 disks ) it boots fine. (Virtualbox
limitation of 5 disk does not come here)If I create two vdevs (raidz2 with
6 disk) I see a boot issue.I need to understand why the problem only comes
in the second case ?

On Thu, Feb 18, 2021 at 10:12 PM Toomas Soome <tsoome at me.com> wrote:

>
>
> On 18. Feb 2021, at 18:15, Thebest videos <sri.chityala504 at gmail.com>
> wrote:
>
> we are able to achieve RAIDZ configuration in other way like , we are able
> to create RAIDZ2 with 5 disks in vdev at initial. After reboot we are
> adding disks to the existing pool as 2nd vdev with 5 disks and then reboot
> again adding disks to the same pool as 3 vdev and so on....... the small
> change we have in command is as below(giving labelname of disk with
> /dev/gpt,  before we were giving disk name as ada0.....)
> ------before-------
> zpool create datapool raidz2 ada0 ada1 ada2 ada3 ada4
> -------after--------
> zpool create datapool raidz2 /dev/gpt/disk0 /dev/gpt/disk1 /dev/gpt/disk2
> /dev/gpt/disk3 /dev/gpt/disk4   #for the first time
> then reboot
> then we are adding disks to the pool in the existing pool with 5 disks.
> This process is repeated for every reboot. to make these 15 disks part of
> RAIDZ. But the problem , this is not our requirement we should create RAIDZ
> with multiple vdev's in single commands instead of adding on reboot
> zpool create datapool raidz2 /dev/gpt/disk0 ...............raidz2
> /dev/gpt/disk4..............raidz2 /dev/gpt/disk9 ................. #this
> way it should work
> in short, we need create RAIDZ with all disks all at once
>
> So any suggestion to achieve at once...!?
>
>
> nono, do not let yourself to be deceived.
>
> on initial setup, all data is on first vdev, once you got second vdev
> added, the initial data is still on first vdev. When you will update the
> OS, the old kernel will not be overwritten, but new blocks will be
> allovated from all the vdevs, especially from most recently added ones -
> because zfs will try to balance the vdev alloctions. Once that will happen,
> the bootability is gone.
>
> rgds,
> toomas
>
>
>
>
> On Thu, Feb 18, 2021 at 6:16 PM Toomas Soome <tsoome at me.com> wrote:
>
>>
>>
>> On 18. Feb 2021, at 14:23, Thebest videos <sri.chityala504 at gmail.com>
>> wrote:
>>
>> I am still new in the freebsd zfs world so I am asking the question below kindly bear with me:Is it necessary to create a boot partition on each disk, to make it part of raidz configuration
>>
>>
>> boot partition (zfs-boot/efi) needs to be on member of bootable pool for
>> two reasons; first, if you have disk failing, you want to be able to boot
>> from other disk, and secondly, it will help to keep devices in pool to have
>> exactly the same layout.
>>
>> When you say boot pool what does it mean exactly ?
>>
>>
>> boot pool is the pool you use to load boot loader and OS (kernel).
>> Specifically, you point your BIOS to use boot disk belonging to boot pool
>> and the pool itself does have bootfs property set (zpool get bootfs).
>>
>> boot pool normally does contain the OS installation.
>>
>>      you mean to say should I create separate boot pool and data pool something like     zpool create bootpool raidz disk1-p1 disk2-p1
>>      zpool create datapool raidz disk1-p3 disk2-p3 Or you mean something else.I am still not able to understand virtualbox limit of 5 disk how it is blocking me.
>>
>>
>> with virtualbox, this limit means your boot pool must be built from max 5
>> disks, and those 5 disks must be first in disk list. If you use more disks,
>> then virtualbox will not see the extra ones and those disks are marked as
>> UNKNOWN. if more than parity number disks are missing, we can not read the
>> pool.
>>
>> what is your recommendation to arrange 13 disk in raidz configuration (you can avoid this question if it is going beyond )
>>
>>
>> There is no one answer for this question, it depends on what kind of IO
>> will be done there. You can create one single 10+2 raidz2 with spare or
>> 10+3 raidz3, but with raidz, all writes are whole stripe writes.
>>
>> rgds,
>> toomas
>>
>>
>> On Thu, Feb 18, 2021 at 4:48 PM Toomas Soome <tsoome at me.com> wrote:
>>
>>>
>>>
>>> On 18. Feb 2021, at 12:52, Thebest videos <sri.chityala504 at gmail.com>
>>> wrote:
>>>
>>> Ok, We also generated .img file using our custom OS from freebsd source.
>>> we are uploaded img file to digital ocean images. then we are creating
>>> droplet. everything working fine for basic operating system. but we are
>>> facing same issue at droplet side. atleast on virtual box we are able to
>>> create single vdev upto 5 disks and 2 vdevs with 3 disk each vdev(i mean
>>> upto 6 disks). but digital ocean side we are unable to create atleast
>>> single vdev with 3 disks. its working fine with 2 disks as mirror pool. we
>>> are raised the issue on digital ocean like any restrictions on number of
>>> disks towards the RAIDZ. but they says there is no constraints on number of
>>> disks. we can create as RAIDZ as many number of disks. we still don't
>>> understand where is the mistake. we also raised same query on freebsd forum
>>> but no response. as i already shares the manual steps which we are
>>> following to create partitions and RAIDZ configuration. are we making any
>>> mistake from commands which we are following towards RAIDZ configuration or
>>> as you said its king of restrictions on number of disks on virtual box and
>>> might digital ocean side. i mean restricitons on vender side?!.  Any
>>> guesses if it works(if no mistakes from commands we are using)if we attach
>>> CD/image to any bare metal server...?! or any suggestions?
>>>
>>>
>>>
>>> I have no personal experience with digital ocean, but the basic test is
>>> the same; if you get loader OK prompt, use lsdev -v command to check how
>>> many disks you can actually see. There actually is another option too —
>>> with BIOS boot, when you see the very first spinner, press space key and
>>> you will get boot: prompt. This is very limited but still useful prompt
>>> from gptzfsboot proagram (the one which will try to find and start
>>> /boot/loader). On boot: prompt, you can enter: status — this will produce
>>> the same report as you get from lsdev.
>>>
>>> So, if you know your VM should have, say, 10 disks, but boot: status or
>>> ok lsdev will show less, then you know, there must be BIOS limit (we do use
>>> BIOS INT13h to access the disks).
>>>
>>> Please note, if the provider does offer option to use UEFI, it *may*
>>> support greater number of boot disks, the same check does apply with UEFI
>>> as well (lsdev -v).
>>>
>>> rgds,
>>> toomas
>>>
>>>
>>> These are commands we are using to create partitions and RAIDZ
>>> configuration
>>> NOTE: we are creating below gpart partitions(boot,swap,root) on all hard
>>> disks then adding those hard disks in zpool command
>>> Doubt: should we create partitions(boot,swap,root) on all hard disks to
>>> make part of RAIDZ configuration or is it enough to add in zpool as raw
>>> disks or making 2-3 disks as bootable then remaining as raw disks? anyway
>>> please check below commands we are using to create partitions and zpool
>>> configurations
>>>     gpart create -s gpt /dev/da0
>>>     gpart add -a 4k -s 512K -t freebsd-boot da0
>>>     gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
>>>     gpart add -a 1m -s 2G -t freebsd-swap -l swap1 da0
>>>     gpart add -a 1m -t freebsd-zfs -l disk1 da0
>>>    zpool create -o altroot=/mnt datapool raidz2 ada0p3 ada1p3 ada2p3
>>> ada3p3  raidz2 ada4p3 ada5p3 ada6p3 ada7p3
>>>     zfs create -o mountpoint=/ -o canmount=noauto datapool/boot
>>>     mount -t zfs datapool/boot /mnt
>>>     cp -r /temp/* /mnt/.
>>>     zpool set bootfs=datapool/boot datapool
>>>     zfs create -o mountpoint=/storage -o canmount=noauto datapool/storage
>>>     zfs create -o mountpoint=/conf -o canmount=noauto datapool/conf
>>>     shutdown and remove iso/img and start it again
>>>     zpool import datapool
>>>     mkdir /conf /storage
>>>     mount -t zfs datapool/conf /conf
>>>     mount -t zfs datapool/storage /storage
>>>
>>>
>>>
>>> On Thu, Feb 18, 2021 at 3:33 PM Toomas Soome <tsoome at me.com> wrote:
>>>
>>>>
>>>>
>>>> On 18. Feb 2021, at 11:52, Thebest videos <sri.chityala504 at gmail.com>
>>>> wrote:
>>>>
>>>> as per your reply, im not clear
>>>> although i've tried to create 2 pools with 4 disks(for testing purpose)
>>>> each pool in a single vdev as expected it works. but that is not our
>>>> requirement since we intended to chose single pool as many number of disks
>>>> which should part of multiple vdev's based of condition(max 5 disks each
>>>> vdev) and any disks left after part of vdev should act as spare disks.
>>>> finally max 5 disks are coming ONLINE in vdev remaining disks going as
>>>> says OFFLINE state disk state is UNKNOWN. is there anyway to fix this issue.
>>>>
>>>>
>>>>
>>>> If you want to use virtualbox, then there is limit that virtualbox does
>>>> only see first 5 disk devices. This is vbox limit and there are only two
>>>> options about it - either accept it or to file feature request to
>>>> virtualbox developers.
>>>>
>>>> Different systems can set different limits there, for example, VMware
>>>> Fusion does support booting from first 12 disks. It also can have more
>>>> disks than 12, but  only first 12 are visible for boot loader.
>>>>
>>>> Real hardware is vendor specific.
>>>>
>>>> rgds,
>>>> toomas
>>>>
>>>>
>>>> On Thu, Feb 18, 2021 at 1:24 AM Toomas Soome via openindiana-discuss <
>>>> openindiana-discuss at openindiana.org> wrote:
>>>>
>>>>>
>>>>>
>>>>> > On 17. Feb 2021, at 20:49, Thebest videos <sri.chityala504 at gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > NOTE: we are getting issues after shutdown , then remove ISO file
>>>>> from
>>>>> > virtualBox then power on the server. if we attach an ISO file we are
>>>>> safe
>>>>> > with our Zpool stuff. and we are creating boot,swap,root partitions
>>>>> on each
>>>>> > disks.
>>>>>
>>>>> vbox seems to have limit on boot disks - it appears to “see” 5, My
>>>>> vbox has IDE for boot disk, and I did add 6 sas disks, I only can see 5 —
>>>>> ide + 4 sas.
>>>>>
>>>>> So all you need to do is to add disk for boot pool, and make sure it
>>>>> is first one - once kernel is up, it can see all the disks.
>>>>>
>>>>> rgds,
>>>>> toomas
>>>>>
>>>>>
>>>>> > I'm not able to understand First 5 disks are ONLINE and remaining
>>>>> disks are
>>>>> > UNKNOWN state after power off and then power on
>>>>> > actually our requirement is to create RAIDZ1/RAIDZ2 with single
>>>>> vdev(upto 5
>>>>> > disks per vdev) if more than 5 or less than 10 disks then those
>>>>> disks(after
>>>>> > 5disks) are spare part shouldn't be included any vdev. if we have
>>>>> > multiple's of 5 disks then we need to create multiple vdev in a pool
>>>>> > example: RAIDZ2 : if total 7 disks then 5 disks as single vdev,
>>>>> remaining 2
>>>>> > disks as spare parts nothing to do. and if we have 12 disks intotal
>>>>> then 2
>>>>> > vdevs (5 disks per vdev) so total 10 disks in 2 vdevs remaining
>>>>> 2disks as
>>>>> > spare.
>>>>> > RAIDZ1: if we have only 3 disks then we should create RAIDZ1
>>>>> >
>>>>> > Here, we wrote a zfs script for our requirements(but currently
>>>>> testing with
>>>>> > manual commands). We are able to createRAIDZ2 with a single vdev in
>>>>> a pool
>>>>> > for 5 disks. it works upto 9 disks but if we have 10 disks then 2
>>>>> vdevs are
>>>>> > created after power on the same error coming like zfs: i/o error all
>>>>> copies
>>>>> > blocked.
>>>>> > I was testing the RAIDZ like I'm creating 2 vdevs which have 3 disks
>>>>> per
>>>>> > each vdev.its working fine even after shutdown and power on(as says
>>>>> that we
>>>>> > are removing the ISO file after shutdown).
>>>>> > but the issue is when we create 2 vdevs with 4 disks per each
>>>>> vdev.this
>>>>> > time we are not getting error its giving options like we press esc
>>>>> button
>>>>> > what kind of options we see those options are coming. if i type
>>>>> lsdev -v(as
>>>>> > you said before). first 5 disks are online and the remaining 3 disks
>>>>> are
>>>>> > UNKNOWN.
>>>>> >
>>>>> > FInally, I need to setup RAIDZ configuration with 5 multiples of
>>>>> disks per
>>>>> > each vdev.  please look once again below commands im using to create
>>>>> > partitions and RAIDZ configuration
>>>>> >
>>>>> > NOTE: below gpart commands are running for each disk
>>>>> >
>>>>> > gpart create -s gpt ada0
>>>>> >
>>>>> >
>>>>> > gpart add -a 4k -s 512K -t freebsd-boot ada0
>>>>> >
>>>>> >
>>>>> > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
>>>>> >
>>>>> >
>>>>> > gpart add -a 1m -s 2G -t freebsd-swap -l swap0 ada0
>>>>> >
>>>>> >
>>>>> > gpart add -a 1m -t freebsd-zfs -l disk0 ada0
>>>>> >
>>>>> >
>>>>> > zpool create -o altroot=/mnt datapool raidz2 ada0p3 ada1p3 ada2p3
>>>>> > ada3p3  raidz2 ada4p3 ada5p3 ada6p3 ada7p3
>>>>> >
>>>>> >
>>>>> > zfs create -o mountpoint=/ -o canmount=noauto datapool/boot
>>>>> >
>>>>> >
>>>>> > mount -t zfs datapool/boot /mnt
>>>>> >
>>>>> >
>>>>> > mount_cd9660 /dev/cd0 /media
>>>>> >
>>>>> >
>>>>> > cp -r /media/* /mnt/.
>>>>> >
>>>>> >
>>>>> > zpool set bootfs=datapool/boot datapool
>>>>> >
>>>>> >
>>>>> > shutdown and remove ISO and power on the server
>>>>> >
>>>>> >
>>>>> > kindly suggest me steps if im wrong
>>>>> >
>>>>> > On Wed, Feb 17, 2021 at 11:51 PM Thebest videos <
>>>>> sri.chityala504 at gmail.com>
>>>>> > wrote:
>>>>> >
>>>>> >> prtconf -v | grep biosdev not working on freebsd
>>>>> >> i think its legacy boot system(im not sure actually i didnt find
>>>>> anything
>>>>> >> about EFI related stuff) is there anyway to check EFI
>>>>> >>
>>>>> >> Create the pool with EFI boot:
>>>>> >> # zpool create -B rpool raidz c0t0d0 c0t1d0 c0t3d0
>>>>> >>
>>>>> >> how can i create pool with EFI
>>>>> >> and -B what it refers?
>>>>> >>
>>>>> >> On Wed, Feb 17, 2021 at 11:00 PM John D Groenveld <
>>>>> groenveld at acm.org>
>>>>> >> wrote:
>>>>> >>
>>>>> >>> In message <272389262.2537371.1613575739056 at mail.yahoo.com>,
>>>>> Reginald
>>>>> >>> Beardsley
>>>>> >>> via openindiana-discuss writes:
>>>>> >>>> I was not aware that it was possible to boot from RAIDZ. It wasn't
>>>>> >>> possible wh
>>>>> >>>
>>>>> >>> With the current text installer, escape to a shell.
>>>>> >>> Confirm the disks are all BIOS accessible:
>>>>> >>> # prtconf -v | grep biosdev
>>>>> >>> Create the pool with EFI boot:
>>>>> >>> # zpool create -B rpool raidz c0t0d0 c0t1d0 c0t3d0
>>>>> >>> Exit and return to the installer and then F5 Install to an
>>>>> Existing Pool
>>>>> >>>
>>>>> >>> John
>>>>> >>> groenveld at acm.org
>>>>> >>>
>>>>> >>> _______________________________________________
>>>>> >>> openindiana-discuss mailing list
>>>>> >>> openindiana-discuss at openindiana.org
>>>>> >>> https://openindiana.org/mailman/listinfo/openindiana-discuss
>>>>> >>>
>>>>> >>
>>>>> > _______________________________________________
>>>>> > openindiana-discuss mailing list
>>>>> > openindiana-discuss at openindiana.org
>>>>> > https://openindiana.org/mailman/listinfo/openindiana-discuss
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> openindiana-discuss mailing list
>>>>> openindiana-discuss at openindiana.org
>>>>> https://openindiana.org/mailman/listinfo/openindiana-discuss
>>>>>
>>>> <Screenshot 2021-02-18 at 12.38.35 PM.png>
>>>>
>>>>
>>>>
>>>
>>
>