Xen 6.0.2 software Raid – installation procedure

This document describes how to install XenServer 6.0.2 on a node without hardware raid.

Install Software

Install XenServer 6.0.2 on /dev/sda and do NOT configure any local storage (it is easier to do that afterwards). /dev/sda should containt three partitions, please verify with the following command:

sgdisk -p /dev/sda

The first partition is used for XenServer installation, the second one is used for backups during XenServer upgrades.

1. Now we are going to use /dev/sdb as the mirror disk. Clear the partition table.

sgdisk --zap-all /dev/sdb

2. Install a GPT table on /dev/sdb

sgdisk --mbrtogpt --clear /dev/sdb

3. Create partitions on /dev/sdb. Please note that the following commands are dependent on your installation. Copy the start and the last sectors from the /dev/sda (output of sgdisk -p /dev/sda)

sgdisk --new=1:34:8388641 /dev/sdb
sgdisk --typecode=1:fd00 /dev/sdb
sgdisk --attributes=1:set:2 /dev/sdb
sgdisk --new=2:8388642:16777249 /dev/sdb
sgdisk --typecode=2:fd00 /dev/sdb
sgdisk --new=3:16777250:3907029134 /dev/sdb
sgdisk --typecode=3:fd00 /dev/sdb

4. Create RAID devices

mknod /dev/md0 b 9 0
mknod /dev/md1 b 9 1
mknod /dev/md2 b 9 2
mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb1
mdadm --create /dev/md1 --level=1 --raid-devices=2 missing /dev/sdb2
mdadm --create /dev/md2 --level=1 --raid-devices=2 missing /dev/sdb3

5. Create bitmaps for each RAID device. Bitmaps slightly impact throughput but significantly reduce the rebuilt time when the array fails.

mdadm --grow /dev/md0 -b internal
mdadm --grow /dev/md1 -b internal
mdadm --grow /dev/md2 -b internal

6. Format the root disk and mount it at /mnt

mkfs.ext3 /dev/md0
mount /dev/md0 /mnt

7. Copy the root filesystem to the RAID array (please be patient this step may take a while).

cp -vxpR / /mnt

8. Change the root filesystem in /mnt/etc/fstab to /dev/md0.

sed -r -i 's,LABEL=root-\w+ ,/dev/md0 ,g' /mnt/etc/fstab

9. Install the bootloader on the second hard disk.

mount --bind /dev /mnt/dev
mount -t sysfs none /mnt/sys
mount -t proc none /mnt/proc
chroot /mnt /sbin/extlinux --install /boot
dd if=/mnt/usr/share/syslinux/gptmbr.bin of=/dev/sdb

10. Make a new initrd image which contains a driver for the new root filesystem on the software RAID array.

chroot /mnt
mkinitrd -v -f --theme=/usr/share/splash --without-multipath /boot/initrd-`uname -r`.img `uname -r`

11. edit /mnt/boot/extlinux.conf and replace every mention of the old root filesystem (root=LABEL=xxx) with root=/dev/md0.

sed -r -i 's,root=LABEL=root-\w+ ,root=/dev/md0 ,g' /mnt/boot/extlinux.conf
sed -r -i 's,root=LABEL=root-\w+ ,root=/dev/md0 ,g' /boot/extlinux.conf

12. Unmount the new root and reboot. Important: Remember to use the boot menu of your BIOS to boot from the second hard disk this time!

umount /mnt/proc
umount /mnt/sys
umount /mnt/dev
umount /mnt

13. XenServer is up again, include /dev/sda in the array

sgdisk --typecode=1:fd00 /dev/sda
sgdisk --typecode=2:fd00 /dev/sda
sgdisk --typecode=3:fd00 /dev/sda
mdadm -a /dev/md0 /dev/sda1
mdadm -a /dev/md1 /dev/sda2
mdadm -a /dev/md2 /dev/sda3

14. The array needs to complete its initial build/synchronisation. That will take a while.

watch --interval=1 cat /proc/mdstat

15. Add /dev/md2 as a local SR to XenServer.

xe sr-create content-type=user device-config:device=/dev/md2 name-label="Local Storage" shared=false type=lvm

type=ext is required if you turned on thin provisioning in the installer. Otherwise use type=lvm

Final notes:

* The second partition is used by XenServer for backups, which is why its the same size as the first partition. If you put the install CD in and boot it, an option shows up for "restore XenServer 6.0 from backup partition"

* I have created bitmaps for each raid as well. In the event of the host going down dirty, the raids can require a synch. Simply doing this is enough to add a bitmap for changed pages.

doing cat /proc/mdstat will now say something like

cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sda1[1] sdb1[0]
4193216 blocks [2/2] [UU]
bitmap: 128/128 pages [512KB], 16KB chunk

md1 : active raid1 sda2[1] sdb2[0]
4193216 blocks [2/2] [UU]
bitmap: 0/128 pages [0KB], 16KB chunk

md2 : active raid1 sda3[1] sdb3[0]
968372864 blocks [2/2] [UU]
bitmap: 0/231 pages [0KB], 2048KB chunk

* If you are installing on server which can not boot from the second disk, you must physically swap the two drives to make the machine boot off sdb and use /dev/md0 as root

* If you are going to setup a Xen 6 installation over network (via PXE) and the installation process hangs right after "Freeing unused kernel memory: 280k freed", you pass to the kernel (/tftpboot/pxelinux.cfg/main.menu) the following parameter xencons as follows:

append xenserver6/xen.gz dom0_mem=752M com1=9600,8n1 console=com1,tty --- xenserver6/vmlinuz console=tty0 console=ttyS0,9600n8 xencons=ttyS0,9600n8 answerfile=http://netboot.vendio.com/xenserver6/answers.xml install --- xenserver6/install.img

* To speed up the raid build process the following command can be used (default value is 1000):

echo 100000 > /proc/sys/dev/raid/speed_limit_min

* TIP: You can use the attached script to automate the steps 1 to 9.

Good luck

The script: xen6.sh.zip

Source: http://blog.codeaddict.org/?p=5

8 thoughts on “Xen 6.0.2 software Raid – installation procedure”

  1. Just a short question: Got a running RAID1 from your tutarial, many Thanks for that.

    What should I do, if one disk now goes down completely?

    Buy a new one same size and standard, put in and partition manually like that?
    sgdisk –new=1:34:8388641 /dev/sdb
    sgdisk –typecode=1:fd00 /dev/sdb
    sgdisk –attributes=1:set:2 /dev/sdb
    sgdisk –new=2:8388642:16777249 /dev/sdb
    sgdisk –typecode=2:fd00 /dev/sdb
    sgdisk –new=3:16777250:3907029134 /dev/sdb
    sgdisk –typecode=3:fd00 /dev/sdb

    Then reboot and the system will start rebuilding automatically?

  2. Hi Simon,

    You are welcome. I am glad my post helped you. Basically you should do as follows:
    1. mark the disk as failed
    2. remove it from array
    3. shutdown the computer
    4. install the new disk
    5. add the new disk to existing array

    I already wrote a post about how to do these things. Please look at this page https://sysadmin.compxtreme.ro/mdadm-cheat-sheet/

    Please let me know if you solve the problem.

    1. Hi,

      I have a question about recovery RAID-1. I’m little confuse. ;-)
      If I replace disk (e.g. sda) after rebuild my array, do I have to install bootloader on new disk?

      Of course, as you I’m using GPT partition and extlinux.
      # cat /usr/lib/syslinux/gptmbr.bin > /dev/sda
      + extlinux –install /boot or I don’t have to do anything?

      /boot is not on separate partition.

      Regards and thank for your help

      1. Hi,

        Welcome back Szymon.

        1. Mark the sda disk as failed and remove it from array
        mdadm /dev/md0 –fail /dev/sda1 –remove /dev/sda1
        mdadm /dev/md1 –fail /dev/sda1 –remove /dev/sda2
        mdadm /dev/md2 –fail /dev/sda1 –remove /dev/sda3

        2. Shutdown the computer and replace the defective disk.

        3. Add the new disk to existing array

        mdadm -a /dev/md0 /dev/sda1
        mdadm -a /dev/md1 /dev/sda2
        mdadm -a /dev/md2 /dev/sda3

        4. You can watch the sync status by doing:
        watch -n1 cat /proc/mdstat

        This all you have to do, nothing more. I hope this helps


  3. Hi Joe,

    I have tried this a half-a-dozen times, but I always get Missing OS message when I boot from second disk I get Boot error or Missing OS.

    Any thoughts or checklist?

    WR, -d

Leave a Reply

Your email address will not be published. Required fields are marked *