Xen 6.0.2 software Raid - installation procedure

This document describes how to install XenServer 6.0.2 on a node without hardware raid.

Install Software

Install XenServer 6.0.2 on /dev/sda and do NOT configure any local storage (it is easier to do that afterwards). /dev/sda should containt three partitions, please verify with the following command:

[codesyntax lang="bash"]

sgdisk -p /dev/sda

[/codesyntax]

The first partition is used for XenServer installation, the second one is used for backups during XenServer upgrades.

1. Now we are going to use /dev/sdb as the mirror disk. Clear the partition table.
[codesyntax lang="bash"]

sgdisk --zap-all /dev/sdb

[/codesyntax]

2. Install a GPT table on /dev/sdb
[codesyntax lang="bash"]

sgdisk --mbrtogpt --clear /dev/sdb

[/codesyntax]

3. Create partitions on /dev/sdb. Please note that the following commands are dependent on your installation. Copy the start and the last sectors from the /dev/sda (output of sgdisk -p /dev/sda)
[codesyntax lang="bash"]

sgdisk --new=1:34:8388641 /dev/sdb
sgdisk --typecode=1:fd00 /dev/sdb
sgdisk --attributes=1:set:2 /dev/sdb
sgdisk --new=2:8388642:16777249 /dev/sdb
sgdisk --typecode=2:fd00 /dev/sdb
sgdisk --new=3:16777250:3907029134 /dev/sdb
sgdisk --typecode=3:fd00 /dev/sdb

[/codesyntax]

4. Create RAID devices
[codesyntax lang="bash"]

mknod /dev/md0 b 9 0
mknod /dev/md1 b 9 1
mknod /dev/md2 b 9 2
mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb1
mdadm --create /dev/md1 --level=1 --raid-devices=2 missing /dev/sdb2
mdadm --create /dev/md2 --level=1 --raid-devices=2 missing /dev/sdb3

[/codesyntax]

5. Create bitmaps for each RAID device. Bitmaps slightly impact throughput but significantly reduce the rebuilt time when the array fails.
[codesyntax lang="bash"]

mdadm --grow /dev/md0 -b internal
mdadm --grow /dev/md1 -b internal
mdadm --grow /dev/md2 -b internal

[/codesyntax]

6. Format the root disk and mount it at /mnt
[codesyntax lang="bash"]

mkfs.ext3 /dev/md0
mount /dev/md0 /mnt

[/codesyntax]

7. Copy the root filesystem to the RAID array (please be patient this step may take a while).
[codesyntax lang="bash"]

cp -vxpR / /mnt

[/codesyntax]

8. Change the root filesystem in /mnt/etc/fstab to /dev/md0.
[codesyntax lang="bash"]

sed -r -i 's,LABEL=root-\w+ ,/dev/md0 ,g' /mnt/etc/fstab

[/codesyntax]

9. Install the bootloader on the second hard disk.
[codesyntax lang="bash"]

mount --bind /dev /mnt/dev
mount -t sysfs none /mnt/sys
mount -t proc none /mnt/proc
chroot /mnt /sbin/extlinux --install /boot
dd if=/mnt/usr/share/syslinux/gptmbr.bin of=/dev/sdb

[/codesyntax]

10. Make a new initrd image which contains a driver for the new root filesystem on the software RAID array.
[codesyntax lang="bash"]

chroot /mnt
mkinitrd -v -f --theme=/usr/share/splash --without-multipath /boot/initrd-`uname -r`.img `uname -r`
exit

[/codesyntax]

11. edit /mnt/boot/extlinux.conf and replace every mention of the old root filesystem (root=LABEL=xxx) with root=/dev/md0.
[codesyntax lang="bash"]

sed -r -i 's,root=LABEL=root-\w+ ,root=/dev/md0 ,g' /mnt/boot/extlinux.conf
sed -r -i 's,root=LABEL=root-\w+ ,root=/dev/md0 ,g' /boot/extlinux.conf

[/codesyntax]

12. Unmount the new root and reboot. Important: Remember to use the boot menu of your BIOS to boot from the second hard disk this time!
[codesyntax lang="bash"]

umount /mnt/proc
umount /mnt/sys
umount /mnt/dev
umount /mnt
reboot

[/codesyntax]

13. XenServer is up again, include /dev/sda in the array
[codesyntax lang="bash"]

sgdisk --typecode=1:fd00 /dev/sda
sgdisk --typecode=2:fd00 /dev/sda
sgdisk --typecode=3:fd00 /dev/sda
mdadm -a /dev/md0 /dev/sda1
mdadm -a /dev/md1 /dev/sda2
mdadm -a /dev/md2 /dev/sda3

[/codesyntax]

14. The array needs to complete its initial build/synchronisation. That will take a while.
[codesyntax lang="bash"]

watch --interval=1 cat /proc/mdstat

[/codesyntax]

15. Add /dev/md2 as a local SR to XenServer.
[codesyntax lang="bash"]

xe sr-create content-type=user device-config:device=/dev/md2 name-label="Local Storage" shared=false type=lvm

[/codesyntax]

type=ext is required if you turned on thin provisioning in the installer. Otherwise use type=lvm

Final notes:

* The second partition is used by XenServer for backups, which is why its the same size as the first partition. If you put the install CD in and boot it, an option shows up for "restore XenServer 6.0 from backup partition"

* I have created bitmaps for each raid as well. In the event of the host going down dirty, the raids can require a synch. Simply doing this is enough to add a bitmap for changed pages.

doing cat /proc/mdstat will now say something like

cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sda1[1] sdb1[0]
4193216 blocks [2/2] [UU]
bitmap: 128/128 pages [512KB], 16KB chunk

md1 : active raid1 sda2[1] sdb2[0]
4193216 blocks [2/2] [UU]
bitmap: 0/128 pages [0KB], 16KB chunk

md2 : active raid1 sda3[1] sdb3[0]
968372864 blocks [2/2] [UU]
bitmap: 0/231 pages [0KB], 2048KB chunk

* If you are installing on server which can not boot from the second disk, you must physically swap the two drives to make the machine boot off sdb and use /dev/md0 as root

* If you are going to setup a Xen 6 installation over network (via PXE) and the installation process hangs right after "Freeing unused kernel memory: 280k freed", you pass to the kernel (/tftpboot/pxelinux.cfg/main.menu) the following parameter xencons as follows:

append xenserver6/xen.gz dom0_mem=752M com1=9600,8n1 console=com1,tty --- xenserver6/vmlinuz console=tty0 console=ttyS0,9600n8 xencons=ttyS0,9600n8 answerfile=http://netboot.vendio.com/xenserver6/answers.xml install --- xenserver6/install.img

* To speed up the raid build process the following command can be used (default value is 1000):
[codesyntax lang="bash"]

echo 100000 > /proc/sys/dev/raid/speed_limit_min

[/codesyntax]

* TIP: You can use the attached script to automate the steps 1 to 9.

Good luck

The script: xen6.sh.zip

Source: http://blog.codeaddict.org/?p=5

  1. Just a short question: Got a running RAID1 from your tutarial, many Thanks for that.

    What should I do, if one disk now goes down completely?

    Buy a new one same size and standard, put in and partition manually like that?
    sgdisk --new=1:34:8388641 /dev/sdb
    sgdisk --typecode=1:fd00 /dev/sdb
    sgdisk --attributes=1:set:2 /dev/sdb
    sgdisk --new=2:8388642:16777249 /dev/sdb
    sgdisk --typecode=2:fd00 /dev/sdb
    sgdisk --new=3:16777250:3907029134 /dev/sdb
    sgdisk --typecode=3:fd00 /dev/sdb

    Then reboot and the system will start rebuilding automatically?

  2. Hi Simon,

    You are welcome. I am glad my post helped you. Basically you should do as follows:
    1. mark the disk as failed
    2. remove it from array
    3. shutdown the computer
    4. install the new disk
    5. add the new disk to existing array

    I already wrote a post about how to do these things. Please look at this page https://sysadmin.compxtreme.ro/mdadm-cheat-sheet/

    Please let me know if you solve the problem.

    • Hi,

      I have a question about recovery RAID-1. I'm little confuse. ;-)
      If I replace disk (e.g. sda) after rebuild my array, do I have to install bootloader on new disk?

      Of course, as you I'm using GPT partition and extlinux.
      # cat /usr/lib/syslinux/gptmbr.bin > /dev/sda
      + extlinux --install /boot or I don't have to do anything?

      /boot is not on separate partition.

      Regards and thank for your help
      Szymon

      • Hi,

        Welcome back Szymon.

        1. Mark the sda disk as failed and remove it from array
        mdadm /dev/md0 --fail /dev/sda1 --remove /dev/sda1
        mdadm /dev/md1 --fail /dev/sda1 --remove /dev/sda2
        mdadm /dev/md2 --fail /dev/sda1 --remove /dev/sda3

        2. Shutdown the computer and replace the defective disk.

        3. Add the new disk to existing array

        mdadm -a /dev/md0 /dev/sda1
        mdadm -a /dev/md1 /dev/sda2
        mdadm -a /dev/md2 /dev/sda3

        4. You can watch the sync status by doing:
        watch -n1 cat /proc/mdstat

        This all you have to do, nothing more. I hope this helps

        Regards!

  3. Hi Joe,

    I have tried this a half-a-dozen times, but I always get Missing OS message when I boot from second disk I get Boot error or Missing OS.

    Any thoughts or checklist?

    WR, -d

  4. very useful.. thank you!

Scrie si tu o vorbulita


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Trackbacks and Pingbacks: