Tag Archives: raid

mdadm refuses to re-add failed member

Publicat la 23 May 2015 No comments 
Radu, thank you for helping me with this problem!

My beloved power distribution company had always have problems. It is common thing to have a power outage up to half an hour every few weeks. Lately however, the problems increased in rate and gravity. I don't know exactly what happened last time, but there were 2 power outages in 30 minutes and 6 in 24 hours. Voltage peaks and spikes, variations in the frequency and everything. The result: the UPS and one disk were broken. What doesn't kill you makes you stronger. I guess mine were not strong enough for this challenge. I also had a very old HITACHI disk which was already showing signs of failure (part of an RAID1 array) .

So I decided to replace both disks. This is a piece of cake operation, isn't it? It wasn't because the server was located in Romania and I am in Sweden.

After removing the faulty disk from the array and shutting down the server

[codesyntax lang="bash"]

mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1
mdadm /dev/md1 --fail /dev/sdb2 --remove /dev/sdb2
mdadm /dev/md2 --fail /dev/sdb3 --remove /dev/sdb3

[/codesyntax]

I asked a friend of mine to physically replace the disk.

Then I re-added the disk in array

[codesyntax lang="bash"]

mdadm --add /dev/md0 /dev/sdb1
mdadm --add /dev/md1 /dev/sdb2
mdadm --add /dev/md2 /dev/sdb3

[/codesyntax]

and waited to sync.

[codesyntax lang="bash"]

echo 100000 > /proc/sys/dev/raid/speed_limit_min
watch -n1 cat /proc/mdstat

[/codesyntax]

The next thing was to reboot the server to make sure everything was fine.

Well, it wasn't fine. I couldn't connect to the server, so something happened with the filesystem, raid or whatever.
After having a chat with my friend from Romania I find out that there was a problem with /dev/md2.

AvOeT3szbh6yExpxltAnYtHi69O3WzGwvAnW4tbaTs2p

AuQ0YqQNMQa_Ap1sPLpensRHAEkzdOj6YrZJvTTD--AC

AqUu6fs4CpZGMAsoB5KNkK9ruCo2SGjlKdtLEx5hhrcY

AhbCtrh2Yf6g5WgvtGMrmutdXKI9MGtYEiSIQnMNfRwe

Something really strange was going on.

We tried different things like removing and re-adding the disk, --zero-superblock the new disk... More or less the same error.

When I was out of ideas and just about to give up when I received this messages from my friend "I fixed it for you!!!! Apparently there is a bug in metadata 0.90 and you have to use 1.2, but you can't choose metadata version with the debian installer. I sent you an email. Read it!".

Basically the email with the solution contained only one line: http://serverfault.com/questions/265056/mdadm-assembles-with-drives-instead-of-partitions

After he added DEVICE ... to /etc/mdadm/mdadm.conf the problem was solved.

[codesyntax lang="bash"]

vim /etc/mdadm/mdadm.conf

[/codesyntax]

DEVICE /dev/sda1
DEVICE /dev/sda2
DEVICE /dev/sda3

DEVICE /dev/sdb1
DEVICE /dev/sdb2
DEVICE /dev/sdb3

MDADM Cheat Sheet

Publicat la 9 November 2012 No comments

Create a new RAID array

Create (mdadm --create) is used to create a new array:
[codesyntax lang="bash"]

mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb2

[/codesyntax]

or using the compact notation:
[codesyntax lang="bash"]

mdadm -Cv /dev/md0 -l1 -n2 /dev/sd[ab]1

[/codesyntax]

/etc/mdadm.conf

/etc/mdadm.conf or /etc/mdadm/mdadm.conf (on debian) is the main configuration file for mdadm. After we create our RAID arrays we add them to this file using:
[codesyntax lang="bash"]

mdadm --detail --scan >> /etc/mdadm.conf

[/codesyntax]

or on debian
[codesyntax lang="bash"]

mdadm --detail --scan >> /etc/mdadm/mdadm.conf

[/codesyntax]

Remove a disk from an array

We can’t remove a disk directly from the array, unless it is failed, so we first have to fail it (if the drive it is failed this is normally already in failed state and this step is not needed):
[codesyntax lang="bash"]

mdadm --fail /dev/md0 /dev/sda1

[/codesyntax]

and now we can remove it:
[codesyntax lang="bash"]

mdadm --remove /dev/md0 /dev/sda1

[/codesyntax]

This can be done in a single step using:
[codesyntax lang="bash"]

mdadm /dev/md0 --fail /dev/sda1 --remove /dev/sda1

[/codesyntax]

Add a disk to an existing array

We can add a new disk to an array (replacing a failed one probably):
[codesyntax lang="bash"]

mdadm --add /dev/md0 /dev/sdb1

[/codesyntax]

Verifying the status of the RAID arrays

We can check the status of the arrays on the system with:
[codesyntax lang="bash"]

cat /proc/mdstat

[/codesyntax]

or

[codesyntax lang="bash"]

mdadm --detail /dev/md0

[/codesyntax]

The output of this command will look like:

[codesyntax lang="bash"]

cat /proc/mdstat

[/codesyntax]
Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[0]
104320 blocks [2/2] [UU]

md1 : active raid1 sdb3[1] sda3[0]
19542976 blocks [2/2] [UU]

md2 : active raid1 sdb4[1] sda4[0]
223504192 blocks [2/2] [UU]

here we can see both drives are used and working fine - U. A failed drive will show as F, while a degraded array will miss the second disk _

Note: while monitoring the status of a RAID rebuild operation using watch can be useful:
[codesyntax lang="bash"]

watch -n1 cat /proc/mdstat

[/codesyntax]

Checking a linux MD raid array

In this example my RAID array will be /dev/md0

[codesyntax lang="bash"]

cd /sys/block/md0/md
echo check >sync_action
watch -n1 cat /proc/mdstat

[/codesyntax]

OR:

[codesyntax lang="bash"]

/usr/share/mdadm/checkarray -a /dev/md0
watch -n1 cat /proc/mdstat

[/codesyntax]

Note: if you receive the following error message:

[codesyntax lang="bash"]

checkarray: W: array md0 in auto-read-only state, skipping...

[/codesyntax]

Then you should do this:
[codesyntax lang="bash"]

mdadm --readwrite /dev/md0

[/codesyntax]

Stop and delete a RAID array

If we want to completely remove a raid array we have to stop if first and then remove it:
[codesyntax lang="bash"]

mdadm --stop /dev/md0
mdadm --remove /dev/md0

[/codesyntax]

and finally we can even delete the superblock from the individual drives:
[codesyntax lang="bash"]

mdadm --zero-superblock /dev/sda

[/codesyntax]

Finally in using RAID1 arrays, where we create identical partitions on both drives this can be useful to copy the partitions from sda to sdb:
[codesyntax lang="bash"]

sfdisk -d /dev/sda | sfdisk /dev/sdb

[/codesyntax]

(this will dump the partition table of sda, removing completely the existing partitions on sdb, so be sure you want this before running this command, as it will not warn you at all).

There are many other usages of mdadm particular for each type of RAID level, and I would recommend to use the manual page (man mdadm) or the help (mdadm --help) if you need more details on its usage. Hopefully these quick examples will put you on the fast track with how mdadm works.

Source: This info is taken from here and here

Create bitmaps for RAID device

Publicat la 9 November 2012 No comments

It's a good idea to create bitmaps for RAID device that are stored internally (storing bitmap files on other filesystems may result in serious problems). Bitmaps slightly impact throughput but significantly reduce the rebuilt time when the array fails.

[codesyntax lang="bash"]

mdadm --grow /dev/mdX -b internal

[/codesyntax]

How to migrate a single disk Linux System to software RAID1

Publicat la 23 October 2012 19 comentarii

This guide explains how to set up software RAID1 on an already running Linux (Ubuntu 12.10) system. The GRUB2 bootloader will be configured in such a way that the system will still be able to boot if one of the hard drives fails (no matter which one).

Preliminary Note

In this tutorial I am using an Ubuntu 12.10 system with two disks, /dev/sda and /dev/sdb which are identical in size.
/dev/sdb is currently unused, and /dev/sda has the following partition:

/dev/sda1: / partition, ext4;
/dev/sda5: swap

After completing this guide I will have the following situation:
/dev/md0: / partition, ext4;
/dev/md1: swap

The current situation:

[codesyntax lang="bash"]

df -h

[/codesyntax]

root@ubuntu:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        19G  969M   17G   6% /
udev            494M  4.0K  494M   1% /dev
tmpfs           201M  272K  201M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            502M     0  502M   0% /run/shm
none            100M     0  100M   0% /run/user
root@ubuntu:~#

[codesyntax lang="bash"]

fdisk -l

[/codesyntax]

root@ubuntu:~# fdisk -l

Disk /dev/sda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00059a4b

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048    39845887    19921920   83  Linux
/dev/sda2        39847934    41940991     1046529    5  Extended
/dev/sda5        39847936    41940991     1046528   82  Linux swap / Solaris

Disk /dev/sdb: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table
root@ubuntu:~#

Installing mdadm

First of all install md tools:

[codesyntax lang="bash"]

aptitude install initramfs-tools mdadm

[/codesyntax]

In order to avoid reboot, let's load few kernel modules:

[codesyntax lang="bash"]

modprobe linear
modprobe multipath
modprobe raid0
modprobe raid1
modprobe raid5
modprobe raid6
modprobe raid10

[/codesyntax]

Now:

[codesyntax lang="bash"]

cat /proc/mdstat

[/codesyntax]

root@ubuntu:~# cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
unused devices:
root@ubuntu:~#

Preparing the second disk

To create a software RAID1 on a running system, we have to prepare the second disk added to the system (in this case /dev/sdb) for RAID1, then copy the contents from the first disk (/dev/sda) to it, and finally add the first disk to the RAID1 array.

Let's copy the partition table from /dev/sda to /dev/sdb so that the both disks have the exactly same layout:

[codesyntax lang="bash"]

sfdisk -d /dev/sda | sfdisk --force /dev/sdb

[/codesyntax]

root@ubuntu:~# sfdisk -d /dev/sda | sfdisk --force /dev/sdb
Checking that no-one is using this disk right now ...
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
OK

Disk /dev/sdb: 2610 cylinders, 255 heads, 63 sectors/track

sfdisk: ERROR: sector 0 does not have an msdos signature
/dev/sdb: unrecognized partition table type
Old situation:
No partitions found
New situation:
Units = sectors of 512 bytes, counting from 0

Device Boot    Start       End   #sectors  Id  System
/dev/sdb1   *      2048  39845887   39843840  83  Linux
/dev/sdb2      39847934  41940991    2093058   5  Extended
/dev/sdb3             0         -          0   0  Empty
/dev/sdb4             0         -          0   0  Empty
/dev/sdb5      39847936  41940991    2093056  82  Linux swap / Solaris
Warning: partition 1 does not end at a cylinder boundary
Warning: partition 2 does not start at a cylinder boundary
Warning: partition 2 does not end at a cylinder boundary
Warning: partition 5 does not end at a cylinder boundary
Successfully wrote the new partition table

Re-reading the partition table ...

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
root@ubuntu:~#

And the output of the command:

[codesyntax lang="bash"]

fdisk -l

[/codesyntax]

root@ubuntu:~# fdisk -l

Disk /dev/sda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00059a4b

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048    39845887    19921920   83  Linux
/dev/sda2        39847934    41940991     1046529    5  Extended
/dev/sda5        39847936    41940991     1046528   82  Linux swap / Solaris

Disk /dev/sdb: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *        2048    39845887    19921920   83  Linux
/dev/sdb2        39847934    41940991     1046529    5  Extended
/dev/sdb5        39847936    41940991     1046528   82  Linux swap / Solaris
root@ubuntu:~#

Change the partitions type on /dev/sdb to Linux raid autodetect:

[codesyntax lang="bash"]

sfdisk --change-id /dev/sdb 1 fd
sfdisk --change-id /dev/sdb 5 fd

[/codesyntax]

root@ubuntu:~# sfdisk --change-id /dev/sdb 1 fd
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Done

root@ubuntu:~# sfdisk --change-id /dev/sdb 5 fd
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Done

root@ubuntu:~#

To make sure that there are no remains from previous RAID installations on /dev/sdb, we run the following commands:

[codesyntax lang="bash"]

mdadm --zero-superblock /dev/sdb1
mdadm --zero-superblock /dev/sdb5

[/codesyntax]

If you receive the following error messages then there are no remains from previous RAID installations, which is nothing to worry about:

root@ubuntu:~# mdadm --zero-superblock /dev/sdb1
mdadm: Unrecognised md component device - /dev/sdb1
root@ubuntu:~# mdadm --zero-superblock /dev/sdb5
mdadm: Unrecognised md component device - /dev/sdb5
root@ubuntu:~#

Creating RAID arrays

Now use mdadm to create the raid arrays. We mark the first drive (sda) as "missing" so it doesn't wipe out our existing data:

[codesyntax lang="bash"]

mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/sdb1
mdadm --create /dev/md1 --level=1 --raid-disks=2 missing /dev/sdb5

[/codesyntax]

root@ubuntu:~# mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/sdb1
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
root@ubuntu:~# mdadm --create /dev/md1 --level=1 --raid-disks=2 missing /dev/sdb5
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md1 started.
root@ubuntu:~#

[codesyntax lang="bash"]

cat /proc/mdstat

[/codesyntax]

root@ubuntu:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid1 sdb5[1]
      1045952 blocks super 1.2 [2/1] [_U]
      
md0 : active raid1 sdb1[1]
      19905408 blocks super 1.2 [2/1] [_U]
      
unused devices: <none>
root@ubuntu:~# 

The output above means that we have two degraded arrays ([_U] or [U_] means that an array is degraded while [UU] means that the array is ok).

Create the filesystems on RAID arrays (ext4 on /dev/md0 and swap on /dev/md1)

[codesyntax lang="bash"]

mkfs.ext4 /dev/md0
mkswap /dev/md1

[/codesyntax]

root@ubuntu:~# mkfs.ext4 /dev/md0
mke2fs 1.42.5 (29-Jul-2012)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
1245184 inodes, 4976352 blocks
248817 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
152 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done   

root@ubuntu:~# mkswap /dev/md1
mkswap: /dev/md1: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.
Setting up swapspace version 1, size = 1045948 KiB
no label, UUID=728f7cfe-bd95-43e5-906d-c8a70023d081
root@ubuntu:~# 

Adjust mdadm configuration file which doesn't contain any information about RAID arrays yet:

[codesyntax lang="bash"]

cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf_orig
mdadm --examine --scan >> /etc/mdadm/mdadm.conf

[/codesyntax]

Display the content of /etc/mdadm/mdadm.conf:

[codesyntax lang="bash"]

cat /etc/mdadm/mdadm.conf

[/codesyntax]

root@ubuntu:~# cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

# This file was auto-generated on Tue, 23 Oct 2012 04:36:40 -0700
# by mkconf $Id$
root@ubuntu:~#

Adjusting The System To RAID1

Let's mount /dev/md0:

[codesyntax lang="bash"]

mkdir /mnt/md0
mount /dev/md0 /mnt/md0

[/codesyntax]

[codesyntax lang="bash"]

mount

[/codesyntax]

root@ubuntu:~# mount
/dev/sda1 on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
/dev/md0 on /mnt/md0 type ext4 (rw)
root@ubuntu:~# 

Change the UID values in /etc/fstab with the UUID values returned by blkid:

[codesyntax lang="bash"]

blkid /dev/md0 /dev/md1

[/codesyntax]

root@ubuntu:~# blkid /dev/md0 /dev/md1
/dev/md0: UUID="4a49251b-e357-40a4-b13f-13b041c55a9d" TYPE="ext4"
/dev/md1: UUID="728f7cfe-bd95-43e5-906d-c8a70023d081" TYPE="swap"
root@ubuntu:~#

After changing the UUID values the /etc/fstab should look as follows:

[codesyntax lang="bash"]

cat /etc/fstab

[/codesyntax]

root@ubuntu:~# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# # / was on /dev/sda1 during installation
UUID=4a49251b-e357-40a4-b13f-13b041c55a9d / ext4 errors=remount-ro 0 1
# swap was on /dev/sda5 during installation
UUID=728f7cfe-bd95-43e5-906d-c8a70023d081 none swap sw 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
root@ubuntu:~#

Next replace /dev/sda1 with /dev/md0 in /etc/mtab:

[codesyntax lang="bash"]

sed -e "s/dev\/sda1/dev\/md0/" -i /etc/mtab

[/codesyntax]

[codesyntax lang="bash"]

cat /etc/mtab

[/codesyntax]

root@ubuntu:~# cat /etc/mtab
/dev/md0 / ext4 rw,errors=remount-ro 0 0
proc /proc proc rw,noexec,nosuid,nodev 0 0
sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0
none /sys/fs/fuse/connections fusectl rw 0 0
none /sys/kernel/debug debugfs rw 0 0
none /sys/kernel/security securityfs rw 0 0
udev /dev devtmpfs rw,mode=0755 0 0
devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=0620 0 0
tmpfs /run tmpfs rw,noexec,nosuid,size=10%,mode=0755 0 0
none /run/lock tmpfs rw,noexec,nosuid,nodev,size=5242880 0 0
none /run/shm tmpfs rw,nosuid,nodev 0 0
none /run/user tmpfs rw,noexec,nosuid,nodev,size=104857600,mode=0755 0 0
/dev/md0 /mnt/md0 ext4 rw 0 0
root@ubuntu:~#

Setup the GRUB2 boot loader.

Create the file /etc/grub.d/09_swraid1_setup as follows:

[codesyntax lang="bash"]

cp /etc/grub.d/40_custom /etc/grub.d/09_swraid1_setup
vim /etc/grub.d/09_swraid1_setup

[/codesyntax]

#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
menuentry 'Ubuntu, with Linux 3.5.0-17-generic' --class ubuntu --class gnu-linux --class gnu --class os {
    recordfail
    insmod mdraid1x
    insmod ext2
    set root='(md/0)'
    linux   /boot/vmlinuz-3.5.0-17-generic root=/dev/md0 ro   quiet
    initrd  /boot/initrd.img-3.5.0-17-generic
}

Make sure you use the correct kernel version in the menuentry (in the linux and initrd lines).

[codesyntax lang="bash"]

uname -r

[/codesyntax]

root@ubuntu:~# uname -r
3.5.0-17-generic
root@ubuntu:~#

Update grub configuration and adjust our ramdisk to the new situation:

[codesyntax lang="bash"]

update-grub
update-initramfs -u

[/codesyntax]

root@ubuntu:~# update-grub
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-3.5.0-17-generic
Found initrd image: /boot/initrd.img-3.5.0-17-generic
Found memtest86+ image: /boot/memtest86+.bin
done
root@ubuntu:~# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-3.5.0-17-generic
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
root@ubuntu:~#

Copy files to the new disk

Copy the files from the first disk (/dev/sda) to the second one (/dev/sdb)

[codesyntax lang="bash"]

cp -dpRx / /mnt/md0

[/codesyntax]

Preparing GRUB2 (Part 1)

Install GRUB2 boot loader on both disks (/dev/sda and /dev/sdb):

[codesyntax lang="bash"]

grub-install /dev/sda
grub-install /dev/sdb

[/codesyntax]

root@ubuntu:~# grub-install /dev/sda
Installation finished. No error reported.
root@ubuntu:~# grub-install /dev/sdb
Installation finished. No error reported.

Now we reboot the system and hope that it boots ok from our RAID arrays:

[codesyntax lang="bash"]

reboot

[/codesyntax]

Preparing /dev/sda

If everything went well, you should now find /dev/md0 in the output of:

[codesyntax lang="bash"]

df -h

[/codesyntax]

root@ubuntu:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0         19G  985M   17G   6% /
udev            494M  4.0K  494M   1% /dev
tmpfs           201M  304K  201M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            502M     0  502M   0% /run/shm
none            100M     0  100M   0% /run/user
root@ubuntu:~#

The output of:

[codesyntax lang="bash"]

cat /proc/mdstat

[/codesyntax]

root@ubuntu:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid1 sdb5[1]
      1045952 blocks super 1.2 [2/1] [_U]
      
md0 : active raid1 sdb1[1]
      19905408 blocks super 1.2 [2/1] [_U]
      
unused devices: <none>
root@ubuntu:~#

Change the partitions type on /dev/sda to Linux raid autodetect:

[codesyntax lang="bash"]

sfdisk --change-id /dev/sda 1 fd
sfdisk --change-id /dev/sda 5 fd

[/codesyntax]

root@ubuntu:~# sfdisk --change-id /dev/sda 1 fd
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Done

root@ubuntu:~# sfdisk --change-id /dev/sda 5 fd
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Done

root@ubuntu:~#

[codesyntax lang="bash"]

fdisk -l

[/codesyntax]

root@ubuntu:~# fdisk -l

Disk /dev/sda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00059a4b

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048    39845887    19921920   fd  Linux raid autodetect
/dev/sda2        39847934    41940991     1046529    5  Extended
/dev/sda5        39847936    41940991     1046528   fd  Linux raid autodetect

Disk /dev/sdb: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *        2048    39845887    19921920   fd  Linux raid autodetect
/dev/sdb2        39847934    41940991     1046529    5  Extended
/dev/sdb5        39847936    41940991     1046528   fd  Linux raid autodetect

Disk /dev/md0: 20.4 GB, 20383137792 bytes
2 heads, 4 sectors/track, 4976352 cylinders, total 39810816 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

Disk /dev/md1: 1071 MB, 1071054848 bytes
2 heads, 4 sectors/track, 261488 cylinders, total 2091904 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table
root@ubuntu:~#

Now we can add /dev/sda1 and /dev/sda5 to the respective RAID arrays:

[codesyntax lang="bash"]

mdadm --add /dev/md0 /dev/sda1
mdadm --add /dev/md1 /dev/sda5

[/codesyntax]

root@ubuntu:~# mdadm --add /dev/md0 /dev/sda1
mdadm: added /dev/sda1
root@ubuntu:~# mdadm --add /dev/md1 /dev/sda5
mdadm: added /dev/sda5
root@ubuntu:~#

Take a look at:

[codesyntax lang="bash"]

cat /proc/mdstat

[/codesyntax]

root@ubuntu:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid1 sda5[2] sdb5[1]
      1045952 blocks super 1.2 [2/1] [_U]
          resync=DELAYED
      
md0 : active raid1 sda1[2] sdb1[1]
      19905408 blocks super 1.2 [2/1] [_U]
      [=======>.............]  recovery = 36.4% (7247872/19905408) finish=1.0min speed=205882K/sec
      
unused devices: <none>
root@ubuntu:~#

Then adjust /etc/mdadm/mdadm.conf to the new situation:

[codesyntax lang="bash"]

cp /etc/mdadm/mdadm.conf_orig /etc/mdadm/mdadm.conf
mdadm --examine --scan >> /etc/mdadm/mdadm.conf

[/codesyntax]

Display the content of /etc/mdadm/mdadm.conf:

[codesyntax lang="bash"]

cat /etc/mdadm/mdadm.conf

[/codesyntax]

root@ubuntu:~# cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

# This file was auto-generated on Tue, 23 Oct 2012 04:36:40 -0700
# by mkconf $Id$
ARRAY /dev/md/0 metadata=1.2 UUID=89e5afc0:2d741a2c:7d0f40f0:a1457396 name=ubuntu:0
ARRAY /dev/md/1 metadata=1.2 UUID=ce9163fc:4e168956:5c9050ad:68f15735 name=ubuntu:1
root@ubuntu:~#

Preparing GRUB2 (Part 2)

Now it's safe to delete /etc/grub.d/09_swraid1_setup

[codesyntax lang="bash"]

rm -f /etc/grub.d/09_swraid1_setup

[/codesyntax]

Update our GRUB2 bootloader configuration and install it again on both disks (/dev/sda and /dev/sdb)

[codesyntax lang="bash"]

update-grub
update-initramfs -u
grub-install /dev/sda
grub-install /dev/sdb

[/codesyntax]

Reboot the machine

[codesyntax lang="bash"]

reboot

[/codesyntax]

 

Xen 6.0.2 software Raid - installation procedure

Publicat la 5 April 2012 8 comentarii

This document describes how to install XenServer 6.0.2 on a node without hardware raid.

Install Software

Install XenServer 6.0.2 on /dev/sda and do NOT configure any local storage (it is easier to do that afterwards). /dev/sda should containt three partitions, please verify with the following command:

[codesyntax lang="bash"]

sgdisk -p /dev/sda

[/codesyntax]

The first partition is used for XenServer installation, the second one is used for backups during XenServer upgrades.

1. Now we are going to use /dev/sdb as the mirror disk. Clear the partition table.
[codesyntax lang="bash"]

sgdisk --zap-all /dev/sdb

[/codesyntax]

2. Install a GPT table on /dev/sdb
[codesyntax lang="bash"]

sgdisk --mbrtogpt --clear /dev/sdb

[/codesyntax]

3. Create partitions on /dev/sdb. Please note that the following commands are dependent on your installation. Copy the start and the last sectors from the /dev/sda (output of sgdisk -p /dev/sda)
[codesyntax lang="bash"]

sgdisk --new=1:34:8388641 /dev/sdb
sgdisk --typecode=1:fd00 /dev/sdb
sgdisk --attributes=1:set:2 /dev/sdb
sgdisk --new=2:8388642:16777249 /dev/sdb
sgdisk --typecode=2:fd00 /dev/sdb
sgdisk --new=3:16777250:3907029134 /dev/sdb
sgdisk --typecode=3:fd00 /dev/sdb

[/codesyntax]

4. Create RAID devices
[codesyntax lang="bash"]

mknod /dev/md0 b 9 0
mknod /dev/md1 b 9 1
mknod /dev/md2 b 9 2
mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb1
mdadm --create /dev/md1 --level=1 --raid-devices=2 missing /dev/sdb2
mdadm --create /dev/md2 --level=1 --raid-devices=2 missing /dev/sdb3

[/codesyntax]

5. Create bitmaps for each RAID device. Bitmaps slightly impact throughput but significantly reduce the rebuilt time when the array fails.
[codesyntax lang="bash"]

mdadm --grow /dev/md0 -b internal
mdadm --grow /dev/md1 -b internal
mdadm --grow /dev/md2 -b internal

[/codesyntax]

6. Format the root disk and mount it at /mnt
[codesyntax lang="bash"]

mkfs.ext3 /dev/md0
mount /dev/md0 /mnt

[/codesyntax]

7. Copy the root filesystem to the RAID array (please be patient this step may take a while).
[codesyntax lang="bash"]

cp -vxpR / /mnt

[/codesyntax]

8. Change the root filesystem in /mnt/etc/fstab to /dev/md0.
[codesyntax lang="bash"]

sed -r -i 's,LABEL=root-\w+ ,/dev/md0 ,g' /mnt/etc/fstab

[/codesyntax]

9. Install the bootloader on the second hard disk.
[codesyntax lang="bash"]

mount --bind /dev /mnt/dev
mount -t sysfs none /mnt/sys
mount -t proc none /mnt/proc
chroot /mnt /sbin/extlinux --install /boot
dd if=/mnt/usr/share/syslinux/gptmbr.bin of=/dev/sdb

[/codesyntax]

10. Make a new initrd image which contains a driver for the new root filesystem on the software RAID array.
[codesyntax lang="bash"]

chroot /mnt
mkinitrd -v -f --theme=/usr/share/splash --without-multipath /boot/initrd-`uname -r`.img `uname -r`
exit

[/codesyntax]

11. edit /mnt/boot/extlinux.conf and replace every mention of the old root filesystem (root=LABEL=xxx) with root=/dev/md0.
[codesyntax lang="bash"]

sed -r -i 's,root=LABEL=root-\w+ ,root=/dev/md0 ,g' /mnt/boot/extlinux.conf
sed -r -i 's,root=LABEL=root-\w+ ,root=/dev/md0 ,g' /boot/extlinux.conf

[/codesyntax]

12. Unmount the new root and reboot. Important: Remember to use the boot menu of your BIOS to boot from the second hard disk this time!
[codesyntax lang="bash"]

umount /mnt/proc
umount /mnt/sys
umount /mnt/dev
umount /mnt
reboot

[/codesyntax]

13. XenServer is up again, include /dev/sda in the array
[codesyntax lang="bash"]

sgdisk --typecode=1:fd00 /dev/sda
sgdisk --typecode=2:fd00 /dev/sda
sgdisk --typecode=3:fd00 /dev/sda
mdadm -a /dev/md0 /dev/sda1
mdadm -a /dev/md1 /dev/sda2
mdadm -a /dev/md2 /dev/sda3

[/codesyntax]

14. The array needs to complete its initial build/synchronisation. That will take a while.
[codesyntax lang="bash"]

watch --interval=1 cat /proc/mdstat

[/codesyntax]

15. Add /dev/md2 as a local SR to XenServer.
[codesyntax lang="bash"]

xe sr-create content-type=user device-config:device=/dev/md2 name-label="Local Storage" shared=false type=lvm

[/codesyntax]

type=ext is required if you turned on thin provisioning in the installer. Otherwise use type=lvm

Final notes:

* The second partition is used by XenServer for backups, which is why its the same size as the first partition. If you put the install CD in and boot it, an option shows up for "restore XenServer 6.0 from backup partition"

* I have created bitmaps for each raid as well. In the event of the host going down dirty, the raids can require a synch. Simply doing this is enough to add a bitmap for changed pages.

doing cat /proc/mdstat will now say something like

cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sda1[1] sdb1[0]
4193216 blocks [2/2] [UU]
bitmap: 128/128 pages [512KB], 16KB chunk

md1 : active raid1 sda2[1] sdb2[0]
4193216 blocks [2/2] [UU]
bitmap: 0/128 pages [0KB], 16KB chunk

md2 : active raid1 sda3[1] sdb3[0]
968372864 blocks [2/2] [UU]
bitmap: 0/231 pages [0KB], 2048KB chunk

* If you are installing on server which can not boot from the second disk, you must physically swap the two drives to make the machine boot off sdb and use /dev/md0 as root

* If you are going to setup a Xen 6 installation over network (via PXE) and the installation process hangs right after "Freeing unused kernel memory: 280k freed", you pass to the kernel (/tftpboot/pxelinux.cfg/main.menu) the following parameter xencons as follows:

append xenserver6/xen.gz dom0_mem=752M com1=9600,8n1 console=com1,tty --- xenserver6/vmlinuz console=tty0 console=ttyS0,9600n8 xencons=ttyS0,9600n8 answerfile=http://netboot.vendio.com/xenserver6/answers.xml install --- xenserver6/install.img

* To speed up the raid build process the following command can be used (default value is 1000):
[codesyntax lang="bash"]

echo 100000 > /proc/sys/dev/raid/speed_limit_min

[/codesyntax]

* TIP: You can use the attached script to automate the steps 1 to 9.

Good luck

The script: xen6.sh.zip

Source: http://blog.codeaddict.org/?p=5