mdadm
command, which allows creating and manipulating RAID arrays, as well as scripts and tools integrating it to the rest of the system, including the monitoring system.
sdb
disk, 4 GB, is entirely available;
sdc
disk, 4 GB, is also entirely available;
sdd
disk, only partition sdd2
(about 4 GB) is available;
sde
disk, still 4 GB, entirely available.
#
mdadm --create /dev/md0 --level=0 --raid-devices=2 /dev/sdb /dev/sdc
mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md0 started. #
mdadm --query /dev/md0
/dev/md0: 8.00GiB raid0 2 devices, 0 spares. Use mdadm --detail for more detail. #
mdadm --detail /dev/md0
/dev/md0: Version : 1.2 Creation Time : Wed May 6 09:24:34 2015 Raid Level : raid0 Array Size : 8387584 (8.00 GiB 8.59 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Wed May 6 09:24:34 2015 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Chunk Size : 512K Name : mirwiz:0 (local to host mirwiz) UUID : bb085b35:28e821bd:20d697c9:650152bb Events : 0 Number Major Minor RaidDevice State 0 8 16 0 active sync /dev/sdb 1 8 32 1 active sync /dev/sdc #
mkfs.ext4 /dev/md0
mke2fs 1.42.12 (29-Aug-2014) Creating filesystem with 2095104 4k blocks and 524288 inodes Filesystem UUID: fff08295-bede-41a9-9c6a-8c7580e520a6 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 Allocating group tables: done Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done #
mkdir /srv/raid-0
#
mount /dev/md0 /srv/raid-0
#
df -h /srv/raid-0
Filesystem Size Used Avail Use% Mounted on /dev/md0 7.9G 18M 7.4G 1% /srv/raid-0
mdadm --create
command requires several parameters: the name of the volume to create (/dev/md*
, with MD standing for Multiple Device), the RAID level, the number of disks (which is compulsory despite being mostly meaningful only with RAID-1 and above), and the physical drives to use. Once the device is created, we can use it like we'd use a normal partition, create a filesystem on it, mount that filesystem, and so on. Note that our creation of a RAID-0 volume on md0
is nothing but coincidence, and the numbering of the array doesn't need to be correlated to the chosen amount of redundancy. It's also possible to create named RAID arrays, by giving mdadm
parameters such as /dev/md/linear
instead of /dev/md0
.
#
mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdd2 /dev/sde
mdadm: Note: this array has metadata at the start and may not be suitable as a boot device. If you plan to store '/boot' on this device please ensure that your boot-loader understands md/v1.x metadata, or use --metadata=0.90 mdadm: largest drive (/dev/sdd2) exceeds size (4192192K) by more than 1% Continue creating array?
y
mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md1 started. #
mdadm --query /dev/md1
/dev/md1: 4.00GiB raid1 2 devices, 0 spares. Use mdadm --detail for more detail. #
mdadm --detail /dev/md1
/dev/md1: Version : 1.2 Creation Time : Wed May 6 09:30:19 2015 Raid Level : raid1 Array Size : 4192192 (4.00 GiB 4.29 GB) Used Dev Size : 4192192 (4.00 GiB 4.29 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Wed May 6 09:30:40 2015 State : clean, resyncing (PENDING) Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Name : mirwiz:1 (local to host mirwiz) UUID : 6ec558ca:0c2c04a0:19bca283:95f67464 Events : 0 Number Major Minor RaidDevice State 0 8 50 0 active sync /dev/sdd2 1 8 64 1 active sync /dev/sde #
mdadm --detail /dev/md1
/dev/md1: [...] State : clean [...]
mdadm
notices that the physical elements have different sizes; since this implies that some space will be lost on the bigger element, a confirmation is required.
/dev/md1
is usable, and a filesystem can be created on it, as well as some data copied on it.
mdadm
, in particular its --fail
option, allows simulating such a disk failure:
#
mdadm /dev/md1 --fail /dev/sde
mdadm: set /dev/sde faulty in /dev/md1 #
mdadm --detail /dev/md1
/dev/md1: [...] Update Time : Wed May 6 09:39:39 2015 State : clean, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 1 Spare Devices : 0 Name : mirwiz:1 (local to host mirwiz) UUID : 6ec558ca:0c2c04a0:19bca283:95f67464 Events : 19 Number Major Minor RaidDevice State 0 8 50 0 active sync /dev/sdd2 2 0 0 2 removed 1 8 64 - faulty /dev/sde
sdd
disk fail in turn, the data would be lost. We want to avoid that risk, so we'll replace the failed disk with a new one, sdf
:
#
mdadm /dev/md1 --add /dev/sdf
mdadm: added /dev/sdf #
mdadm --detail /dev/md1
/dev/md1: [...] Raid Devices : 2 Total Devices : 3 Persistence : Superblock is persistent Update Time : Wed May 6 09:48:49 2015 State : clean, degraded, recovering Active Devices : 1 Working Devices : 2 Failed Devices : 1 Spare Devices : 1 Rebuild Status : 28% complete Name : mirwiz:1 (local to host mirwiz) UUID : 6ec558ca:0c2c04a0:19bca283:95f67464 Events : 26 Number Major Minor RaidDevice State 0 8 50 0 active sync /dev/sdd2 2 8 80 1 spare rebuilding /dev/sdf 1 8 64 - faulty /dev/sde #
[...]
[...] #
mdadm --detail /dev/md1
/dev/md1: [...] Update Time : Wed May 6 09:49:08 2015 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 1 Spare Devices : 0 Name : mirwiz:1 (local to host mirwiz) UUID : 6ec558ca:0c2c04a0:19bca283:95f67464 Events : 41 Number Major Minor RaidDevice State 0 8 50 0 active sync /dev/sdd2 2 8 80 1 active sync /dev/sdf 1 8 64 - faulty /dev/sde
sde
disk is about to be removed from the array, so as to end up with a classical RAID mirror on two disks:
#
mdadm /dev/md1 --remove /dev/sde
mdadm: hot removed /dev/sde from /dev/md1 #
mdadm --detail /dev/md1
/dev/md1: [...] Number Major Minor RaidDevice State 0 8 50 0 active sync /dev/sdd2 2 8 80 1 active sync /dev/sdf
sde
disk failure had been real (instead of simulated) and the system had been restarted without removing this sde
disk, this disk could start working again due to having been probed during the reboot. The kernel would then have three physical elements, each claiming to contain half of the same RAID volume. Another source of confusion can come when RAID volumes from two servers are consolidated onto one server only. If these arrays were running normally before the disks were moved, the kernel would be able to detect and reassemble the pairs properly; but if the moved disks had been aggregated into an md1
on the old server, and the new server already has an md1
, one of the mirrors would be renamed.
/etc/mdadm/mdadm.conf
file, an example of which is listed here:
예 12.1. mdadm
configuration file
# mdadm.conf # # Please refer to mdadm.conf(5) for information about this file. # # by default (built-in), scan all partitions (/proc/partitions) and all # containers for MD superblocks. alternatively, specify devices to scan, using # wildcards if desired. DEVICE /dev/sd* # auto-create devices with Debian standard permissions CREATE owner=root group=disk mode=0660 auto=yes # automatically tag new arrays as belonging to the local system HOMEHOST <system> # instruct the monitoring daemon where to send mail alerts MAILADDR root # definitions of existing MD arrays ARRAY /dev/md0 metadata=1.2 name=mirwiz:0 UUID=bb085b35:28e821bd:20d697c9:650152bb ARRAY /dev/md1 metadata=1.2 name=mirwiz:1 UUID=6ec558ca:0c2c04a0:19bca283:95f67464 # This configuration was auto-generated on Thu, 17 Jan 2013 16:21:01 +0100 # by mkconf 3.2.5-3
DEVICE
option, which lists the devices where the system will automatically look for components of RAID volumes at start-up time. In our example, we replaced the default value, partitions containers
, with an explicit list of device files, since we chose to use entire disks and not only partitions, for some volumes.
/dev/md*
device name).
#
mdadm --misc --detail --brief /dev/md?
ARRAY /dev/md0 metadata=1.2 name=mirwiz:0 UUID=bb085b35:28e821bd:20d697c9:650152bb ARRAY /dev/md1 metadata=1.2 name=mirwiz:1 UUID=6ec558ca:0c2c04a0:19bca283:95f67464
/dev
hierarchy, so there's no risk of using them directly.
/dev
, and it can be used as any other physical partition can be (most commonly, to host a filesystem or swap space).
sdb
disk, a sdb2
partition, 4 GB;
sdc
disk, a sdc3
partition, 3 GB;
sdd
disk, 4 GB, is fully available;
sdf
disk, a sdf1
partition, 4 GB; and a sdf2
partition, 5 GB.
sdb
and sdf
are faster than the other two.
pvcreate
:
#
pvdisplay
#
pvcreate /dev/sdb2
Physical volume "/dev/sdb2" successfully created #
pvdisplay
"/dev/sdb2" is a new physical volume of "4.00 GiB" --- NEW Physical volume --- PV Name /dev/sdb2 VG Name PV Size 4.00 GiB Allocatable NO PE Size 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID 0zuiQQ-j1Oe-P593-4tsN-9FGy-TY0d-Quz31I #
for i in sdc3 sdd sdf1 sdf2 ; do pvcreate /dev/$i ; done
Physical volume "/dev/sdc3" successfully created Physical volume "/dev/sdd" successfully created Physical volume "/dev/sdf1" successfully created Physical volume "/dev/sdf2" successfully created #
pvdisplay -C
PV VG Fmt Attr PSize PFree /dev/sdb2 lvm2 --- 4.00g 4.00g /dev/sdc3 lvm2 --- 3.09g 3.09g /dev/sdd lvm2 --- 4.00g 4.00g /dev/sdf1 lvm2 --- 4.10g 4.10g /dev/sdf2 lvm2 --- 5.22g 5.22g
pvdisplay
command lists the existing PVs, with two possible output formats.
vgcreate
. We'll gather only PVs from the fast disks into a vg_critical
VG; the other VG, vg_normal
, will also include slower elements.
#
vgdisplay
No volume groups found #
vgcreate vg_critical /dev/sdb2 /dev/sdf1
Volume group "vg_critical" successfully created #
vgdisplay
--- Volume group --- VG Name vg_critical System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 1 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 2 Act PV 2 VG Size 8.09 GiB PE Size 4.00 MiB Total PE 2071 Alloc PE / Size 0 / 0 Free PE / Size 2071 / 8.09 GiB VG UUID bpq7zO-PzPD-R7HW-V8eN-c10c-S32h-f6rKqp #
vgcreate vg_normal /dev/sdc3 /dev/sdd /dev/sdf2
Volume group "vg_normal" successfully created #
vgdisplay -C
VG #PV #LV #SN Attr VSize VFree vg_critical 2 0 0 wz--n- 8.09g 8.09g vg_normal 3 0 0 wz--n- 12.30g 12.30g
vgdisplay
proposes two output formats). Note that it is quite possible to use two partitions of the same physical disk into two different VGs. Note also that we used a vg_
prefix to name our VGs, but it is nothing more than a convention.
lvcreate
command, and a slightly more complex syntax:
#
lvdisplay
#
lvcreate -n lv_files -L 5G vg_critical
Logical volume "lv_files" created #
lvdisplay
--- Logical volume --- LV Path /dev/vg_critical/lv_files LV Name lv_files VG Name vg_critical LV UUID J3V0oE-cBYO-KyDe-5e0m-3f70-nv0S-kCWbpT LV Write Access read/write LV Creation host, time mirwiz, 2015-06-10 06:10:50 -0400 LV Status available # open 0 LV Size 5.00 GiB Current LE 1280 Segments 2 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:0 #
lvcreate -n lv_base -L 1G vg_critical
Logical volume "lv_base" created #
lvcreate -n lv_backups -L 12G vg_normal
Logical volume "lv_backups" created #
lvdisplay -C
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert lv_base vg_critical -wi-a--- 1.00g lv_files vg_critical -wi-a--- 5.00g lv_backups vg_normal -wi-a--- 12.00g
lvcreate
as options. The name of the LV to be created is specified with the -n
option, and its size is generally given using the -L
option. We also need to tell the command what VG to operate on, of course, hence the last parameter on the command line.
/dev/mapper/
:
#
ls -l /dev/mapper
total 0 crw------- 1 root root 10, 236 Jun 10 16:52 control lrwxrwxrwx 1 root root 7 Jun 10 17:05 vg_critical-lv_base -> ../dm-1 lrwxrwxrwx 1 root root 7 Jun 10 17:05 vg_critical-lv_files -> ../dm-0 lrwxrwxrwx 1 root root 7 Jun 10 17:05 vg_normal-lv_backups -> ../dm-2 #
ls -l /dev/dm-*
brw-rw---T 1 root disk 253, 0 Jun 10 17:05 /dev/dm-0 brw-rw---- 1 root disk 253, 1 Jun 10 17:05 /dev/dm-1 brw-rw---- 1 root disk 253, 2 Jun 10 17:05 /dev/dm-2
#
ls -l /dev/vg_critical
total 0 lrwxrwxrwx 1 root root 7 Jun 10 17:05 lv_base -> ../dm-1 lrwxrwxrwx 1 root root 7 Jun 10 17:05 lv_files -> ../dm-0 #
ls -l /dev/vg_normal
total 0 lrwxrwxrwx 1 root root 7 Jun 10 17:05 lv_backups -> ../dm-2
#
mkfs.ext4 /dev/vg_normal/lv_backups
mke2fs 1.42.12 (29-Aug-2014) Creating filesystem with 3145728 4k blocks and 786432 inodes Filesystem UUID: b5236976-e0e2-462e-81f5-0ae835ddab1d [...] Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done #
mkdir /srv/backups
#
mount /dev/vg_normal/lv_backups /srv/backups
#
df -h /srv/backups
Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_normal-lv_backups 12G 30M 12G 1% /srv/backups #
[...]
[...] #
cat /etc/fstab
[...] /dev/vg_critical/lv_base /srv/base ext4 defaults 0 2 /dev/vg_critical/lv_files /srv/files ext4 defaults 0 2 /dev/vg_normal/lv_backups /srv/backups ext4 defaults 0 2
vg_critical
, we can grow lv_files
. For that purpose, we'll use the lvresize
command, then resize2fs
to adapt the filesystem accordingly:
#
df -h /srv/files/
Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_critical-lv_files 5.0G 4.6G 146M 97% /srv/files #
lvdisplay -C vg_critical/lv_files
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert lv_files vg_critical -wi-ao-- 5.00g #
vgdisplay -C vg_critical
VG #PV #LV #SN Attr VSize VFree vg_critical 2 2 0 wz--n- 8.09g 2.09g #
lvresize -L 7G vg_critical/lv_files
Size of logical volume vg_critical/lv_files changed from 5.00 GiB (1280 extents) to 7.00 GiB (1792 extents). Logical volume lv_files successfully resized #
lvdisplay -C vg_critical/lv_files
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert lv_files vg_critical -wi-ao-- 7.00g #
resize2fs /dev/vg_critical/lv_files
resize2fs 1.42.12 (29-Aug-2014) Filesystem at /dev/vg_critical/lv_files is mounted on /srv/files; on-line resizing required old_desc_blocks = 1, new_desc_blocks = 1 The filesystem on /dev/vg_critical/lv_files is now 1835008 (4k) blocks long. #
df -h /srv/files/
Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_critical-lv_files 6.9G 4.6G 2.1G 70% /srv/files
#
df -h /srv/base/
Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_critical-lv_base 1008M 854M 104M 90% /srv/base #
vgdisplay -C vg_critical
VG #PV #LV #SN Attr VSize VFree vg_critical 2 2 0 wz--n- 8.09g 92.00m
sdb1
partition, which was so far used outside of LVM, only contained archives that could be moved to lv_backups
. We can now recycle it and integrate it to the volume group, and thereby reclaim some available space. This is the purpose of the vgextend
command. Of course, the partition must be prepared as a physical volume beforehand. Once the VG has been extended, we can use similar commands as previously to grow the logical volume then the filesystem:
#
pvcreate /dev/sdb1
Physical volume "/dev/sdb1" successfully created #
vgextend vg_critical /dev/sdb1
Volume group "vg_critical" successfully extended #
vgdisplay -C vg_critical
VG #PV #LV #SN Attr VSize VFree vg_critical 3 2 0 wz--n- 9.09g 1.09g #
[...]
[...] #
df -h /srv/base/
Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_critical-lv_base 2.0G 854M 1.1G 45% /srv/base
sda
and sdc
. They are partitioned identically along the following scheme:
#
fdisk -l /dev/sda
Disk /dev/sda: 300 GB, 300090728448 bytes, 586114704 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x00039a9f Device Boot Start End Sectors Size Id Type /dev/sda1 * 2048 1992060 1990012 1.0G fd Linux raid autodetect /dev/sda2 1992061 3984120 1992059 1.0G 82 Linux swap / Solaris /dev/sda3 4000185 586099395 582099210 298G 5 Extended /dev/sda5 4000185 203977305 199977120 102G fd Linux raid autodetect /dev/sda6 203977306 403970490 199993184 102G fd Linux raid autodetect /dev/sda7 403970491 586099395 182128904 93G 8e Linux LVM
md0
. This mirror is directly used to store the root filesystem.
sda2
and sdc2
partitions are used as swap partitions, providing a total 2 GB of swap space. With 1 GB of RAM, the workstation has a comfortable amount of available memory.
sda5
and sdc5
partitions, as well as sda6
and sdc6
, are assembled into two new RAID-1 volumes of about 100 GB each, md1
and md2
. Both these mirrors are initialized as physical volumes for LVM, and assigned to the vg_raid
volume group. This VG thus contains about 200 GB of safe space.
sda7
and sdc7
, are directly used as physical volumes, and assigned to another VG called vg_bulk
, which therefore ends up with roughly 200 GB of space.
vg_raid
will be preserved even if one of the disks fails, which will not be the case for LVs created in vg_bulk
; on the other hand, the latter will be allocated in parallel on both disks, which allows higher read or write speeds for large files.
lv_usr
, lv_var
and lv_home
LVs on vg_raid
, to host the matching filesystems; another large LV, lv_movies
, will be used to host the definitive versions of movies after editing. The other VG will be split into a large lv_rushes
, for data straight out of the digital video cameras, and a lv_tmp
for temporary files. The location of the work area is a less straightforward choice to make: while good performance is needed for that volume, is it worth risking losing work if a disk fails during an editing session? Depending on the answer to that question, the relevant LV will be created on one VG or the other.
/usr/
can be grown painlessly.