====== RAID - mdadm (Software RAID) ====== ===== (Re)build RAID ===== ==== Replace a failing or failed disk ==== - Check the RAID was createdcat /proc/mdstat - Check disk statussmartctl -H /dev/sdX - Find serial numbersmartctl -i /dv/sdX - If the new disk contains partitions - Stop any Raid partitions withmdadm --stop /dev/md1 - Remove the superblocksmdadm --zero-superblock /dev/sdX1 - Remove existing partitions with ''gdisk /dev/sdX''Command (m for help): d - Create new RAID partition (if asked remove the existing signature)Command (m for help): d Command (m for help): n Command (m for help): t,fd00 - Add the new drive to the RAIDmdadm /dev/md0 --add /dev/sdc1 - If the system does not need to use the disks during resync you may want to (temporarily) increase the sync speed:echo 1000000 > /proc/sys/dev/raid/speed_limit_maxBefore doing this check the current sync speed:cat /proc/sys/dev/raid/speed_limit_max - If the RAID is incomplete, rebuilding (resyncing) of the RAID starts instantly. If the RAID is complete including the bad drive, and you just added a spare drive, you can proceed as follows (requires mdadm 3.3+ and a 3.2+ kernel)mdadm /dev/md0 --replace /dev/sdX1 --with /dev/sdc1sdX1 is the device you want to replace, sdc1 is the preferred device to do so and must be declared as a spare on your array. The ''--with'' option is optional, if not specified, any available spare will be used. After resyncing the RAID the replaced drive will be marked as failed.\\ \\ - Remove the replaced disk which is marked as failed after resyncing has completedmdadm --manage /dev/md0 --remove /dev/sdX1 - Compare ouput of ''mdadm -Es'' to the contents of ''/etc/mdadm/mdadm/conf''mdadm -Es >> /etc/mdadm/mdadm.conf * [[https://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-yet-failed-disk-in-a-linux-raid5-array#using-mdadm-3.3|How to safely replace a not-yet-failed disk in a Linux RAID5 array?]] * [[https://delightlylinux.wordpress.com/2020/12/22/how-to-remove-a-drive-from-a-raid-array/|mdadm: How to Remove a Drive From a RAID Array]] * [[http://en.wikipedia.org/wiki/Mdadm|mdadm]] * [[https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID|Recovering a failed software RAID]] * [[https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Cluster_Logical_Volume_Manager/VG_activate.html|Activating and deactivating volume groups]] * [[https://raid.wiki.kernel.org/index.php/RAID_Recovery|RAID Recovery]] * [[http://www.cyberciti.biz/faq/linux-creating-software-raid-one-arrays/|Linux Create Software RAID 1 (Mirror) Array]] * [[http://www.cyberciti.biz/faq/linux-server-rename-an-mdadm-raid-array/|Linux Rename a RAID Array From md0 to md2]] * [[https://ubuntuforums.org/archive/index.php/t-1947275.html|Mdadm missing superblock]] * [[http://askubuntu.com/questions/63980/how-do-i-rename-an-mdadm-raid-array|How do I rename an mdadm raid array?]] * [[http://askubuntu.com/questions/57908/how-can-i-quickly-copy-a-gpt-partition-scheme-from-one-hard-drive-to-another|How can I quickly copy a GPT partition scheme from one hard drive to another?]] * [[https://www.dlford.io/linux-mdraid-disk-replacement-procedure/|7 Steps to Safely Replace a Drive in a Linux MD RAID Array]] * [[https://rabexc.org/posts/mdadm-replace|Replacing and resizing a linux software raid]] * [[https://rabexc.org/posts/resizing-filesystem|Resizing an encrypted filesystem with LVM on Linux]] * [[https://unix.stackexchange.com/questions/320957/extend-a-luks-encrypted-partition-to-fill-disk|Extend a LUKS encrypted partition to fill disk]] * [[https://superuser.com/questions/625722/reducing-linux-software-raid-rebuild-speed#comments-625724|Reducing Linux Software RAID Rebuild Speed?]] ==== Check wether all volumes get mounted during system boot ==== * to check wether root and swap are mounted, enter:mount free -m -t * to check mismatching uuid's, enter:blkid ls -la /dev/disk/by-uuid cat /etc/fstab * to fix, replace the uuid's found in ''/etc/fstab'' with the ones found in ''/dev/disk''. Make sure you copy the correct uuid (md0, md1) to the respective entry in fstab.vim /etc/fstab ===== Resync ===== Most Debian and Debian-derived distributions create a cron job which issues an array check at 0106 hours each first Sunday of the month in /etc/cron.d/mdadm. This task appears as resync in /proc/mdstat and syslog. So if you suddenly see RAID-resyncing for no apparent reason, this might be a place to take a look. Normally the kernel will throttle the resync activity (c.f. [[https://en.wikipedia.org/wiki/Nice_%28Unix%29|nice]]) to avoid impacting the raid device performance. However, it is a good idea to manage the resync parameters to get optimal performance. ==== Raid 1, 5, 6 ==== === Rebuild speed === * Get current system values:sudo sysctl dev.raid.speed_limit_min sudo sysctl dev.raid.speed_limit_max * Default system values on Debian 10:dev.raid.speed_limit_min = 1000 dev.raid.speed_limit_max = 200,000 * Reduce max limit to make server more responsive during resync (2021-12-05):sudo sysctl -w dev.raid.speed_limit_min=10,000 sudo sysctl -w dev.raid.speed_limit_max=100,000 === read-ahead === * Get current read-ahead (in 512-byte sectors) per Raid device (default value is 512 on Debian 10):blockdev --getra /dev/mdX * Set to 32 MB:blockdev --setra 65536 /dev/mdX * Set to 65536 on a server with 32GB memory, 32768 on a server with 8GB memory (2021-12-05) === Disable NCQ === * Get NCQ depth on each physical Drive in Raid (default value is 31):cat /sys/block/sdX/device/queue_depth * Disable NCQ:echo 1 > /sys/block/sdX/device/queue_depth ==== Raid 5, 6 only ==== === stripe_cache_size === It records the size (in pages per device) of the stripe cache which is used for synchronising all write operations to the array and all read operations if the array is degraded. The default is 256 which equals to 3MB memory consumption. Valid values are 17 to 32768. Make sure your system has enough memory available: memory_consumed = system_page_size * nr_disks * stripe_cache_size. * Find system page size, on Debian 10 this is 4096:getconf PAGESIZE * Set to 384MB memory consumption on a 3 disk:sudo echo 32768> /sys/block/md0/md/stripe_cache_size * Set to 32768 on a server with 32 GB memory, set to 16384 on a server with 8 GB memory (2021-12-05) ===== Prepare RAID with single disk ===== ==== Prepare new disk ==== - If the new disk contains partitions - Stop any Raid partitions withmdadm --stop /dev/md1 mdadm --remove /dev/md1 - Remove the superblocksmdadm --zero-superblock /dev/sdX1 - Remove existing partitions with ''fdisk /dev/sdX'' - Create a new partition utilizing the full disk space. When asked, remove the existing signature. Change partition type to ''Linux RAID''sudo fdisk /dev/sdX Command (m for help): d Command (m for help): n Command (m for help): t,29 - Create the RAIDmdadm --create /dev/mdX --level=raid1 --raid-devices=2 /dev/sdX1 missing - Check the RAID was createdcat /proc/mdstat ls /dev/md* - Add a second diskmdadm --manage /dev/mdX --add /dev/sdX1 ==== Links ==== * [[https://unix.stackexchange.com/questions/63928/can-i-create-a-software-raid-1-with-one-device|Can I create a software RAID 1 with one device]] * [[https://wiki.archlinux.org/title/Convert_a_single_drive_system_to_RAID|Convert a single drive system to RAID]] * [[https://bobcares.com/blog/removal-of-mdadm-raid-devices/|Removal of mdadm RAID Devices – How to do it quickly?]] * [[https://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array|Replacing A Failed Hard Drive In A Software RAID1 Array]] ===== Move RAID to a new machine ===== - Scan for the old raid diskssudo mdadm --assemble --scan - Mount the raid manually to confirmblkid sudo mount /dev/md0 /mnt - Append info to mdadm.confmdadm --detail --scan >> /etc/mdadm/mdadm.conf - Update initramfsupdate-initramfs -u ==== Troubleshooting ==== * Make sure the output of "mdadm --detail --scan" matches your /etc/mdadm/mdadm.conf * Examine /etc/fstab ==== Links ==== * [[https://serverfault.com/questions/32709/how-do-i-move-a-linux-software-raid-to-a-new-machine|How do I move a Linux software RAID to a new machine?]] * [[https://unix.stackexchange.com/questions/23879/using-mdadm-examine-to-write-mdadm-conf/52935#52935|Using mdadm --examine to write mdadm.conf]] * [[https://askubuntu.com/questions/729370/can-i-transfer-my-mdadm-software-raid-to-a-new-system-in-case-of-hardware-failur|Can I transfer my mdadm Software raid to a new system in case of hardware failure?]] ===== Increase drive capacity in a RAID -> LVM -> CRYPT setup ===== ==== Replace drives in RAID ==== - Follow [[#Replace_a_failing_or_failed_disk|Replace a failing or failed disk]] at the top of this page - Resize the array to the maximum supported by the underlying partitionsmdadm --grow /dev/md5 -z max - Follow the progress withwatch -d cat /proc/mdstat ==== Increase LVM ==== - Check size of physical volume withpvdisplay - Increase physical volume to utilize all available spacepvresize /dev/mdX - Increase logical volume to utilize all available spacelvextend -l +100%FREE /dev// ==== Increase LUKS ==== - Inform LUKS to utilize all available space, you need the backup key to do thiscryptsetup resize /dev/mapper//_crypt ==== Increase file system ==== - On-line resize file systemresize2fs -p /dev/mapper//_crypt ===== Links ===== * [[https://raid.wiki.kernel.org/index.php/Resync|Resync]] * [[https://www.cyberciti.biz/tips/linux-raid-increase-resync-rebuild-speed.html#:~:text=The%20%2Fproc%2Fsys%2Fdev,The%20default%20is%201000.|5 Tips To Speed Up Linux Software Raid Rebuilding And Re-syncing]] * [[https://bobcares.com/blog/raid-resync/|RAID resync – Best practices]] * [[https://raid.wiki.kernel.org/index.php/A_guide_to_mdadm#Upgrading_a_mirror_raid_to_a_parity_raid|A guide to mdadm]] * [[https://www.howtoforge.com/how-to-resize-raid-partitions-shrink-and-grow-software-raid|How To Resize RAID Partitions (Shrink & Grow) (Software RAID)]] * [[http://ewams.net/?view=Formatting_a_RAID_volume_with_ext4|Formatting a RAID volume with ext4]] * [[https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-ubuntu-16-04|How To Create RAID Arrays with mdadm on Ubuntu 16.04 ]] * [[https://www.howtoforge.com/how-to-resize-raid-partitions-shrink-and-grow-software-raid|How To Resize RAID Partitions (Shrink & Grow) (Software RAID)]] * [[http://ewams.net/?view=Formatting_a_RAID_volume_with_ext4|Formatting a RAID volume with ext4]] * [[https://superuser.com/questions/19326/how-to-wipe-free-disk-space-in-linux|How to wipe free disk space in Linux?]] * [[https://www.tldp.org/HOWTO/Partition/setting_up_swap.html|Setting up swap space]]