====== RAID - mdadm (Software RAID) ====== ===== Grow RAID1 to RAID5 ===== - Existing RAID1 array is md127, array to be removed is md126 - Backup all data - Make sure the devices are unmountedsudo umount /dev/md126 - Check the detailssudo mdadm --detail /dev/md126 - Stop the devicemdadm -S /dev/md126 - Zero out the superblocks of physical devices that belong to the MD devicesudo mdadm --zero-superblock /dev/sde1 - Change the array from RAID1 to RAID5 (still only 2 drives)sudo mdadm /dev/md127 --grow --level=5 - Add the new drive to the arraysudo mdadm /dev/md127 --add /dev/sde1 - Start the sync for all 3 devicessudo mdadm --grow /dev/md127 --raid-devices 3 - Watch the progress withwatch -n 15 -d cat /proc/mdstat * [[http://stevesubuntutweaks.blogspot.com/2015/06/converting-raid1-array-to-raid5-using.html|Converting RAID1 array to RAID5 using the mdadm grow command]] * [[https://superuser.com/questions/606751/how-to-remove-linux-software-raid-md-device|How to remove Linux software RAID MD device?]] * [[https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-ubuntu-16-04|How To Create RAID Arrays with mdadm on Ubuntu 16.04 ]] * [[https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-ubuntu-16-04|How To Manage RAID Arrays with mdadm on Ubuntu 16.04]] * [[http://tldp.org/HOWTO/Software-RAID-HOWTO-6.html|Software RAID How-to]] * [[https://unix.stackexchange.com/questions/190264/disassemble-a-raid-1-arragement-without-removing-reinstalling-the-system|Disassemble a RAID 1 arragement without removing/reinstalling the system]] ==== Desired configuration of the disks ==== ^ hdd ^ partition ^ raid1 ^ lvm ^ mount point ^ | sda | sda1 | md0 | - | / | | sdb | sdb1 | md0 | | | | sda | sda2 | - | - | swap | | sdb | sdb2 | - | | swap | | sdc | sdc1 | md1 | vg1 | /home, /srv, | | sdd | sdd1 | md1 | vg1 | /media | | sde | sde1 | md2 | vg1 | | | sdf | sdf1 | md2 | vg1 | | | sdg | sdg1 | md3 | vg2 | /backup | | sdh | sdh1 | md3 | vg2 | /backup | see also [[hw:machines#apollo|Hardware configuration]] ==== Replace a failed RAID disk ==== Replace the disk, then copy the partition information from the good disk, randomize the UUID, and re-read the partition information into the system. First, install gdisk from the Debian Universe repositories. apt-get install gdisk sgdisk -R=/dev/sdb /dev/sda sgdisk -G /dev/sdb partprobe Taken from [[http://askubuntu.com/questions/57908/how-can-i-quickly-copy-a-gpt-partition-scheme-from-one-hard-drive-to-another|How can I quickly copy a GPT partition scheme from one hard drive to another?]] Other resources on [[http://en.wikipedia.org/wiki/Mdadm|mdadm]] and [[http://askubuntu.com/questions/63980/how-do-i-rename-an-mdadm-raid-array|How do I rename an mdadm raid array?]]: * [[https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID|Recovering a failed software RAID]] * [[https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Cluster_Logical_Volume_Manager/VG_activate.html|Activating and deactivating volume groups]] * [[https://raid.wiki.kernel.org/index.php/RAID_Recovery|RAID Recovery]] * [[http://www.cyberciti.biz/faq/linux-creating-software-raid-one-arrays/|Linux Create Software RAID 1 (Mirror) Array]] * [[http://www.cyberciti.biz/faq/linux-server-rename-an-mdadm-raid-array/|Linux Rename a RAID Array From md0 to md2]] * [[https://ubuntuforums.org/archive/index.php/t-1947275.html|Mdadm missing superblock]] Setup configuration: mdadm -Es >> /etc/mdadm/mdadm.conf ==== Install Base ==== - Install Ubuntu Hardy (8.04) Alternate Install: F6-F6-Expert Mode - Install Raid1 with MD package activated and assign mount points ==== Check wether all volumes get mounted during system boot ==== to check wether root and swap are mounted, enter: mount free -m -t to check mismatching uuid's, enter: ls -la /dev/disk/by-uuid cat /etc/fstab to fix, do: vim /etc/fstab replace the uuid's found in fstab with the ones found in /dev/disk. Make sure you copy the correct uuid (md0, md1) to the respective entry in fstab. ===== Resync ===== Most Debian and Debian-derived distributions create a cron job which issues an array check at 0106 hours each first Sunday of the month in /etc/cron.d/mdadm. This task appears as resync in /proc/mdstat and syslog. So if you suddenly see RAID-resyncing for no apparent reason, this might be a place to take a look. Normally the kernel will throttle the resync activity (c.f. [[https://en.wikipedia.org/wiki/Nice_%28Unix%29|nice]]) to avoid impacting the raid device performance. However, it is a good idea to manage the resync parameters to get optimal performance. ==== Raid 1, 5, 6 ==== === Rebuild speed === * Get current system values:sudo sysctl dev.raid.speed_limit_min sudo sysctl dev.raid.speed_limit_max * Default system values on Debian 10:dev.raid.speed_limit_min = 1000 dev.raid.speed_limit_max = 200,000 * Reduce max limit to make server more responsive during resync (2021-12-05):sudo sysctl -w dev.raid.speed_limit_min=10,000 sudo sysctl -w dev.raid.speed_limit_max=100,000 === read-ahead === * Get current read-ahead (in 512-byte sectors) per Raid device (default value is 512 on Debian 10):blockdev --getra /dev/mdX * Set to 32 MB:blockdev --setra 65536 /dev/mdX * Set to 65536 on a server with 32GB memory, 32768 on a server with 8GB memory (2021-12-05) === Disable NCQ === * Get NCQ depth on each physical Drive in Raid (default value is 31):cat /sys/block/sdX/device/queue_depth * Disable NCQ:echo 1 > /sys/block/sdX/device/queue_depth ==== Raid 5, 6 only ==== === stripe_cache_size === It records the size (in pages per device) of the stripe cache which is used for synchronising all write operations to the array and all read operations if the array is degraded. The default is 256 which equals to 3MB memory consumption. Valid values are 17 to 32768. Make sure your system has enough memory available: memory_consumed = system_page_size * nr_disks * stripe_cache_size. * Find system page size, on Debian 10 this is 4096:getconf PAGESIZE * Set to 384MB memory consumption on a 3 disk:sudo echo 32768> /sys/block/md0/md/stripe_cache_size * Set to 32768 on a server with 32 GB memory, set to 16384 on a server with 8 GB memory (2021-12-05) ===== Prepare RAID with single disk ===== ==== Prepare new disk ==== - If the new disk contains partitions - Stop any Raid partitions withmdadm --stop /dev/md1 mdadm --remove /dev/md1 - Remove the superblocksmdadm --zero-superblock /dev/sdX1 - Remove existing partitions with ''fdisk /dev/sdX'' - Create a new partition utilizing the full disk space. When asked, remove the existing signature. Change partition type to ''Linux RAID''sudo fdisk /dev/sdX Command (m for help): d Command (m for help): n Command (m for help): t,29 - Create the RAIDmdadm --create /dev/mdX --level=raid1 --raid-devices=2 /dev/sdX1 missing - Check the RAID was createdcat /proc/mdstat ls /dev/md* ==== Links ==== * [[https://unix.stackexchange.com/questions/63928/can-i-create-a-software-raid-1-with-one-device|Can I create a software RAID 1 with one device]] * [[https://wiki.archlinux.org/title/Convert_a_single_drive_system_to_RAID|Convert a single drive system to RAID]] * [[https://bobcares.com/blog/removal-of-mdadm-raid-devices/|Removal of mdadm RAID Devices – How to do it quickly?]] ===== Move RAID to a new machine ===== - Scan for the old raid diskssudo mdadm --assemble --scan - Mount the raid manually to confirmblkid sudo mount /dev/md0 /mnt - Append info to mdadm.confmdadm --detail --scan >> /etc/mdadm/mdadm.conf - Update initramfsupdate-initramfs -u - Copy entire disk to new RAID (add ''-z'' for network transfers)rsync --progress -arHAX - Check size of directorydu -sh ==== Troubleshooting ==== * Make sure the output of "mdadm --detail --scan" matches your /etc/mdadm/mdadm.conf * Examine /etc/fstab ==== Links ==== * [[https://serverfault.com/questions/32709/how-do-i-move-a-linux-software-raid-to-a-new-machine|How do I move a Linux software RAID to a new machine?]] * [[https://unix.stackexchange.com/questions/23879/using-mdadm-examine-to-write-mdadm-conf/52935#52935|Using mdadm --examine to write mdadm.conf]] * [[https://askubuntu.com/questions/729370/can-i-transfer-my-mdadm-software-raid-to-a-new-system-in-case-of-hardware-failur|Can I transfer my mdadm Software raid to a new system in case of hardware failure?]] * [[https://unix.stackexchange.com/questions/324662/how-to-bring-up-an-inactive-mdadm-raid-after-a-reboot-while-adding-a-drive-to-a|How to bring up an inactive mdadm RAID after a reboot while adding a drive to a raid-6?]] * [[https://support.hpe.com/hpesc/public/docDisplay?docId=c02101717&docLocale=en_US|RedHat Enterprise Linux 5: How to Activate a Software Raid Device Manually]] * [[https://www.diskinternals.com/raid-recovery/how-to-remove-software-raid-with-mdadm/|Remove mdadm RAID Devices – How to do it?]] * [[https://www.oreilly.com/library/view/managing-raid-on/9780596802035/ch04s06s01.html|mdadm.conf]] ===== Links ===== * [[https://raid.wiki.kernel.org/index.php/Resync|Resync]] * [[https://www.cyberciti.biz/tips/linux-raid-increase-resync-rebuild-speed.html#:~:text=The%20%2Fproc%2Fsys%2Fdev,The%20default%20is%201000.|5 Tips To Speed Up Linux Software Raid Rebuilding And Re-syncing]] * [[https://bobcares.com/blog/raid-resync/|RAID resync – Best practices]] * [[https://raid.wiki.kernel.org/index.php/A_guide_to_mdadm#Upgrading_a_mirror_raid_to_a_parity_raid|A guide to mdadm]] * [[https://www.howtoforge.com/how-to-resize-raid-partitions-shrink-and-grow-software-raid|How To Resize RAID Partitions (Shrink & Grow) (Software RAID)]] * [[http://ewams.net/?view=Formatting_a_RAID_volume_with_ext4|Formatting a RAID volume with ext4]] * [[https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-ubuntu-16-04|How To Create RAID Arrays with mdadm on Ubuntu 16.04 ]] * [[https://www.howtoforge.com/how-to-resize-raid-partitions-shrink-and-grow-software-raid|How To Resize RAID Partitions (Shrink & Grow) (Software RAID)]] * [[http://ewams.net/?view=Formatting_a_RAID_volume_with_ext4|Formatting a RAID volume with ext4]] * [[https://superuser.com/questions/19326/how-to-wipe-free-disk-space-in-linux|How to wipe free disk space in Linux?]] * [[https://www.tldp.org/HOWTO/Partition/setting_up_swap.html|Setting up swap space]]