Differences

This shows you the differences between two versions of the page.

--- deb13:tuning [2026/08/16 14:04] – Bernard Condrau
+++ deb13:tuning [2026/08/16 14:10] (current) – Bernard Condrau
@@ Line 1: / Line 1: @@
 ====== Tuning ======
-===== (Re)build RAID =====
-==== Replace a failing or failed disk ====
-  - Check the RAID was created<code>cat /proc/mdstat</code>
-  - Check disk status<code>smartctl -H /dev/sdX</code>
-  - Find serial number<code>smartctl -i /dv/sdX</code>
-  - If the new disk contains partitions
-    - Stop any Raid partitions with<file>mdadm --stop /dev/md1</file>
-    - Remove the superblocks<file>mdadm --zero-superblock /dev/sdX1</file>
-    - Remove existing partitions with ''gdisk /dev/sdX''<file>Command (m for help): d</file>
-    - Create new RAID partition (if asked remove the existing signature)<code>Command (m for help): d
-Command (m for help): n
-Command (m for help): t,fd00</code>
-  - Add the new drive to the RAID<code>mdadm /dev/md0 --add /dev/sdc1</code>
-  - If the system does not need to use the disks during resync you may want to (temporarily) increase the sync speed:<file>echo 1000000 > /proc/sys/dev/raid/speed_limit_max</file>Before doing this check the current sync speed:<code>cat /proc/sys/dev/raid/speed_limit_max</code>
-  - If the RAID is incomplete, rebuilding (resyncing) of the RAID starts instantly. If the RAID is complete including the bad drive, and you just added a spare drive, you can proceed as follows (requires mdadm 3.3+ and a 3.2+ kernel)<file>mdadm /dev/md0 --replace /dev/sdX1 --with /dev/sdc1</file>sdX1 is the device you want to replace, sdc1 is the preferred device to do so and must be declared as a spare on your array. The ''--with'' option is optional, if not specified, any available spare will be used. After resyncing the RAID the replaced drive will be marked as failed.
-  - Remove the replaced disk which is marked as failed after resyncing has completed<code>mdadm --manage /dev/md0 --remove /dev/sdX1</code>
-  - Remove a disk from the array which was physically removed before<code>mdadm --remove /dev/md0</code>
-  - Compare ouput of ''mdadm -Es'' to the contents of ''/etc/mdadm/mdadm/conf''<code>mdadm -Es >> /etc/mdadm/mdadm.conf</code>
-  * [[https://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-yet-failed-disk-in-a-linux-raid5-array#using-mdadm-3.3|How to safely replace a not-yet-failed disk in a Linux RAID5 array?]]
-  * [[https://delightlylinux.wordpress.com/2020/12/22/how-to-remove-a-drive-from-a-raid-array/|mdadm: How to Remove a Drive From a RAID Array]]
-  * [[http://en.wikipedia.org/wiki/Mdadm|mdadm]]
-  * [[https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID|Recovering a failed software RAID]]
-  * [[https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Cluster_Logical_Volume_Manager/VG_activate.html|Activating and deactivating volume groups]]
-  * [[https://raid.wiki.kernel.org/index.php/RAID_Recovery|RAID Recovery]]
-  * [[http://www.cyberciti.biz/faq/linux-creating-software-raid-one-arrays/|Linux Create Software RAID 1 (Mirror) Array]]
-  * [[http://www.cyberciti.biz/faq/linux-server-rename-an-mdadm-raid-array/|Linux Rename a RAID Array From md0 to md2]]
-  * [[https://ubuntuforums.org/archive/index.php/t-1947275.html|Mdadm missing superblock]]
-  * [[http://askubuntu.com/questions/63980/how-do-i-rename-an-mdadm-raid-array|How do I rename an mdadm raid array?]]
-  * [[http://askubuntu.com/questions/57908/how-can-i-quickly-copy-a-gpt-partition-scheme-from-one-hard-drive-to-another|How can I quickly copy a GPT partition scheme from one hard drive to another?]]
-  * [[https://www.dlford.io/linux-mdraid-disk-replacement-procedure/|7 Steps to Safely Replace a Drive in a Linux MD RAID Array]]
-  * [[https://rabexc.org/posts/mdadm-replace|Replacing and resizing a linux software raid]]
-  * [[https://rabexc.org/posts/resizing-filesystem|Resizing an encrypted filesystem with LVM on Linux]]
-  * [[https://unix.stackexchange.com/questions/320957/extend-a-luks-encrypted-partition-to-fill-disk|Extend a LUKS encrypted partition to fill disk]]
-  * [[https://superuser.com/questions/625722/reducing-linux-software-raid-rebuild-speed#comments-625724|Reducing Linux Software RAID Rebuild Speed?]]
-==== Check wether all volumes get mounted during system boot ====
-  * to check wether root and swap are mounted, enter:<code>mount
-free -m -t</code>
-  * to check mismatching uuid's, enter:<code>blkid
-ls -la /dev/disk/by-uuid
-cat /etc/fstab</code>
-  * to fix, replace the uuid's found in ''/etc/fstab'' with the ones found in ''/dev/disk''. Make sure you copy the correct uuid (md0, md1) to the respective entry in fstab.<code>vim /etc/fstab</code>
-===== Resync =====
-Most Debian and Debian-derived distributions create a cron job which issues an array check at 0106 hours each first Sunday of the month in /etc/cron.d/mdadm. This task appears as resync in /proc/mdstat and syslog. So if you suddenly see RAID-resyncing for no apparent reason, this might be a place to take a look.
-Normally the kernel will throttle the resync activity (c.f. [[https://en.wikipedia.org/wiki/Nice_%28Unix%29|nice]]) to avoid impacting the raid device performance.
-However, it is a good idea to manage the resync parameters to get optimal performance.
-==== Raid 1, 5, 6 ====
-=== Rebuild speed ===
-  * Get current system values:<code>sudo sysctl dev.raid.speed_limit_min
-sudo sysctl dev.raid.speed_limit_max</code>
-  * Default system values on Debian 10:<code>dev.raid.speed_limit_min = 1000
-dev.raid.speed_limit_max = 200,000</code>
-  * Reduce max limit to make server more responsive during resync (2021-12-05):<code>sudo sysctl -w dev.raid.speed_limit_min=10,000
-sudo sysctl -w dev.raid.speed_limit_max=100,000</code>
-=== read-ahead ===
-  * Get current read-ahead (in 512-byte sectors) per Raid device (default value is 512 on Debian 10):<code>blockdev --getra /dev/mdX</code>
-  * Set to 32 MB:<code>blockdev --setra 65536 /dev/mdX</code>
-  * Set to 65536 on a server with 32GB memory, 32768 on a server with 8GB memory (2021-12-05)
-=== Disable NCQ ===
-  * Get NCQ depth on each physical Drive in Raid (default value is 31):<code>cat /sys/block/sdX/device/queue_depth</code>
-  * Disable NCQ:<code>echo 1 > /sys/block/sdX/device/queue_depth</code>
-==== Raid 5, 6 only ====
-=== stripe_cache_size ===
-It records the size (in pages per device) of the stripe cache which is used for synchronising all write operations to the array and all read operations if the array is degraded. The default is 256 which equals to 3MB memory consumption.  Valid values are 17 to 32768. Make sure your system has enough memory available: memory_consumed = system_page_size * nr_disks * stripe_cache_size.
-  * Find system page size, on Debian 10 this is 4096:<code>getconf PAGESIZE</code>
-  * Set to 384MB memory consumption on a 3 disk:<code>sudo echo 32768> /sys/block/md0/md/stripe_cache_size</code>
-  * Set to 32768 on a server with 32 GB memory, set to 16384 on a server with 8 GB memory (2021-12-05)
-===== Prepare RAID with single disk =====
-==== Prepare new disk ====
-  - If the new disk contains partitions
-    - Stop any Raid partitions with<code>mdadm --stop /dev/md1
-mdadm --remove /dev/md1</code>
-    - Remove the superblocks<code>mdadm --zero-superblock /dev/sdX1</code>
-    - Remove existing partitions with ''fdisk /dev/sdX''
-  - Create a new partition utilizing the full disk space. When asked, remove the existing signature. Change partition type to ''Linux RAID''<code>sudo fdisk /dev/sdX
-Command (m for help): d
-Command (m for help): n
-Command (m for help): t,29</code>
-  - Create the RAID<code>mdadm --create /dev/mdX --level=raid1 --raid-devices=2 /dev/sdX1 missing</code>
-  - Check the RAID was created<code>cat /proc/mdstat
-ls /dev/md*</code>
-  - Add a second disk<code>mdadm --manage /dev/mdX --add /dev/sdX1</code>
-==== Links ====
-  * [[https://unix.stackexchange.com/questions/63928/can-i-create-a-software-raid-1-with-one-device|Can I create a software RAID 1 with one device]]
-  * [[https://wiki.archlinux.org/title/Convert_a_single_drive_system_to_RAID|Convert a single drive system to RAID]]
-  * [[https://bobcares.com/blog/removal-of-mdadm-raid-devices/|Removal of mdadm RAID Devices – How to do it quickly?]]
-  * [[https://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array|Replacing A Failed Hard Drive In A Software RAID1 Array]]
-===== Move RAID to a new machine =====
-  - Scan for the old raid disks<code>sudo mdadm --assemble --scan</code>
-  - Mount the raid manually to confirm<code>blkid
-sudo mount /dev/md0 /mnt</code>
-  - Append info to mdadm.conf<code>mdadm --detail --scan >> /etc/mdadm/mdadm.conf</code>
-  - Update initramfs<code>update-initramfs -u</code>
-==== Troubleshooting ====
-  * Make sure the output of "mdadm --detail --scan" matches your /etc/mdadm/mdadm.conf
-  * Examine /etc/fstab
-==== Links ====
-  * [[https://serverfault.com/questions/32709/how-do-i-move-a-linux-software-raid-to-a-new-machine|How do I move a Linux software RAID to a new machine?]]
-  * [[https://unix.stackexchange.com/questions/23879/using-mdadm-examine-to-write-mdadm-conf/52935#52935|Using mdadm --examine to write mdadm.conf]]
-  * [[https://askubuntu.com/questions/729370/can-i-transfer-my-mdadm-software-raid-to-a-new-system-in-case-of-hardware-failur|Can I transfer my mdadm Software raid to a new system in case of hardware failure?]]
-===== Increase drive capacity in a RAID -> LVM -> CRYPT setup =====
-==== Replace drives in RAID ====
-  - Follow [[#Replace_a_failing_or_failed_disk|Replace a failing or failed disk]] at the top of this page
-  - Resize the array to the maximum supported by the underlying partitions<code>mdadm --grow /dev/md5 -z max</code>
-  - Follow the progress with<code>watch -d cat /proc/mdstat</code>
-==== Increase LVM ====
-  - Check size of physical volume with<code>pvdisplay</code>
-  - Increase physical volume to utilize all available space<code>pvresize /dev/mdX</code>
-  - Increase logical volume to utilize all available space<code>lvextend -l +100%FREE /dev/<volume_group>/<logical_volume></code>
-==== Increase LUKS ====
-  - Inform LUKS to utilize all available space, you need the backup key to do this<code>cryptsetup resize /dev/mapper/<volume_group>/<logical_volume>_crypt</code>
 ===== Swap behavior =====
   * Set swappiness to 10 in ''/etc/sysctl.conf'' or ''/etc/sysctl.d/99-webserver-tuning.conf'':<code>vm.swappiness=10</code>
@@ Line 134: / Line 5: @@
 cat /proc/sys/vm/swappiness</code>
-===== (Re)build RAID =====
+===== Network =====
+==== Test performance ====
+  * Install iperf on client and host machine<code>sudo apt install iperf3 iotop</code>
+  * Open firewall on host<code>ufw allow 5201</code>
+  * Test network from client to host<code>iperf3 -s # on host
+iperf3 -c <host> # on client</code>
+  * Test disk access<code>iotop</code>
-==== Replace a failing or failed disk ====
+==== Speedtest ====
-  - Check the RAID was created<code>cat /proc/mdstat</code>
+  * Remove speedtest-cli if you have previously installed it:<code>sudo apt remove speedtest-cli</code>
-  - Check disk status<code>smartctl -H /dev/sdX</code>
+  * Install python-pip on Python v3:<code>sudo apt install python3-pip</code>
-  - Find serial number<code>smartctl -i /dv/sdX</code>
+  * Install and run speedtest:<code>sudo pip3 install speedtest_cli
-  - If the new disk contains partitions
+speedtest-cli</code>
-    - Stop any Raid partitions with<file>mdadm --stop /dev/md1</file>
-    - Remove the superblocks<file>mdadm --zero-superblock /dev/sdX1</file>
-    - Remove existing partitions with ''gdisk /dev/sdX''<file>Command (m for help): d</file>
-    - Create new RAID partition (if asked remove the existing signature)<code>Command (m for help): d
-Command (m for help): n
-Command (m for help): t,fd00</code>
-  - Add the new drive to the RAID<code>mdadm /dev/md0 --add /dev/sdc1</code>
-  - If the system does not need to use the disks during resync you may want to (temporarily) increase the sync speed:<file>echo 1000000 > /proc/sys/dev/raid/speed_limit_max</file>Before doing this check the current sync speed:<code>cat /proc/sys/dev/raid/speed_limit_max</code>
-  - If the RAID is incomplete, rebuilding (resyncing) of the RAID starts instantly. If the RAID is complete including the bad drive, and you just added a spare drive, you can proceed as follows (requires mdadm 3.3+ and a 3.2+ kernel)<file>mdadm /dev/md0 --replace /dev/sdX1 --with /dev/sdc1</file>sdX1 is the device you want to replace, sdc1 is the preferred device to do so and must be declared as a spare on your array. The ''--with'' option is optional, if not specified, any available spare will be used. After resyncing the RAID the replaced drive will be marked as failed.
-  - Remove the replaced disk which is marked as failed after resyncing has completed<code>mdadm --manage /dev/md0 --remove /dev/sdX1</code>
-  - Remove a disk from the array which was physically removed before<code>mdadm --remove /dev/md0</code>
-  - Compare ouput of ''mdadm -Es'' to the contents of ''/etc/mdadm/mdadm/conf''<code>mdadm -Es >> /etc/mdadm/mdadm.conf</code>
+==== CPU and IO priorities ====
+  * IO priority<code>ionice -c scheduling_class -n priority_nice_value command
+ionice -c 3 cp /home/user/largefile /Backups</code>
-  * [[https://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-yet-failed-disk-in-a-linux-raid5-array#using-mdadm-3.3|How to safely replace a not-yet-failed disk in a Linux RAID5 array?]]
-  * [[https://delightlylinux.wordpress.com/2020/12/22/how-to-remove-a-drive-from-a-raid-array/|mdadm: How to Remove a Drive From a RAID Array]]
-  * [[http://en.wikipedia.org/wiki/Mdadm|mdadm]]
-  * [[https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID|Recovering a failed software RAID]]
-  * [[https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Cluster_Logical_Volume_Manager/VG_activate.html|Activating and deactivating volume groups]]
-  * [[https://raid.wiki.kernel.org/index.php/RAID_Recovery|RAID Recovery]]
-  * [[http://www.cyberciti.biz/faq/linux-creating-software-raid-one-arrays/|Linux Create Software RAID 1 (Mirror) Array]]
-  * [[http://www.cyberciti.biz/faq/linux-server-rename-an-mdadm-raid-array/|Linux Rename a RAID Array From md0 to md2]]
-  * [[https://ubuntuforums.org/archive/index.php/t-1947275.html|Mdadm missing superblock]]
-  * [[http://askubuntu.com/questions/63980/how-do-i-rename-an-mdadm-raid-array|How do I rename an mdadm raid array?]]
-  * [[http://askubuntu.com/questions/57908/how-can-i-quickly-copy-a-gpt-partition-scheme-from-one-hard-drive-to-another|How can I quickly copy a GPT partition scheme from one hard drive to another?]]
-  * [[https://www.dlford.io/linux-mdraid-disk-replacement-procedure/|7 Steps to Safely Replace a Drive in a Linux MD RAID Array]]
-  * [[https://rabexc.org/posts/mdadm-replace|Replacing and resizing a linux software raid]]
-  * [[https://rabexc.org/posts/resizing-filesystem|Resizing an encrypted filesystem with LVM on Linux]]
-  * [[https://unix.stackexchange.com/questions/320957/extend-a-luks-encrypted-partition-to-fill-disk|Extend a LUKS encrypted partition to fill disk]]
-  * [[https://superuser.com/questions/625722/reducing-linux-software-raid-rebuild-speed#comments-625724|Reducing Linux Software RAID Rebuild Speed?]]
-==== Check wether all volumes get mounted during system boot ====
-  * to check wether root and swap are mounted, enter:<code>mount
-free -m -t</code>
-  * to check mismatching uuid's, enter:<code>blkid
-ls -la /dev/disk/by-uuid
-cat /etc/fstab</code>
-  * to fix, replace the uuid's found in ''/etc/fstab'' with the ones found in ''/dev/disk''. Make sure you copy the correct uuid (md0, md1) to the respective entry in fstab.<code>vim /etc/fstab</code>
-===== Resync =====
-Most Debian and Debian-derived distributions create a cron job which issues an array check at 0106 hours each first Sunday of the month in /etc/cron.d/mdadm. This task appears as resync in /proc/mdstat and syslog. So if you suddenly see RAID-resyncing for no apparent reason, this might be a place to take a look.
-Normally the kernel will throttle the resync activity (c.f. [[https://en.wikipedia.org/wiki/Nice_%28Unix%29|nice]]) to avoid impacting the raid device performance.
-However, it is a good idea to manage the resync parameters to get optimal performance.
-==== Raid 1, 5, 6 ====
-=== Rebuild speed ===
-  * Get current system values:<code>sudo sysctl dev.raid.speed_limit_min
-sudo sysctl dev.raid.speed_limit_max</code>
-  * Default system values on Debian 10:<code>dev.raid.speed_limit_min = 1000
-dev.raid.speed_limit_max = 200,000</code>
-  * Reduce max limit to make server more responsive during resync (2021-12-05):<code>sudo sysctl -w dev.raid.speed_limit_min=10,000
-sudo sysctl -w dev.raid.speed_limit_max=100,000</code>
-=== read-ahead ===
-  * Get current read-ahead (in 512-byte sectors) per Raid device (default value is 512 on Debian 10):<code>blockdev --getra /dev/mdX</code>
-  * Set to 32 MB:<code>blockdev --setra 65536 /dev/mdX</code>
-  * Set to 65536 on a server with 32GB memory, 32768 on a server with 8GB memory (2021-12-05)
-=== Disable NCQ ===
-  * Get NCQ depth on each physical Drive in Raid (default value is 31):<code>cat /sys/block/sdX/device/queue_depth</code>
-  * Disable NCQ:<code>echo 1 > /sys/block/sdX/device/queue_depth</code>
-==== Raid 5, 6 only ====
-=== stripe_cache_size ===
-It records the size (in pages per device) of the stripe cache which is used for synchronising all write operations to the array and all read operations if the array is degraded. The default is 256 which equals to 3MB memory consumption.  Valid values are 17 to 32768. Make sure your system has enough memory available: memory_consumed = system_page_size * nr_disks * stripe_cache_size.
-  * Find system page size, on Debian 10 this is 4096:<code>getconf PAGESIZE</code>
-  * Set to 384MB memory consumption on a 3 disk:<code>sudo echo 32768> /sys/block/md0/md/stripe_cache_size</code>
-  * Set to 32768 on a server with 32 GB memory, set to 16384 on a server with 8 GB memory (2021-12-05)
-===== Prepare RAID with single disk =====
-==== Prepare new disk ====
-  - If the new disk contains partitions
-    - Stop any Raid partitions with<code>mdadm --stop /dev/md1
-mdadm --remove /dev/md1</code>
-    - Remove the superblocks<code>mdadm --zero-superblock /dev/sdX1</code>
-    - Remove existing partitions with ''fdisk /dev/sdX''
-  - Create a new partition utilizing the full disk space. When asked, remove the existing signature. Change partition type to ''Linux RAID''<code>sudo fdisk /dev/sdX
-Command (m for help): d
-Command (m for help): n
-Command (m for help): t,29</code>
-  - Create the RAID<code>mdadm --create /dev/mdX --level=raid1 --raid-devices=2 /dev/sdX1 missing</code>
-  - Check the RAID was created<code>cat /proc/mdstat
-ls /dev/md*</code>
-  - Add a second disk<code>mdadm --manage /dev/mdX --add /dev/sdX1</code>
 ==== Links ====
-  * [[https://unix.stackexchange.com/questions/63928/can-i-create-a-software-raid-1-with-one-device|Can I create a software RAID 1 with one device]]
+  * [[https://www.maketecheasier.com/manage-priority-io-processes-linux/|How to Manage the Priority of I/O Processes in Linux]]
-  * [[https://wiki.archlinux.org/title/Convert_a_single_drive_system_to_RAID|Convert a single drive system to RAID]]
+  * [[http://manpages.ubuntu.com/manpages/bionic/man1/ionice.1.html|ionice]]
-  * [[https://bobcares.com/blog/removal-of-mdadm-raid-devices/|Removal of mdadm RAID Devices – How to do it quickly?]]
+  * [[https://bugs.gentoo.org/show_bug.cgi?id=287840|Make samba run ioniced]]
-  * [[https://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array|Replacing A Failed Hard Drive In A Software RAID1 Array]]
+  * [[https://serverfault.com/questions/55560/nice-rsync-on-remote-machine|Nice rsync on remote machine]]
-===== Move RAID to a new machine =====
+  * [[https://www.ittsystems.com/network-throughput/|What is Network Throughput and How to Measure & Monitor it!]]
-  - Scan for the old raid disks<code>sudo mdadm --assemble --scan</code>
+  * [[https://software.es.net/iperf/invoking.html|Invoking iperf3]]
-  - Mount the raid manually to confirm<code>blkid
-sudo mount /dev/md0 /mnt</code>
-  - Append info to mdadm.conf<code>mdadm --detail --scan >> /etc/mdadm/mdadm.conf</code>
-  - Update initramfs<code>update-initramfs -u</code>
-==== Troubleshooting ====
-  * Make sure the output of "mdadm --detail --scan" matches your /etc/mdadm/mdadm.conf
-  * Examine /etc/fstab
-==== Links ====
-  * [[https://serverfault.com/questions/32709/how-do-i-move-a-linux-software-raid-to-a-new-machine|How do I move a Linux software RAID to a new machine?]]
-  * [[https://unix.stackexchange.com/questions/23879/using-mdadm-examine-to-write-mdadm-conf/52935#52935|Using mdadm --examine to write mdadm.conf]]
-  * [[https://askubuntu.com/questions/729370/can-i-transfer-my-mdadm-software-raid-to-a-new-system-in-case-of-hardware-failur|Can I transfer my mdadm Software raid to a new system in case of hardware failure?]]
-===== Increase drive capacity in a RAID -> LVM -> CRYPT setup =====
-==== Replace drives in RAID ====
-  - Follow [[#Replace_a_failing_or_failed_disk|Replace a failing or failed disk]] at the top of this page
-  - Resize the array to the maximum supported by the underlying partitions<code>mdadm --grow /dev/md5 -z max</code>
-  - Follow the progress with<code>watch -d cat /proc/mdstat</code>
-==== Increase LVM ====
-  - Check size of physical volume with<code>pvdisplay</code>
-  - Increase physical volume to utilize all available space<code>pvresize /dev/mdX</code>
-  - Increase logical volume to utilize all available space<code>lvextend -l +100%FREE /dev/<volume_group>/<logical_volume></code>
-==== Increase LUKS ====
-  - Inform LUKS to utilize all available space, you need the backup key to do this<code>cryptsetup resize /dev/mapper/<volume_group>/<logical_volume>_crypt</code>

Differences

Know How