Moving /var/log to Software RAID+LVM on CentOS5 with SELinux

That’s one exciting title. I found myself needing to migrate /var/log to a bigger drive as part of setting up a centralized syslog server. Normally, that’s not so hard but my requirements included:

• Software RAID – mirroring drives for fault tolerance and no need for hardware RAID
• Logical Volume Manager – LVM gives me additional flexibility in growing disk space over time independent of the physical disks in the box
• SELinux – In enforcing mode, SELinux can be a pain in the ass but we made the decision to live with it for it’s extra security and auditing capabilties

Even if you’re not using LVM, this is just as applicable because the real bear here is the SELinux part and some non-intuitiveness around using mdadm.

Prepare the Disk

First let’s get the disk set up and ready to go. In these examples I’m setting up the 3rd and 4th SCSI drives in my system, /dev/sdc and /dev/sdd. The order of operation is: fdisk, software raid, LVM, mkfs, mount, copy files, fix SElinux, edit fstab, reboot.

1. fdisk the drives in question creating the correct partition size and number: `fdisk /dev/sdc` or similar.
• Use partition type “fd” for linux auto-detecting raid
• If you have an odd-sized drive like 18.3gb, don’t use the full size of the drive! In the event you need to replace the drive with another that is smaller (say, 18.1gb), you will be unable to rebuild the array. I usually leave a little space at the end just in case.
2. /dev/mdX probably does not yet exist so you’ll get an error about the device not existing because `mdadm –create` is broken; create it: `mknod /dev/mdX b 9 X` where X is your partition number. If you already have /dev/md0 and /dev/md1 and want /dev/md2: `mknod /dev/md1 b 9 2`
3. Create RAID-1 with `mdadm –create –verbose /dev/md2 –level=1 –raid-devices=2 /dev/sdc1 /dev/sdd1`. Note that you are referencing partitions and not drives (e.g., /dev/sdc1 not /dev/sdc).
4. Verify it worked: `mdadm –detail /dev/md2`
5. Logical volume manager manages logical volumes composed of physical volumes. Create the physical volume first: `pvcreate /dev/md2`
6. We name our volume groups vgX; create a volume group: `vgcreate vg1 /dev/md2`
7. Verify it worked: `vgdisplay vg1`. Find out how many PEs or physical extents (“chunks”) are available.
8. We name our logical volumes lvX; create a logical volume: `lvcreate -l <# of PEs to use> vg1 -n lv0`
   • Consider, again, not using the entire disks. Commands like vgextend and lvextend will let you grow the volume size on the fly but shrinking them is just about impossible. I used about half of the available PEs.
9. Now you have a partition, just like /dev/sda1, called /dev/vg1/lv0. You could create more logical volumes on this same volume group to have /dev/vg1/lv1, /dev/vg1/lv2, etc.
10. Create a file system (like ext3): `mkfs.ext3 /dev/vg1/lv0`
11. Mount the drive `mount /dev/vg1/lv0 /mnt`
12. Check the drive looks right, `df -k` and look at /mnt. Does it have the right size and show up like the others?

Ok, nice work. You’ve got a mounted, fault-tolerant, expandable drive on your system just waiting to hold log files.

Replace /var/log & Fix SELinux

13. I use rsyslog (see here for details. If you have syslog, just replace syslog for rsyslog here. Stop rsyslog: `service rsyslog stop`
14. Copy the existing log files: `cp -a /var/log/* /mnt/`.
15. Restore all contexts: `cd /mnt; restorecon -R *`. This fixes everything but the top level directory.
16. Get existing context `ls -alZ /var` to see what /var/log has (system_u:object_r:var_log_t)
17. Set this context on new drive: `chcon system_u:object_r:var_log_t /mnt/.` and `chmod 755 /mnt/`
18. Unmount new disk: `umount /mnt`
19. Remount in proper place: `mount /dev/vg1/lv0 /var/log`
20. Start rsyslog: `service rsyslog start`
21. Verify things are running properly; are there new entries in /var/log/messages?
22. Edit /etc/fstab and add an entry for the new disk like: `/dev/vg1/lv0 /var/log ext3 defaults 1 3`
23. Reboot and verify it’s all still working by tailing /var/log/messages.

If that final reboot worked, then congrats! You’re finished!

Credits go to wisdom from Greg’s excellent RAID+LVM resource and a post from Thomas about restoring SELinux permissions. I added bits about chcon, mknod and tied it all together to actually work with CentOS 5/RHEL 5. This should apply equally well to Fedora Core 5, 6 and beyond or any Linux using SELinux.