Wednesday 6 August 2014

Creating an Amazon Centos HVM AMI with Resizable LVM Root

With the introduction of t2 instance types on aws I needed a base AMI which was of a HVM type but used LVM for its partitions. I wanted the ability to modify the partiton sizes both on the fly by added a new ebs volume and when launching images without the hassle of repartitioning ebs volumes. The following is a step by step guide to creating your AMI. I assume you have a good understanding of using AWS.

Launch any existing CentOS 6 AMI. I run updates to make sure all packages version match not sure this is actually required.

Create a 1G EBS and a 9G EBS drive and attach these to your new Image Builder instance. The 1G will contain the boot partition and swap space. The 9G will contain the lvm volumes.

These should appear as
  • /dev/sdf 1G
  • /dev/sdg 9G
Partion /dev/sdf into a 256Mb Boot and the rest as Swap.

fdisk -l /dev/sdf should return
   Device Boot      Start         End      Blocks   Id  System
/dev/sdf1               1          34      273073+  83  Linux
/dev/sdf2              35         130      771120   82  Linux swap / Solaris 
Format the partitions on /dev/sdf
mkfs.ext2 /dev/sdf1
mkswap -L swapfs /dev/sdf2
e2label /dev/sdf1 bootfs
Create the Volume Group and Logical Volumes. I use a modified CIS partitioning scheme removing the /home as the systems doesn't have users.
vgcreate vgEBS /dev/sdg
lvcreate -l 1279 -n lvRoot /dev/vgEBS
lvcreate -L 2G -n lvVar /dev/vgEBS
lvcreate -L 1G -n lvLog /dev/vgEBS
lvcreate -L 512 -n lvAudit /dev/vgEBS
lvcreate -L 512 -n lvTmp /dev/vgEBS
Running lvs should return
LV      VG    Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
lvAudit vgEBS -wi-ao---- 512.00m                                         
lvLog   vgEBS -wi-ao----   1.00g                                         
lvRoot  vgEBS -wi-ao----   5.00g                                         
lvTmp   vgEBS -wi-ao---- 512.00m                                         
lvVar   vgEBS -wi-ao----   2.00g                                             
Format the lvms
mkfs.ext4 /dev/vgEBS/lvRoot 
mkfs.ext4 /dev/vgEBS/lvVar
mkfs.ext4 /dev/vgEBS/lvLog
mkfs.ext4 /dev/vgEBS/lvAudit
mkfs.ext4 /dev/vgEBS/lvTmp
Modify the ext4 disks to remove reserved space, Setting the root to use 1% reserved space.
tune2fs -m 0 /dev/vgEBS/lvVar 
tune2fs -m 0 /dev/vgEBS/lvLog 
tune2fs -m 0 /dev/vgEBS/lvAudit
tune2fs -m 0 /dev/vgEBS/lvTmp
tune2fs -m 1 /dev/vgEBS/lvRoot
Mount the folders and make sub folders
mount /dev/vgEBS/lvRoot /mnt/
mkdir -p /mnt/proc /mnt/etc /mnt/dev /mnt/var /mnt/sys /mnt/tmp /mnt/boot
mount /dev/sdf1 /mnt/boot
mount /dev/vgEBS/lvVar /mnt/var
mount /dev/vgEBS/lvTmp /mnt/tmp
mkdir -p /mnt/var/log /mnt/var/cache /mnt/var/lock/rpm
mount /dev/vgEBS/lvLog /mnt/var/log/
mkdir -p /mnt/var/log/audit
mount /dev/vgEBS/lvAudit /mnt/var/log/audit/
Create the fstab
cat <<EOT >/mnt/etc/fstab
/dev/vgEBS/lvRoot  /                   ext4            defaults            1 1
LABEL=bootfs       /boot               ext3            defaults            1 2
/dev/vgEBS/lvTmp   /tmp                ext4            nodev,nosuid        0 0
/dev/vgEBS/lvVar   /var                ext4            defaults            0 0
/dev/vgEBS/lvLog   /var/log            ext4            defaults            0 0
/dev/vgEBS/lvAudit /var/log/audit      ext4            defaults            0 0
LABEL=swapfs       swap                swap            defaults            0 0
/tmp               /var/tmp            none            bind                0 0
tmpfs              /dev/shm            tmpfs           noexec,nodev,nosuid 0 0
devpts             /dev/pts            devpts          gid=5,mode=620      0 0
sysfs              /sys                sysfs           defaults            0 0
proc               /proc               proc            defaults            0 0
EOT
 Install the Base system into the /mnt folder
yum --releasever=6 --installroot=/mnt/ -y groupinstall Base 
Configure the network settings
cat <<EOT >/mnt/etc/sysconfig/network
NETWORKING=yes
HOSTNAME=localhost.localdomain
NETWORKING_IPV6=no
IPV6INIT=no
EOT
cat <<EOT >/mnt/etc/sysconfig/network-scripts/ifcfg-eth0 
DEVICE="eth0"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="dhcp"
EOT
Turn off ipv6 the correct way
echo "sysctl.conf:net.ipv6.conf.all.disable_ipv6 = 1" >> /mnt/etc/sysctl.conf
Setup Grub Configuration file. Get the version of the kernel installed into your chroot environment.
CHROOT_KERNEL=`yum --installroot=/mnt/ install kernel | awk '/kernel/ {print $2}' | cut -c 8-`
 Setup the grub conf file
cat <<EOT > /mnt/boot/grub/grub.conf
default=0
timeout=0
hiddenmenu

title CentOS ($CHROOT_KERNEL)
    root (hd0,0)
    kernel /vmlinuz-$CHROOT_KERNEL ro root=/dev/vgEBS/lvRoot rd_LVM_LV=vgEBS/lvRoot rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us console=ttyS0
    initrd /initramfs-$CHROOT_KERNEL.img
EOT
ln -s /boot/grub/grub.conf /mnt/boot/grub/menu.lst
Setup the chroot environment.
MAKEDEV -d /mnt/dev -x console
MAKEDEV -d /mnt/dev -x null
MAKEDEV -d /mnt/dev -x zero
 
mount -o bind /dev /mnt/dev
mount -o bind /dev/pts /mnt/dev/pts
mount -o bind /dev/shm /mnt/dev/shm
mount -o bind /proc /mnt/proc
mount -o bind /sys /mnt/sys

cp /etc/resolv.conf /mnt/etc/resolv.conf
Enter the chrooted environment
chroot /mnt
Remove extra things we don't need
yum remove cpuspeed abrt* at hal* iptables-ipv6 irqbalance kexec-tools psacct quota sendmail smartmontools rng-tools mdadm
Add things in we will need.
yum install openssh-server yum-plugin-fastestmirror e2fsprogs dhclient yum-presto vi audit postfix cronie  
Setup a ec2-user and set a secure password
adduser ec2-user
passwd ec2-user
Setup aws scripts to import ssh keys (Use this as a base for other on boot tasks)
cat <<EOT > /usr/local/bin/awsUpdateKeys
#!/bin/bash

if [ ! -d /home/ec2-user/.ssh ]; then
  mkdir -m 0700 -p /home/ec2-user/.ssh
  restorecon /home/ec2-user/.ssh
fi

# Get the home/ec2-user ssh key setup
ReTry=0
Status=1
while [ \$Status -gt 0 ] && [ \$ReTry -lt 5 ]; do
    if (( \$ReTry > 0 ))
    then
        sleep 2
    fi 
    curl -f http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key > /home/ec2-user/.ssh/authorized_keys 2> /dev/null
    Status=\$?
    ReTry=\$[Retry+1]
done

chown -R ec2-user:ec2-user /home/ec2-user/.ssh/
chmod 600 /home/ec2-user/.ssh/authorized_keys && restorecon /home/ec2-user/.ssh/authorized_keys

if (( \$ReTry >= 5 ))
then
    exit 1
fi
EOT
chmod +x /usr/local/bin/awsUpdateKeys
cat <<EOT > /etc/init.d/aws
#!/bin/bash
#
# chkconfig: 2345 11 01
# description: AWS Configuration on Boot

source /etc/rc.d/init.d/functions
case "\$1" in
    start)
        echo -n "Update SSH Keys: "
        /usr/local/bin/awsUpdateKeys && echo_success || echo_failure
        echo    

        # Add your own initialisation scripts here.

        RETVAL=0
    ;;
    stop)
        RETVAL=0
        ;;
    status)
        RETVAL=0
        ;;
    *)
        echo $"Usage: $0 {start|stop|status}"
        RETVAL=2
esac
exit $RETVAL
EOT
chmod +x /etc/init.d/aws

chkconfig --add aws
Install Grub
grub-install /dev/sdf
Probing devices to guess BIOS drives. This may take a long time.
Unknown partition table signature
/dev/xvdf does not have any corresponding BIOS drive.
Don't worry about the output this has populated the /boot/grub folder as required for us. Now manually install grub.
grub
 grub> device (hd0) /dev/sdf
 grub> root (hd0,0)
 grub> setup (hd0)
The setup set should output the following
setup (hd0)
 Checking if "/boot/grub/stage1" exists... no
 Checking if "/grub/stage1" exists... yes
 Checking if "/grub/stage2" exists... yes
 Checking if "/grub/e2fs_stage1_5" exists... yes
 Running "embed /grub/e2fs_stage1_5 (hd0)"...  27 sectors are embedded.
succeeded
 Running "install /grub/stage1 (hd0) (hd0)1+27 p (hd0,0)/grub/stage2 /grub/grub.conf"... succeeded
Add the ec2-user into the sudoers files.
echo "%ec2-user     ALL=(ALL)       NOPASSWD: ALL" >> /etc/sudoers  
Set the system to relabel the whole file-system on boot for selinux (Seriously, stop disabling SELinux)
touch /.autorelabel
Now on the AWS management console take a snapshot from both the 1G and 9G drives. Create a HVM image from these snapshot using the 1G as your root drive. And the 9G as the second drive. This process won't work with a para-virtual style image.

You can now launch the AMI as an instance and set the second drive to be as big as you like resizing the partitions using standard lvm tools.