Creating an Amazon Centos HVM AMI with Resizable LVM Root

With the introduction of t2 instance types on aws I needed a base AMI which was of a HVM type but used LVM for its partitions. I wanted the ability to modify the partiton sizes both on the fly by added a new ebs volume and when launching images without the hassle of repartitioning ebs volumes. The following is a step by step guide to creating your AMI. I assume you have a good understanding of using AWS.

Launch any existing CentOS 6 AMI. I run updates to make sure all packages version match not sure this is actually required.

Create a 1G EBS and a 9G EBS drive and attach these to your new Image Builder instance. The 1G will contain the boot partition and swap space. The 9G will contain the lvm volumes.

These should appear as

• /dev/sdf 1G
• /dev/sdg 9G

Partion /dev/sdf into a 256Mb Boot and the rest as Swap.

fdisk -l /dev/sdf should return

   Device Boot      Start         End      Blocks   Id  System
/dev/sdf1               1          34      273073+  83  Linux
/dev/sdf2              35         130      771120   82  Linux swap / Solaris 

Format the partitions on /dev/sdf

mkfs.ext2 /dev/sdf1
mkswap -L swapfs /dev/sdf2
e2label /dev/sdf1 bootfs

Create the Volume Group and Logical Volumes. I use a modified CIS partitioning scheme removing the /home as the systems doesn't have users.

vgcreate vgEBS /dev/sdg
lvcreate -l 1279 -n lvRoot /dev/vgEBS
lvcreate -L 2G -n lvVar /dev/vgEBS
lvcreate -L 1G -n lvLog /dev/vgEBS
lvcreate -L 512 -n lvAudit /dev/vgEBS
lvcreate -L 512 -n lvTmp /dev/vgEBS

Running lvs should return

LV      VG    Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
lvAudit vgEBS -wi-ao---- 512.00m                                         
lvLog   vgEBS -wi-ao----   1.00g                                         
lvRoot  vgEBS -wi-ao----   5.00g                                         
lvTmp   vgEBS -wi-ao---- 512.00m                                         
lvVar   vgEBS -wi-ao----   2.00g                                             

Format the lvms

mkfs.ext4 /dev/vgEBS/lvRoot 
mkfs.ext4 /dev/vgEBS/lvVar
mkfs.ext4 /dev/vgEBS/lvLog
mkfs.ext4 /dev/vgEBS/lvAudit
mkfs.ext4 /dev/vgEBS/lvTmp

Modify the ext4 disks to remove reserved space, Setting the root to use 1% reserved space.

tune2fs -m 0 /dev/vgEBS/lvVar 
tune2fs -m 0 /dev/vgEBS/lvLog 
tune2fs -m 0 /dev/vgEBS/lvAudit
tune2fs -m 0 /dev/vgEBS/lvTmp
tune2fs -m 1 /dev/vgEBS/lvRoot

Mount the folders and make sub folders

mount /dev/vgEBS/lvRoot /mnt/
mkdir -p /mnt/proc /mnt/etc /mnt/dev /mnt/var /mnt/sys /mnt/tmp /mnt/boot
mount /dev/sdf1 /mnt/boot
mount /dev/vgEBS/lvVar /mnt/var
mount /dev/vgEBS/lvTmp /mnt/tmp
mkdir -p /mnt/var/log /mnt/var/cache /mnt/var/lock/rpm
mount /dev/vgEBS/lvLog /mnt/var/log/
mkdir -p /mnt/var/log/audit
mount /dev/vgEBS/lvAudit /mnt/var/log/audit/

Create the fstab

cat <<EOT >/mnt/etc/fstab
/dev/vgEBS/lvRoot  /                   ext4            defaults            1 1
LABEL=bootfs       /boot               ext3            defaults            1 2
/dev/vgEBS/lvTmp   /tmp                ext4            nodev,nosuid        0 0
/dev/vgEBS/lvVar   /var                ext4            defaults            0 0
/dev/vgEBS/lvLog   /var/log            ext4            defaults            0 0
/dev/vgEBS/lvAudit /var/log/audit      ext4            defaults            0 0
LABEL=swapfs       swap                swap            defaults            0 0
/tmp               /var/tmp            none            bind                0 0
tmpfs              /dev/shm            tmpfs           noexec,nodev,nosuid 0 0
devpts             /dev/pts            devpts          gid=5,mode=620      0 0
sysfs              /sys                sysfs           defaults            0 0
proc               /proc               proc            defaults            0 0
EOT

 Install the Base system into the /mnt folder

yum --releasever=6 --installroot=/mnt/ -y groupinstall Base 

Configure the network settings

cat <<EOT >/mnt/etc/sysconfig/network
NETWORKING=yes
HOSTNAME=localhost.localdomain
NETWORKING_IPV6=no
IPV6INIT=no
EOT
cat <<EOT >/mnt/etc/sysconfig/network-scripts/ifcfg-eth0 
DEVICE="eth0"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="dhcp"
EOT

Turn off ipv6 the correct way

echo "sysctl.conf:net.ipv6.conf.all.disable_ipv6 = 1" >> /mnt/etc/sysctl.conf

Setup Grub Configuration file. Get the version of the kernel installed into your chroot environment.

CHROOT_KERNEL=`yum --installroot=/mnt/ install kernel | awk '/kernel/ {print $2}' | cut -c 8-`

 Setup the grub conf file

cat <<EOT > /mnt/boot/grub/grub.conf
default=0
timeout=0
hiddenmenu

title CentOS ($CHROOT_KERNEL)
    root (hd0,0)
    kernel /vmlinuz-$CHROOT_KERNEL ro root=/dev/vgEBS/lvRoot rd_LVM_LV=vgEBS/lvRoot rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us console=ttyS0
    initrd /initramfs-$CHROOT_KERNEL.img
EOT

ln -s /boot/grub/grub.conf /mnt/boot/grub/menu.lst

Setup the chroot environment.

MAKEDEV -d /mnt/dev -x console
MAKEDEV -d /mnt/dev -x null
MAKEDEV -d /mnt/dev -x zero
 
mount -o bind /dev /mnt/dev
mount -o bind /dev/pts /mnt/dev/pts
mount -o bind /dev/shm /mnt/dev/shm
mount -o bind /proc /mnt/proc
mount -o bind /sys /mnt/sys

cp /etc/resolv.conf /mnt/etc/resolv.conf

Enter the chrooted environment

chroot /mnt

Remove extra things we don't need

yum remove cpuspeed abrt* at hal* iptables-ipv6 irqbalance kexec-tools psacct quota sendmail smartmontools rng-tools mdadm

Add things in we will need.

yum install openssh-server yum-plugin-fastestmirror e2fsprogs dhclient yum-presto vi audit postfix cronie  

Setup a ec2-user and set a secure password

adduser ec2-user
passwd ec2-user

Setup aws scripts to import ssh keys (Use this as a base for other on boot tasks)

cat <<EOT > /usr/local/bin/awsUpdateKeys
#!/bin/bash

if [ ! -d /home/ec2-user/.ssh ]; then
  mkdir -m 0700 -p /home/ec2-user/.ssh
  restorecon /home/ec2-user/.ssh
fi

# Get the home/ec2-user ssh key setup
ReTry=0
Status=1
while [ \$Status -gt 0 ] && [ \$ReTry -lt 5 ]; do
    if (( \$ReTry > 0 ))
    then
        sleep 2
    fi 
    curl -f http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key > /home/ec2-user/.ssh/authorized_keys 2> /dev/null
    Status=\$?
    ReTry=\$[Retry+1]
done

chown -R ec2-user:ec2-user /home/ec2-user/.ssh/
chmod 600 /home/ec2-user/.ssh/authorized_keys && restorecon /home/ec2-user/.ssh/authorized_keys

if (( \$ReTry >= 5 ))
then
    exit 1
fi
EOT
chmod +x /usr/local/bin/awsUpdateKeys

cat <<EOT > /etc/init.d/aws
#!/bin/bash
#
# chkconfig: 2345 11 01
# description: AWS Configuration on Boot

source /etc/rc.d/init.d/functions
case "\$1" in
    start)
        echo -n "Update SSH Keys: "
        /usr/local/bin/awsUpdateKeys && echo_success || echo_failure
        echo    

        # Add your own initialisation scripts here.

        RETVAL=0
    ;;
    stop)
        RETVAL=0
        ;;
    status)
        RETVAL=0
        ;;
    *)
        echo $"Usage: $0 {start|stop|status}"
        RETVAL=2
esac
exit $RETVAL
EOT
chmod +x /etc/init.d/aws

chkconfig --add aws

Install Grub

grub-install /dev/sdf
Probing devices to guess BIOS drives. This may take a long time.
Unknown partition table signature
/dev/xvdf does not have any corresponding BIOS drive.

Don't worry about the output this has populated the /boot/grub folder as required for us. Now manually install grub.

grub
 grub> device (hd0) /dev/sdf
 grub> root (hd0,0)
 grub> setup (hd0)

The setup set should output the following

setup (hd0)
 Checking if "/boot/grub/stage1" exists... no
 Checking if "/grub/stage1" exists... yes
 Checking if "/grub/stage2" exists... yes
 Checking if "/grub/e2fs_stage1_5" exists... yes
 Running "embed /grub/e2fs_stage1_5 (hd0)"...  27 sectors are embedded.
succeeded
 Running "install /grub/stage1 (hd0) (hd0)1+27 p (hd0,0)/grub/stage2 /grub/grub.conf"... succeeded

Add the ec2-user into the sudoers files.

echo "%ec2-user     ALL=(ALL)       NOPASSWD: ALL" >> /etc/sudoers  

Set the system to relabel the whole file-system on boot for selinux (Seriously, stop disabling SELinux)

touch /.autorelabel

Now on the AWS management console take a snapshot from both the 1G and 9G drives. Create a HVM image from these snapshot using the 1G as your root drive. And the 9G as the second drive. This process won't work with a para-virtual style image.

You can now launch the AMI as an instance and set the second drive to be as big as you like resizing the partitions using standard lvm tools.