Adding and mounting a new disk in Linux to expand storage and house the contents of /home

So you want to add another disk to your Linux install. There could be several reasons for this, for instance running out of space on your existing setup, or wanting to split your system across multiple drives.

In my case I'd setup a new cPanel instance and wanted more capacity for the content in /home using a second disk. As such this post also explains how to move the existing /home content onto the newly added disk, such that everything continues to work like it did before.

I'll start by assuming you've added the new drive to your server, whether physically or as a new virtual disk, in your chosen hyper-visor.

Check that the system has detected the new disk by running :

    ls /dev/sd*

You'll see a list of all the physical disks detected, and the partitions on them. The first drive is /dev/sda, and if that has two partitions on it currently you'll see /dev/sda1 and /dev/sda2 listed. This increments for each drive, so a newly added second drive will be /dev/sdb.

You can check and ensure /dev/sdb has nothing on it already by running :

    fdisk -l

which will display the disk and partition details, and indicate that /dev/sdb doesn't contain a valid partition table. So lets get a partition added. Run :

    fdisk /dev/sdb

then at the prompt press n for New. You'll be asked what type of partition you want, so press p for primary. Now you'll be asked to select how much of the disk to use, these default to the first and last cylinders, so if you're using the entire disk you can simply press return on both and use the defaults. Finally at the last prompt press w to write the table to disk and exit. Note, if you quit without pressing w then nothing will be changed.

Now if you check in /dev you'll see a newly added /dev/sdb1 listed, but before we can use it we need to format it. To format it as ext3 run :

    /sbin/mkfs.ext3 -L /home /dev/sdb1

which will format the disk. The "-L /home" part of the command specifies the volume label, so if you're not using this disk for your /home folder then you might want to adjust it accordingly.

Now we want to move the existing contents of /home onto the newly formatted disk, but obviously new and old can't be /home at the same time so...

    mkdir /mnt/home
    mount /dev/sdb1 /mnt/home

to create a temporary mount point for the new drive and then mount it. Now copy the data with :

    cp -Ra /home/* /mnt/home

and then check in /mnt/home that the data is all there. Now we're finished with the temporary mount point so we can remove it with :

    umount /mnt/home
    rmdir /mnt/home

Before we can mount the new disk to its proper home we need to get rid of the existing /home folder, so let's rename it rather than delete it :

    mv /home /home-backup

which also means that if things go wrong you can switch back to the original data. Then, because we still need a folder to mount the new drive to :

    mkdir /home

To permanently mount the new disk we need to edit the fstab file, otherwise if we simply mount it then it won't reload next time the server is booted. Edit /etc/fstab and then add the following line :

    /dev/sdb1    /home    ext3    defaults    0 2

This then mounts the drive partition /dev/sdb1 to the folder /home using the ext3 file system and using the default options. The 0 relates to the archiving frequency (generally set to 0) and 2 controls the order that fsck checks for errors on the drive at boot time. You'd set that to 1 for the boot device (so not relevant here), or 2 for a subsequent drive. Many people blogging about this set it to 0 which disables checking, which I wouldn't recommend for anything other than a drive you were using short term.

Once that's saved we just need to test that it's working. You could reboot, but personally I prefer to run :

    mount -a

which will execute all of the entries listed in fstab (but do nothing where something's already mounted). If your new entry is OK then it should mount the new disk and make the contents accessible via /home, or if there's a problem then it will indicate where the issue is so you can edit /etc/fstab again to correct the issue.

So now you have a newly mounted drive, containing the same folder structure as before, but with the /home content stored on a separate drive. If you run :

    df

you'll see the newly added mount point and drive, and the disk usage / availability.

The above steps were tested on a CentOS 6.4 Hyper-V virtual machine, but should obviously work on most Linux installs.

Changing the cPanel "Configure Email Client" server addresses in WHM 11.34

One of the great things about cPanel is how easy it makes it for non-technical people to use, for instance within Email Accounts, clicking More and then Configure Email Client provides you with links to automatically configure several email clients, or the details required to do it manually. The data is automatically generated to show the relevant domain's POP3/IMAP/SMTP server details (specific to each domain), but what if you prefer to have a single set of service addresses, how do you update this page to show your preferred details.

In the previous version it was quite simple, you simply edited the clientconfinclude.html that's found within /usr/local/cpanel/base/frontend/x3/mail/ and replaced the dynamic code with your static text.

With WHM 11.34 that's changed, and the .html file no longer controls any of the data, instead it calls the template file clientconfinclude.tmpl which then generates all of the content. Simply doing the switch like before isn't as simple any more, especially if you're not much of a coder, but it's still doable with some minor modifications. Unfortunately the code seems to use multiple methods to call the server addresses, so you have to do different mods for different sections of code.

First all we have the SSL hosts, which is easily the simplest part. Line 11 to line 24 deals with getting those details, with the next section storing them for use later. Without deleting anything simply insert these three new lines immediately before line 26 that begins "SET ssl_hosts = {" :

SET imap_ssl_host = 'imap.mydomain.com';

SET pop3_ssl_host = 'pop3.mydomain.com';
SET smtp_ssl_host = 'smtp.mydomain.com';

That will then force all SSL references to use your information. Now look for the line :

SET non_ssl_mail_server = webmail_settings.domain;

Copy it, and edit the entries so you have :

SET non_ssl_mail_server = 'pop3.mydomain.com';

SET non_ssl_mail_server_smtp = 'smtp.mydomain.com';

In the manual settings section, if you'd prefer the default POP3/IMAP to be IMAP then change pop3 to imap above. Now look for the line :

[% locale.maketext("Outgoing Server:") %]

at around line 685. On the next line, change "non_ssl_mail_server" to be "non_ssl_mail_server_smtp" (so it matches the name used in the last step).

If you save this file and reload the page you should now find most of the links / text now points to your preferred addresses. Unfortunately it's not perfect, there are a few entries that display incorrectly, and I haven't yet worked out where they get their information or how to change it, but hopefully it will be a bit of help for someone.

When Linux load averages lie

To follow on from my last post, having data showing the load on your server throughout the day is great, but what do you do with it and how do you interpret the information?

Looking through the data I could see clear short periods where the load average would sky rocket, into double figures and into the 20's. From every article and post I'd seen it said that if the load divided by the number of cores/threads was more than one you have a problem. No ifs, no buts, you have a problem. I was seeing 20+ on a dual cpu server... oh dear!

The problem was I couldn't see an obvious cause. CPU usage wasn't high, the idle time was good, memory was fine with the swap file not being used, the disk queue wasn't long, and the process list didn't show anything to indicate an issue.

Fortunately I came across a few fantastic explanations of Load Averages that explained where I (and it seems many others) had been going wrong. I've linked to them all below, and I recommend reading them for more info, but the upshot is that it's not as cut and dry as people make out.

To quote from Jon Emmons blog, the load average "is the average sum of the number of processes waiting in the run-queue plus the number currently executing over 1, 5, and 15 minute time periods.".

The load average is far more complex than many people make out, and while it can be a good initial indicator of a problem it must be examined in conjunction with other factors. It doesn't allow for the fact that a process could be waiting for not just the CPU, but also disk or network IO, and doesn't allow for the priority of the running/waiting processes in the queue.

If you have a long running low priority process running for instance, that will always make way for more urgent time critical requests. In the mean time that process will sit in the queue, and will cause the load average to increase. Add some more of these low priority processes, for instance a backup job, and the load average will increase, which indicates a problem. Higher priority processes like email, websites etc will be handled immediately however, causing no delay for users, and as such the reported high load isn't really an issue.

So the key is that it's fine to track the load average to indicate a possible problem, but don't rely on it for proof that you have one. Always remember to check the other figures provided to see IF you have a problem, not necessarily WHAT the problem is.

http://www.lifeaftercoffee.com/2006/03/13/unix-load-averages-explained/
http://blog.mellowhost.com/confusing-server-load-average-explained.html
http://www.teamquest.com/pdfs/whitepaper/ldavg1.pdf
http://www.teamquest.com/pdfs/whitepaper/ldavg2.pdf

Tracking Linux server load and processes over time

Tracking down the cause for high load on a server can be a challenge, especially when issues are happening when you can't keep an eye on it, but it's a challenge I've been facing recently. While there were no issues with the server's response, I discovered at certain times (often in the evenings) the load average would become high enough to cause Exim to briefly pause processing the mail queue.

After hunting round for a easy way to track the server's state I came across a simple script from Craig Edmonds that did the job. It very simply generates an email containing a variety of status information, including most importantly the process list data from Top, and sends that to you in an email. By scheduling the script to run every minute you get a snapshot of the servers state at regular intervals. Since the subject line includes the load average you can easily look through the messages, spot those times with a high load, and see what the server is doing.

In my case I adjusted the script slightly to include $todaydate in the subject line, since the above issue with Exim meant I couldn't always rely on the message being received in the correct order.

There was one problem I found with this solution. The script runs a single iteration of Top and inserts the output of that into the email, however as you can see from the Man page for Top :

       The  top command calculates Cpu(s) by looking at the change in CPU time
       values between samples. When you first run it, it has no previous  sam-
       ple  to  compare  to, so these initial values are the percentages since
       boot. It means you need at least two loops or you have to  ignore  sum-
       mary output from the first loop.  This is problem for example for batch
       mode. There is a possible workaround if you define the CPULOOP=1  envi-
       ronment variable. The top command will be run one extra hidden loop for
       CPU data before standard output.

each email I received had identical CPU data. While I could tell the server's load was high, I couldn't see what the state of the processor was at that time. I didn't fancy messing around with environment variables, so instead opted for a solution found here, and adjusted the line calling Top as follows :

    $process_list = shell_exec('top -b -n2 | awk "/^top/{i++}i==2"');

So I found a simple and easy way to track what's happening on the server, though of course with an email a minute it's not something I'll be running long term.

I wish I could say that was the end of it, but unfortunately this turned out to be the beginning of my struggle and confusion, brought on in no small part to the number of confused explanations of Load Average operations, but I'll discuss that in my next post.

References :

http://www.craig-edmonds.com/abuse-monitoring-script-for-cpanel/

http://www.unix.com/gentoo/77494-top-batch-mode-cpu-info-wrong.html