Sunday, 18 November 2012

Windows 8 and importing VirtualBox VM's into Hyper-V

Upgraded to Windows 8 this weekend and it's definitely nice and painless. Other than the new start menu and the loss of Aero it's really not much different to Windows 7 externally, and after the upgrade all my apps continued to work like before as if nothing had changed.

The only real issue I had initially was with VirtualBox. The version I was running won't run on Windows 8 (the latest might do), but that wasn't an issue as I was already planning to switch to using Hyper-V instead. Importing my existing .vhd's into Hyper-V should have been straight forward, but I kept getting errors similar to this :

New Virtual Machine Wizard
The Virtual machine management service encountered an error while configuring the hard disk on virtual machine <Guest name>.

Failed to add device 'Synthetic Disk Drive'

'<Guest name>' failed to added device 'Synthetic Disk Drive'. (Virtual machine ID xxx)
The Virtual Machines configuration xxx at '<Hyper-V Path>' is no longer accessible: The requested operation cannot be performed on a file with a user-mapped section open (0x800704C8)

when I tried to create a new VM using the existing vhd as the disk. The issue turned out to be AVG and its resident shield scanning the files as mentioned here, once I'd excluded all those folders I was good to go and things started working.

Wednesday, 31 October 2012

Changing the cPanel "Configure Email Client" server addresses in WHM 11.34

One of the great things about cPanel is how easy it makes it for non-technical people to use, for instance within Email Accounts, clicking More and then Configure Email Client provides you with links to automatically configure several email clients, or the details required to do it manually. The data is automatically generated to show the relevant domain's POP3/IMAP/SMTP server details (specific to each domain), but what if you prefer to have a single set of service addresses, how do you update this page to show your preferred details.

In the previous version it was quite simple, you simply edited the clientconfinclude.html that's found within /usr/local/cpanel/base/frontend/x3/mail/ and replaced the dynamic code with your static text.

With WHM 11.34 that's changed, and the .html file no longer controls any of the data, instead it calls the template file clientconfinclude.tmpl which then generates all of the content. Simply doing the switch like before isn't as simple any more, especially if you're not much of a coder, but it's still doable with some minor modifications. Unfortunately the code seems to use multiple methods to call the server addresses, so you have to do different mods for different sections of code.

First all we have the SSL hosts, which is easily the simplest part. Line 11 to line 24 deals with getting those details, with the next section storing them for use later. Without deleting anything simply insert these three new lines immediately before line 26 that begins "SET ssl_hosts = {" :

SET imap_ssl_host = '';
SET pop3_ssl_host = '';
SET smtp_ssl_host = '';

That will then force all SSL references to use your information. Now look for the line :

SET non_ssl_mail_server = webmail_settings.domain;

Copy it, and edit the entries so you have :

SET non_ssl_mail_server = '';
SET non_ssl_mail_server_smtp = '';

In the manual settings section, if you'd prefer the default POP3/IMAP to be IMAP then change pop3 to imap above. Now look for the line :

[% locale.maketext("Outgoing Server:") %]

at around line 685. On the next line, change "non_ssl_mail_server" to be "non_ssl_mail_server_smtp" (so it matches the name used in the last step).

If you save this file and reload the page you should now find most of the links / text now points to your preferred addresses. Unfortunately it's not perfect, there are a few entries that display incorrectly, and I haven't yet worked out where they get their information or how to change it, but hopefully it will be a bit of help for someone.

Monday, 20 August 2012

Windows 8 - First thoughts and future telephone support issues

Now that Windows 8 has been released I've been having a play with it this weekend. In terms of usability I'm not entirely sure how I feel about it currently. On the one hand I'm sure in time I could come to get used to the new way of doing things, and the workarounds to enable the old ways, but on the other I can only imagine the pain and misery to come from trying to do telephone support for it.

The key problem I see is with the schizophrenic nature of Windows 8. Having two completely different UIs within a single OS will at best make things interesting, at worst confuse the hell out of people.

It's not helped by a lack of distinction between the two in regards to names. For instance, now that it's no longer officially called Metro, we have the Desktop and the Modern UI, but the latter isn't a name it's a description! The Desktop UI doesn't exactly look dated either, in fact I'd say it was positively modern. What will we call it in several years time when the "Modern UI" is anything but? It's not helped by many sources also referring to it as the Windows 8 UI, therefore throwing three different names into the mix.

Then there's Internet Explorer, with two essentially different applications using the same name. They function differently, have different options and ways of doing things, yet how do you easily differentiate them verbally? I've seen enough users struggle to understand what IE is at the best of times, but at least then you can tell them to look for the big blue "e". Now you also have to check which UI they're using, otherwise your instructions will be meaningless.

Internet Explorer also handles downloads differently, which could be fun for anyone using a remote support application (eg, getting the client to go to a URL, enter some details, and run a program to give you remote access to their machine). Since IE 9 the download dialog window has been tied to the window that initiated it. If the download was initiated via a popup (eg it provides the download link to click), and that popup closed once the download started, the download dialog would disappear once completed. Fortunately you could bring it back by pressing CTRL-J, and then choose to save or run the download. This still works in the desktop IE 10, but doesn't in Metro IE 10. Instead you get a dialog appear for 2 or 3 seconds giving you the run / save / cancel options, and if you don't respond in time it vanishes. Thus far I've been unable to find a way to retrieve it other than running the download again.

All that said, I have found a few handy features.

Windows + D - Show Desktop - Nothing new here, but importantly it will get you out of Metro to the normal desktop UI.

Windows + C - Charms menu - This is the Metro menu you get by hovering to the right hand corners to access things like settings, shutdown/restart etc.

Windows + Z - Tabs / Address menu in Metro IE and All apps from Metro start page - Again, saves messing around with tricky mouse positioning.

Finally, while there's no way to disable Metro on startup, there is a work around to make getting to the desktop easier. On the metro start screen, drag the Desktop tile up to the top left hand corner (eg top row, first column). Now after typing your password and pressing Enter to login you can immediately go to the desktop by pressing Enter a second time (since the top left tile is automatically selected).

Sunday, 19 August 2012

The joys of technet

Thanks to the wonders of a Technet subscription and a new computer I've been having a play with VirtualBox at home. While certainly more basic than VMWare or Hyper-V it does the job, I think it's easier to use than VMWare and of course Hyper-V won't run on Windows 7.

I'm definitely very tempted by Windows 8 since that includes Hyper-V Client, though due to the train reck that is the Metro interface I may well stick with this if it continues to do what I need, especially if the reviews of the latest release preview turn out to be accurate.

Perhaps it's sad, but I'm actually looking forward to being able to play with various things at home. All those things that have caught my interest, that I fancy looking at, but which have no justification to spend time looking at while at work.

Sunday, 12 August 2012

When Linux load averages lie

To follow on from my last post, having data showing the load on your server throughout the day is great, but what do you do with it and how do you interpret the information?

Looking through the data I could see clear short periods where the load average would sky rocket, into double figures and into the 20's. From every article and post I'd seen it said that if the load divided by the number of cores/threads was more than one you have a problem. No ifs, no buts, you have a problem. I was seeing 20+ on a dual cpu server... oh dear!

The problem was I couldn't see an obvious cause. CPU usage wasn't high, the idle time was good, memory was fine with the swap file not being used, the disk queue wasn't long, and the process list didn't show anything to indicate an issue.

Fortunately I came across a few fantastic explanations of Load Averages that explained where I (and it seems many others) had been going wrong. I've linked to them all below, and I recommend reading them for more info, but the upshot is that it's not as cut and dry as people make out.

To quote from Jon Emmons blog, the load average "is the average sum of the number of processes waiting in the run-queue plus the number currently executing over 1, 5, and 15 minute time periods.".

The load average is far more complex than many people make out, and while it can be a good initial indicator of a problem it must be examined in conjunction with other factors. It doesn't allow for the fact that a process could be waiting for not just the CPU, but also disk or network IO, and doesn't allow for the priority of the running/waiting processes in the queue.

If you have a long running low priority process running for instance, that will always make way for more urgent time critical requests. In the mean time that process will sit in the queue, and will cause the load average to increase. Add some more of these low priority processes, for instance a backup job, and the load average will increase, which indicates a problem. Higher priority processes like email, websites etc will be handled immediately however, causing no delay for users, and as such the reported high load isn't really an issue.

So the key is that it's fine to track the load average to indicate a possible problem, but don't rely on it for proof that you have one. Always remember to check the other figures provided to see IF you have a problem, not necessarily WHAT the problem is.

Tuesday, 7 August 2012

Tracking Linux server load and processes over time

Tracking down the cause for high load on a server can be a challenge, especially when issues are happening when you can't keep an eye on it, but it's a challenge I've been facing recently. While there were no issues with the server's response, I discovered at certain times (often in the evenings) the load average would become high enough to cause Exim to briefly pause processing the mail queue.

After hunting round for a easy way to track the server's state I came across a simple script from Craig Edmonds that did the job. It very simply generates an email containing a variety of status information, including most importantly the process list data from Top, and sends that to you in an email. By scheduling the script to run every minute you get a snapshot of the servers state at regular intervals. Since the subject line includes the load average you can easily look through the messages, spot those times with a high load, and see what the server is doing.

In my case I adjusted the script slightly to include $todaydate in the subject line, since the above issue with Exim meant I couldn't always rely on the message being received in the correct order.

There was one problem I found with this solution. The script runs a single iteration of Top and inserts the output of that into the email, however as you can see from the Man page for Top :

       The  top command calculates Cpu(s) by looking at the change in CPU time
       values between samples. When you first run it, it has no previous  sam-
       ple  to  compare  to, so these initial values are the percentages since
       boot. It means you need at least two loops or you have to  ignore  sum-
       mary output from the first loop.  This is problem for example for batch
       mode. There is a possible workaround if you define the CPULOOP=1  envi-
       ronment variable. The top command will be run one extra hidden loop for
       CPU data before standard output.

each email I received had identical CPU data. While I could tell the server's load was high, I couldn't see what the state of the processor was at that time. I didn't fancy messing around with environment variables, so instead opted for a solution found here, and adjusted the line calling Top as follows :

    $process_list = shell_exec('top -b -n2 | awk "/^top/{i++}i==2"');

So I found a simple and easy way to track what's happening on the server, though of course with an email a minute it's not something I'll be running long term.

I wish I could say that was the end of it, but unfortunately this turned out to be the beginning of my struggle and confusion, brought on in no small part to the number of confused explanations of Load Average operations, but I'll discuss that in my next post.

References :

Tuesday, 17 April 2012

Move all SQL databases from one server to another

When it comes to moving databases there a number of different methods that you can use. The best method will depend on the situation, but most comments/guides/blogs online concentrate on moving individual databases, either by importing/exporting/copying the data between servers using SSMS, backing up and restoring the databases, or detaching and attaching them.

The method described here is a way to move all of the MS SQL databases from one server to another. The basic idea is to stop all of the MS SQL Server services on both servers, copy all of the data files including the system database files (eg the .mdf, and .ldf files) to the same location(s) on the new server, and then start SQL Server on the new server.

There are a few key points to note about this method when considering if it is suitable to your situation :

  • The new and old servers must have the same version and build of SQL Server installed.
  • The file paths for your data files must be the same on both servers.
  • It doesn't matter if the two servers have different names.
  • The new server must have enough storage space to hold the data files in their current location.
  • Depending on the amount of data this method can involve a long window of downtime while the data copies over the network

Before stopping SQL Server and starting the move it's worth considering any housekeeping tasks that might need doing. A couple of things I've done before starting a move :

  • Remove any old / unwanted databases. This process moves all of them, so if they're no longer required why waste time copying a database you don't need.
  • Consider shrinking database files. Keeping your transaction log files large can avoid situations where the file has to grow, and shrinking isn't generally recommended, however, copying gigs of empty space over a network will needlessly increase the time this process takes. Therefore it may be worth shrinking the file initially, and then manually increasing it again once it's running on the new server.

The steps to the move are :

  • Install SQL Server on the new server, ensuring it's the same version and build as the old server, and with the same options installed. Ensure that your system database location is configured to be the same as what you have on the old server.
  • Stop all of the SQL Server services on both servers.
  • Copy all of the database mdf, ldf and ndf files from the old server to the new server, including all of the system database files. Each file must be copied to the same location on the new server as it was on the old server, otherwise SQL won't be able to find them when when it starts.
  • Copy any other related files that will be required, eg if any jobs are configured to access or use files on the server then copy them as well.
  • Start the SQL Server services on the new server.
  • Open SSMS and confirm that all of the databases are being shown and are accessible.
  • Either update your application(s) or site(s) to use the new server, or change the server name and network details to match that of the old server so they can connect with the existing details.
  • Test and confirm that everything is working correctly.


I've successfully used this method on SQL Server 2008 R2, however it should also work on all other versions and in fact other people have confirmed it to work with SQL 7 and 2000.

The installations were fairly straight forward setups, with no replication, clustering, Analysis or Reporting services configured, so while I doubt they would stop this from working I can't confirm that for certain.

If you're unsure of the exact setup of your existing installation, check the "Server Server Installation Center" on the old server, go to Tools, and click on "Installed SQL Server features discovery report". This will generate a report showing all of the components installed and what build that are currently.