Archive for June, 2009

As many of you know we have been dealing with an issue for the last couple of weeks where one of our servers (cpanel1) has been crashing. We have been investigating the issue and have not come up with any definitive issues but today I cam across a graph that may have some clews to the issue. The attached graph shows the connection times from our monitoring software to cpanel1 for the last month. As you can see on 6/16/09 the connection times increased considerably.

The one thing that corresponds to that date is a memory upgrade that was performed on 6/16/09 on cPanel1. We just pulled out the memory suspecting that it may be bad memory. We will continue to monitor the server for the next few days to see if this was the root cause of the issues.

This message is an outage report for server cpanel1.alurium.com.

The server went offline at 8:31PM PST on June 17, 2009 and came back online at 9:52PM PST on June 17, 2009.

Downtime: 1h 21min

This outage effected all accounts provisioned on cpanel1. If your account is provisioned on a different server your were not effected by this outage.

Cause: We are still investigating the root cause of the outage. The underlying cluster was operational, all host members of the cluster were operational, but the virtual machine (cpanel1) on the host member (AL03) was stuck in an unstable state. The virtual server could not be stopped, rebooted, or suspended. The guest OS was unresponsive and all attempts to gracefully shutdown the system complained of a running task. Eventually we had to force the host member of the cluster to reboot. We are investigating the root cause of this issue.

Well a new update was just released for Railo and we wanted to upgrade right away. There was a lot of buzz on the various lists of how long it took people to apply the update using the built in features in the Server Administrator, so we decided to time it.

It took only 39.5 seconds to apply the update. That is just incredible. I’ve been using ColdFusion since version 1.5, in fact back then we called it Cold Fusion, and have used various CFML engines during the past decade. But this built it upgrade process is the smoothest I have experience from any vendor.