Tuesday, November 6, 2007 - 11:15am EDT
November 6, 2007
Good Morning Everyone,
I wish I had better news. I really do. In 8 years of website hosting, this is the worst case scenario realized.
First, here’s what happened.
In 1999, we partnered with Alabanza, Inc. to manage our network of hosting servers. They managed the day-to-day maintenance of the servers, including load-balancing, DDOS attacks and other issues that separate local web hosts from “world-class hosting management”. They were stable - very stable, and fast, with a Network Operations Center in Baltimore, MD that included raised flooring, Raytheon cooling systems, redundant backbones and redundant connectivity. Some of you have hosted with me for 7 years and you’ll agree that you’ve probably never had to report even one minute of downtime. It was that good.
In October, Alabanza was purchased by a company called Navisite. (www.navisite.com) which is a publicly traded company, over 10 years old. Navisite purchased Alabanza and Jupiterhosting.com for more than $15 Million and after I completed my due-diligence, I was satisfied that it was a good thing for my business and customers. They offered a broader base of technology than Alabanza and their business philosophy was to purchase existing companies and merge them, rather than grow by smaller account sales.
Their plan was to move our server virtually to a Blade server at their Andover, Mass location over the weekend with an estimated down time of 20 minutes. Obviously things went horribly wrong. To date, they have missed 19 separate deadlines for restoring service. I will document the failure later, but be assured that this is not a failure that I take lightly. There are more than 250,000 websites worldwide down at the moment and many people losing not only business, but their very livelihoods.
So, as of this writing, we’ve moved our entire hosting infrastructure to our backup server off the Alabanza/Navisite network. The last e-mail to me from them this morning was:
“Alabanza Clients - 11/06/2007 8 a.m. EST
As we were bringing more and more hosts on last night, we were creating a significant number of ARP requests on the network. The number of ARP requests is primarily due to the fact that we were still resolving name servers issues in part of the environment. The number of ARP requests ultimately overloaded the network which is why you were seeing instability.
In the process of troubleshooting the over utilization issue with Cisco, a routing error was made which has caused most of the environment to be inaccessible. We are working to reverse this change out with Cisco and will then work both the name server and ARP’ng issue.
Sincerely:
Mark Clayman
Senior Vice President of Hosting Services”
ARP means Address Resolution Protocol - a basic website traffic function that any college graduate these days should be able to anticipate and troubleshoot. What this tells me is that they continue to make critical errors at a critical time, and I am not willing to leave my clients there with no end in sight.
The question you’re asking is what does this mean to you. Right now, every client has e-mail capability. Is is different, but it is there. The switch I threw yesterday to move the sites to the backup server wasn’t thrown lightly, because it can take up to 72 hours for everything to be fully sorted out. Let me answer a few questions.
Q. When will e-mail sent over the weekend arrive?
A. It depends on the SENDER’s mailserver. If their server is set to continue to try, then it will try for as long as its programmed to. So, it could show up today, or tomorrow, or never. It might have already been bounced back.
Q. Why can’t I see my website?
A. We do have backups of all the websites and are in the process of retrieval and restoration. Since e-mail has become such a huge part of business today, it has always been our priority to restore that function first, and concentrate on websites second.
Q. Do I have to use webmail to get mail today?
A. No. You can change your mail settings in Outlook/Outlook Express/Thunderbird or whatever mail program you use to the following:
Manual Settings
Mail Server Username: username+domainname (REPLACE WITH YOUR OWN INFO
Incoming Mail Server: 66.225.221.45
Outgoing Mail Server: 66.225.221.45 (server requires authentication)
Q. Can I administer my own e-mail?
A. Yes! Go to 66.225.221.45/cpanel to login with your MASTER user name and password. I will be forwarding this information again shortly, but it is the SAME one you’ve always had. This will allow you to add/change/delete boxes, passwords, etc. You can also setup a Spam Filtering system. More on that later as I get time to update.
Q. When will this all be resolved.
A. Good Question. We are working independently of Navisite to restore our client service. We have been for more than 40 hours, and we will continue. We appreciate your patience and will keep you updated. That’s the best I can do.
Q. Can someone please call me?
A. No. And that’s not because we don’t want to. It is because we simply can’t. I am fielding about 300 e-mails per hour answering questions, and we’re trying to get tutorials and training in place for everyone for the backup system. We can’t call and do individual training for just your office. Please understand.
Q. How do I get some training?
A. For questions regarding the new control panels, including e-mail, spam, etc. go to http://www.cpanel.net/docs/cpanel/cp11/cPanel-11-User-Manual.htm
For a step-by-step on setting up desktop e-mail, go to:
http://www.cpanel.net/docs/cpanel/cp11/Mail/Configure_Mail_Client.htm
I will be converting www.gemtravelsites.com to a blog shortly to keep more information coming. You’ll be able to post questions there as well, which when answered could help others with their questions. Once we’ve been able to help the majority, we’ll be free to help with specific office issues.
So, that’s the update. Again, I wish it were better news, but chances are many of your colleagues are in the same situation, and I promise you, we are in better shape than most of the other hosts who had no contingency plan. As a travel industry veteran I remember how hard we were hit after 9/11 and have always had a plan in place to insure against permanent website/email failure. Some others are not so lucky. We’ll get through this.
I’ll be in touch shortly,
Chelle
Comments
Got something to say?