Tuesday Update
November 13, 2007
At this point, all email and web functions should be back to normal. If you’re having trouble switching your settings on Outlook/Outlook Express then please call 877-442-1975 and we’ll work with you to get setup.
If you’re switching to GMail.com, I’ll have a cheat sheet published shortly outlining the process as well as a way to import your OLD webmail into Gmail.com. If you need help with this please hold off calling until I have the cheat sheet ready for you to reference.
And the question you’re asking is…
November 12, 2007
What happened? What caused this massive problem in the first place?
Now that the dust is finally settling, I’ll post some links here so you can read first-hand what happened.
Here’s a good article about it in the Boston Globe with one “correction” from me. I was not a reseller. I paid Navisite to manage and protect MY servers. They were a vendor of my business. Here’s one from Computer World that’s a little geeky.
Bottom Line: Navisite’s CEO is Arthur Becker, husband of Vera Wang. This is a huge, high-profile, publicly traded International Corporation, with huge liability at this point. I will be joining any viable class-action lawsuit that comes my way.
Changing Mail Settings
November 9, 2007
You will need to make some simple changes in your mail program (Outlook/Outlook Express/Thunderbird) to be able to get your mail.
- If you have been using an IP address such as 65.109.77.xx then you MUST change it ASAP - that is the OLD server and no mail is going there anymore. You may have some old mail there that you might want to download for reference, but no NEW mail is going to show up - ever.
- If you changed it earlier this week to the 66.225.221.45 address, then you need to change it to 216.246.45.203
Yes, you need to change it. Yes, you really do.
Right now: Change your INCOMING MAILSERVER to 216.246.45.203
Thank You. That wasn’t so hard, was it?
If you use Webmail then I STRONGLY encourage you to switch to gmail - see my post below. It is the only program I have ever endorsed since AOL, Hotmail, Yahoo, etc. are not designed for business - but gmail is. You will absolutely not have to change your email address with us. They integrate seamlessly. You will still send mail, and receive mail with your domain name using gmail.
Chelle
Instead of Webmail, Use Gmail
November 8, 2007
There’s a lot of reasons why. I have a whole article on it, in fact - which is in a web-box around here somewhere, lost in the move, so to speak. I’ll dig it out and mail it in a little while.
(Edit: Here’s the link: http://mentalgaragesale.com/2006/12/review-using-gmail-for-travel-agencies.html)
In the meantime:
Thursday Morning Update
November 8, 2007
Well, after a very long night, here’s the progress:
- All current e-mail should be coming in fine now - there should be no lost or bouncing e-mails at this point. If you have reports from your clients of not being able to get through to you let me know ASAP.
- All the websites are displaying, but could still throw some errors when clicking on “hardcoded” links or when an image on the site points to a full path (ie, gemtravelsites.com/images/aaa.jpg) because they are pointing to a numerical address once you hit the site. If your graphics or links are dead they will be restored when we perform our audit this weekend.
Since our backup server was designed to be a short-term solution and not long term, I have purchased a newer, faster server with our backup company and we will be transitioning over this weekend. Don’t Worry! (*says the sleep deprived geeky-girl) The only change you will need to make is to log into your control panel a little differently on Monday. All mail functions will remain the same from this point on.
Now for some good news: When this is all over, I’ll have some “goodies” to thank everyone for their patience through this mess. A couple of new templates that you can switch to for free (waiving the $250 template change fee!) to give your site a makeover, some newsletter templates you can use to send a Holiday card to your clients, and a copy of my new e-marketing book from travelwebmarketing.com.
I’m hoping to be able to have a couple of conference calls tomorrow to help with e-mail challenges. I’ll be on the line to answer questions about setups, migrations to web-based mail and anything else that’s still troubling. Look for details in the morning.
Finally, in my zeal to setup everyone else’s boxes, I forgot my damn self. So, if you sent a mail to chelle@crossoverconsulting.com or chelle@gemtravelsites.com this week, it might not have made it. Re-send it to chelleyarbrough@gmail.com
We’re almost through. A big thank you to those that have offered wine (and stronger), chocolate, hugs and Rogaine (in case I was pulling my hair out) - I’m pleased to say my hair is in tact, I have some awesome coffee the sun is shining through my office window, and I’ve got this playlist going.
Random Notes
November 7, 2007
First, the migration of website data is going very smoothly. I would expect to have all sites restored tonight or tomorrow morning.
Second, just because the kids hide the Halloween candy in a pillow case in a pile of clothes in their room doesn’t mean I can’t find it while they’re at school. After four days without leaving my office (really) the chocolate just called to me. And I answered. And there will be hell to pay at 4pm.
Squashin’ Bugs.
November 7, 2007
OK, I just had to take a personal moment and fix the date on this template. It was displaying the date in European format 06.11.2007 instead of 11.06.07 like we do in America.
But I digress.
After the conference call with Alabanza/Navisite @ 6pm yesterday, I was promised a call-back, which came last night at 11pm. I worked with a very good tech (mark) to restore the old server to the point that I could get the files I needed to restore all the websites. I started that process last night and I’m about 10% migrated over. I anticipate it will take the better part of today and tonight to have the sites back up, but there will be some things that won’t be “pretty” yet. Since the sites are powered by databases they have to be moved manually and all the sites need to be reconnected to the database. This means you might get errors on the sites, even though they are “up” and responding.
Which brings me to the title of this post. We will be restoring all the sites to have files available to view first, and then I will go back through and squash the bugs and clean up the sites. You don’t need to email me with errors until after I have posted that I’m done, and then I’ll turn it over to you for an audit. Then you can e-mail me with any issues, and I’ll squish those bugs, too.
On a personal note, I want to thank everyone that sent a note of encouragement this week. I actually cried when I got this one:
Chelle,
I am sympathetic to your plight. I cannot imagine how stressed you must be after all you have been through. Believe me, I appreciate your control of the situation. I once heard someone say that the true measure of a person is to see how they react under real stress. You have passed the test!!!
I am pleased to say that your excellent explanation and instructions have enabled us to be up and running as if this never happened. Yes, we missed some e-mails but the world did not end. We will manage the situation.
Thank you for your professionalism in this trying time. I am now TOTALLY convinced that we should stay with you.
If you were here, I would give you a well-deserved hug.
—————————————————–
WOW. I have awesome clients. Thank you all for your patience and loyalty. I promise to do a better job of earning it in non-crisis times.
Please continue to let me know what you need. I’ll do my best to get everything fixed ASAP.
Chelle
Signs of Life Everywhere
November 6, 2007
I just finished speaking with the VP of Customer Service and Hosting Management on a conference call after being locked out of the two previous calls. I am able to “see” the old server but can’t retrieve any data yet, but I was able to get to the right person to agree to a call back tonight.
100% of e-mail functions are back up. At this time I have no e-mails that haven’t been replied to. If you need something e-mail me chelleyarbrough@gmail.com
My hope is to have the data retrieved from the old server tonight, and updated to the new server, which would bring us back to 100% back up. I’ll keep you posted.
Chelle
Support Links
November 6, 2007
I’ve posted a number of links for e-mail to the left. They should help you setup accounts on the server, which we mostly did for everyone, but you may need to change a password, or create new ones.
The big one that everyone probably wants is Setting up Mail on the Desktop which shows you how to configure Outlook/Outlook Express.
There are some video tutorials there as well which I hope will help clarify some of the e-mails I’ve been sending if you ask a specific “how-to” question.
Tuesday, November 6, 2007 - 11:15am EDT
November 6, 2007
Good Morning Everyone,
I wish I had better news. I really do. In 8 years of website hosting, this is the worst case scenario realized.
First, here’s what happened.
In 1999, we partnered with Alabanza, Inc. to manage our network of hosting servers. They managed the day-to-day maintenance of the servers, including load-balancing, DDOS attacks and other issues that separate local web hosts from “world-class hosting management”. They were stable - very stable, and fast, with a Network Operations Center in Baltimore, MD that included raised flooring, Raytheon cooling systems, redundant backbones and redundant connectivity. Some of you have hosted with me for 7 years and you’ll agree that you’ve probably never had to report even one minute of downtime. It was that good.
In October, Alabanza was purchased by a company called Navisite. (www.navisite.com) which is a publicly traded company, over 10 years old. Navisite purchased Alabanza and Jupiterhosting.com for more than $15 Million and after I completed my due-diligence, I was satisfied that it was a good thing for my business and customers. They offered a broader base of technology than Alabanza and their business philosophy was to purchase existing companies and merge them, rather than grow by smaller account sales.
Their plan was to move our server virtually to a Blade server at their Andover, Mass location over the weekend with an estimated down time of 20 minutes. Obviously things went horribly wrong. To date, they have missed 19 separate deadlines for restoring service. I will document the failure later, but be assured that this is not a failure that I take lightly. There are more than 250,000 websites worldwide down at the moment and many people losing not only business, but their very livelihoods.
So, as of this writing, we’ve moved our entire hosting infrastructure to our backup server off the Alabanza/Navisite network. The last e-mail to me from them this morning was:
“Alabanza Clients - 11/06/2007 8 a.m. EST
As we were bringing more and more hosts on last night, we were creating a significant number of ARP requests on the network. The number of ARP requests is primarily due to the fact that we were still resolving name servers issues in part of the environment. The number of ARP requests ultimately overloaded the network which is why you were seeing instability.
In the process of troubleshooting the over utilization issue with Cisco, a routing error was made which has caused most of the environment to be inaccessible. We are working to reverse this change out with Cisco and will then work both the name server and ARP’ng issue.
Sincerely:
Mark Clayman
Senior Vice President of Hosting Services”
ARP means Address Resolution Protocol - a basic website traffic function that any college graduate these days should be able to anticipate and troubleshoot. What this tells me is that they continue to make critical errors at a critical time, and I am not willing to leave my clients there with no end in sight.
The question you’re asking is what does this mean to you. Right now, every client has e-mail capability. Is is different, but it is there. The switch I threw yesterday to move the sites to the backup server wasn’t thrown lightly, because it can take up to 72 hours for everything to be fully sorted out. Let me answer a few questions.
Q. When will e-mail sent over the weekend arrive?
A. It depends on the SENDER’s mailserver. If their server is set to continue to try, then it will try for as long as its programmed to. So, it could show up today, or tomorrow, or never. It might have already been bounced back.
Q. Why can’t I see my website?
A. We do have backups of all the websites and are in the process of retrieval and restoration. Since e-mail has become such a huge part of business today, it has always been our priority to restore that function first, and concentrate on websites second.
Q. Do I have to use webmail to get mail today?
A. No. You can change your mail settings in Outlook/Outlook Express/Thunderbird or whatever mail program you use to the following:
Manual Settings
Mail Server Username: username+domainname (REPLACE WITH YOUR OWN INFO
Incoming Mail Server: 66.225.221.45
Outgoing Mail Server: 66.225.221.45 (server requires authentication)
Q. Can I administer my own e-mail?
A. Yes! Go to 66.225.221.45/cpanel to login with your MASTER user name and password. I will be forwarding this information again shortly, but it is the SAME one you’ve always had. This will allow you to add/change/delete boxes, passwords, etc. You can also setup a Spam Filtering system. More on that later as I get time to update.
Q. When will this all be resolved.
A. Good Question. We are working independently of Navisite to restore our client service. We have been for more than 40 hours, and we will continue. We appreciate your patience and will keep you updated. That’s the best I can do.
Q. Can someone please call me?
A. No. And that’s not because we don’t want to. It is because we simply can’t. I am fielding about 300 e-mails per hour answering questions, and we’re trying to get tutorials and training in place for everyone for the backup system. We can’t call and do individual training for just your office. Please understand.
Q. How do I get some training?
A. For questions regarding the new control panels, including e-mail, spam, etc. go to http://www.cpanel.net/docs/cpanel/cp11/cPanel-11-User-Manual.htm
For a step-by-step on setting up desktop e-mail, go to:
http://www.cpanel.net/docs/cpanel/cp11/Mail/Configure_Mail_Client.htm
I will be converting www.gemtravelsites.com to a blog shortly to keep more information coming. You’ll be able to post questions there as well, which when answered could help others with their questions. Once we’ve been able to help the majority, we’ll be free to help with specific office issues.
So, that’s the update. Again, I wish it were better news, but chances are many of your colleagues are in the same situation, and I promise you, we are in better shape than most of the other hosts who had no contingency plan. As a travel industry veteran I remember how hard we were hit after 9/11 and have always had a plan in place to insure against permanent website/email failure. Some others are not so lucky. We’ll get through this.
I’ll be in touch shortly,
Chelle