The "StableHost" blog

We’ve worked with a certain provider many times throughout the last few years; they’ve hosted external services for us that we’ve used internally that needed to be outside our network. This post isn’t to slam this provider, in fact we won’t be naming the provider, however the lesson we learned that day was critical to us and we’re making sure our customers never experience the same lesson as we did.

It’s 1AM, we started to get paged regarding a certain internal service that was down, this particular service was hosted on a VPS at another provider’s network. We opened up a ticket with them as many of our own customers would do and the first response we received was “It’s not down, it’s just slow”. While that answer may be perfectly acceptable in certain situations, when we can’t ping or ssh into the server at all, it doesn’t matter if it’s “just slow”, it’s down for us and that’s all that matters. We replied back letting them know not only was it “not slow, but completely down” we didn’t hear back for quite some time.

We jumped on their live chat a few hours later and asked what’s going on, their response was ‘we’re working on it, no eta’. Few hours later, we replied to our ticket again asking for an update, no response.

12 hours later, our server is accessible again.

To this date, we have not yet received a response from them as to what happened, no apology, nothing.

So, what did we learn from this?

• Downtime will happen regardless! It doesn’t matter if you’re Joe’s Small Hosting Company or Google. You will have outages big and small, computers have a limited time span before they finally fail on you.

• Communication during an outage is #1. We’ve learned customers are a lot more understanding knowing what’s being done to bring the server back up, even if they don’t exactly understand the “tech lingo”. There should be status updates via twitter and facebook at least hourly and constant updates to support tickets. No communication during an outage is the worst thing you can do, however it’s also the easiest thing to do as you’re busy fixing the server, the last thing you want to do is update people on the status. We believe, in our situation, the entire server had to be restored and in that case 10+ hours is perfectly acceptable as there’s a lot of work to be done, however without any updates from the company you automatically assume the worst.

• Follow up with customers after the fact, be truthful and learn from your mistakes. If your customers have been down for the past few hours, they deserve some kind of notification letting them know what happened, why it happened and what you’re going to do to prevent it from happening again.

You can run the perfect company in terms of pricing, support, have a great looking website and so on, however if you forget the simple things such as communication when it matters most, you’re not going to succeed in this business.

Several years ago, prior to working here, I was a manager of a technical support organization. Every few months, I’d have an employee come to me and let me know they were leaving the company because they were unhappy with a certain aspect of their job. I always asked them why they didn’t let me know prior and always felt like a failure because as a manager it’s your job to realize when people are unhappy and make them happy again. I could never get a honest answer from them, I just assume employees feel it’s easier and less risky to find another job then to to be honest with their boss.

Similar situations has haunted me here, however it’s not with employees but customers this time. Very few customers will write in and tell us they are unhappy with us but instead change web hosts and we find out months down the road when they write a thread on a web hosting forum letting people know why they don’t recommend us.

As we run a shared platform, we have to use technology and standards that will fit ‘most’ customers, now this doesn’t mean every customer is going to have a positive experience because we know that won’t happen. However, in most cases, we can always make the customer happy by moving them to another server, adjusting a server configuration or upgrading them to another plan/service. I can count on my right hand how many times we’ve exhausted all resources for a single client and end up letting them know they are better off with another service because we simply can’t help them. That feeling really sucks and we’re lucky we don’t have to do it very often.

I encourage everyone if there’s a problem in life, not just web hosting, be honest with the other person/company and in most cases you will see it’s a lot easier to work out the issue then to move on. It’s our job to make sure you’re happy and if we never hear from you, we will always expect there’s nothing we could do to make your experience a better one.

We recently sent out an invitation to rate us on our performance to a few thousand people — we didn’t send it out to new customers because we wanted people to have a chance to experience us a bit more before reviewing.  We only expected 20 or so people to respond because let’s be honest, surveys are a pain.  We all get them constantly and they just annoy us — so seeing a few hundred people answered ours was amazing!

Normally survey results are kept fairly confidential — however, we were stunned by the results so we wanted to share with everyone.  Usually you won’t see companies release survey results because they are always negative, companies send out surveys to figure out “What’s wrong” and “Why sales are horrible”.  We don’t feel that way at all, however customer needs are always changing and we want to make sure we’re supporting our customers as much as possible…… and now to the results:

First off, we’re extremely happy to see the majority of people voted us “Excellent” in all categories — that is outstanding and we couldn’t be more happy.

As we expected, speed was the lowest rating (insert sad face here).  As we pointed out a few months ago, shared web hosting isn’t for every website out there.  If it’s a personal website where you store pictures and use for email, you won’t have speed issues, if it’s a small business website that you run, shouldn’t be a problem either.  Slow speeds come into play when you’re running multiple sites with database-driven applications (Huge blogs, forums, chat servers, etc).

In most cases, when you notice speed issues, it’s usually not due to the server being overloaded (even though that has happened due to various reasons in the past, it’s unlikely), it’s usually due to you running out of your own CPU/memory restrictions we place on each account.  We do this, not to make you mad, however to protect the server and protect the other hundreds of clients on our servers.  However, as we allow for each website to use “up to 10%” (not to be confused with use 10% constantly), a few bad apples in the hundreds of accounts can really slow things down, all it takes is 6 or 7 accounts and the server is running on 30% CPU idle for the rest of the hundreds of accounts, not really ideal.  We do our best to make sure our server’s are always monitored and they page us as soon as they notice CPU issues so if it comes to that point, we fix it before it becomes a major issue.  We recently released VPS plans (both budget — for those who are experienced system administrators who can need a VPS for a low-resource project, and enterprise VPS’s, which are for high end websites (high end blogs, forums, etc) who don’t want to upgrade to a dedicated server but want the benefit and speed of one.

However, our servers are getting a little bit of upgrade in the next few months (starting with the first one tomorrow, actually).  Each of our US servers has 12G of ram, we are upgrading all of our Phoenix-based servers to 24G of ram, doubling the amount of things we can keep in cache.  This will help increase speeds greatly across the board.  Some of our EU servers already have 24gigs, we’ll be rolling it out to all of them after we finish with our Phoenix-based servers.

The next survey question was regarding communication and how often we communicated with you.  Sometimes I feel like we communicate too much with you so I’m glad we asked this question — the results were not expected.   We’ll continue to send updates as much as needed — hopefully in the future, customers can check what kind of communication they want from us (outages, maintenance’s, features, etc) and so they only receive certain emails from us if they feel we’re sending too many.

The next question related to how we notify customers, we’re glad phone wasn’t a favorite.   It would take a long time to call thousands of people anytime we made a change — and we’re nerds, we hate talking on the phone!

The question related to referrals was a huge sign to us we’re doing the right thing — in most cases, you don’t refer other people to services which you aren’t happy with, therefore we’re very happy to see the results of that poll

As far as ‘What could we do better?’ this was a comment box with lots of comments — most of them say ‘Nothing, you’re doing a great job’ which is awesome!    However, I wanted to publicly address on the blog:

Concerns about our terms of service – If you are a newer customer of ours, you might have no idea what we are talking about here, but maybe 8 months ago, we completely re-did our terms with our lawyer and he wrote up about 25 pages of “lawyer-speak”.     We thought it was a good idea due to recent events at the time, however we received a lot of negative feedback about it.    We decided to change our terms a few months back to make it much easier to read — therefore, for those who complained about our terms, we invite you to check out our new terms and let us know any feedback you have!   It’s best to contact us directly regarding terms questions/comments.

Concerns we are getting too big and service will decline – Every big company has this issue, either they grow too fast and can’t support things or they start going cheap on things to make more profitable.     We’re doing our best NOT to become those companies, we’re not a fan of these “mega web hosting companies” that have millions of customers and 1000′s of employees because to them, a customer is nothing.    We’re growing at a steady pace that gives us the ability to not have hundreds of employees and we still try to treat each customer like they are a part of our family.      I hope we never lose that feeling as we continue to grow.     If it wasn’t for the customers, we wouldn’t be here — as people grow big, they forget that very thing.

Coupons for existing customers - The majority of our coupons on for new accounts only — this is pretty standard across most web hosting companies.   However, after reading some of these comments, we’re going to change some things up and start offering limited-time coupons to existing customers to encourage them to bring their additional accounts over to us.  When this happens, we’ll make sure to announce it!

That’s about it — we greatly thank each and every one of you for taking your time and answering our survey.

Recently, our phone company grasshopper.com had a massive outage that lasted over 24 hours. The original outage was due to a raid failure which is bound to happen sooner or later. Hard drives are always the part that fails the most — however, that’s why you make backups, right?

Grasshopper.com decided to fail over to their secondary/failover site while they tried to fix the raid, however their secondary site didn’t work as expected. I’ve personally seen this time and time again — companies put thousands (sometimes millions) into a secondary / redundant setup and in the prime moment where you need it the most, it doesn’t work for you. It’s easy to forget about your secondary site — usually you never log into it, once it’s setup and working — people just let it run itself. However, is that the right thing to do? Of course not.

So how do we prepare for failures? There’s three major failures we’re concerned about, we’ll outline them below:

Data loss – As I mentioned earlier, hard drives are not reliable — even at times, raids aren’t reliable, however we do our best to make sure customer data is safe. We run raid on all of our servers except 3 — those servers are very old and will be getting replaced shortly. By running raid, it gives us the safety net knowing if one drive fails, the server is still going to stay up and continue processing. So, what happens if we lose two drives at once? In the unlikely situation that we do lose multiple drives at once, we have local backups on the server. We do local backups several times a week syncing customer websites to a secondary drive on the server, not related to the raid. Therefore, if the raid dies, we still have local copies of the data. So, what happens if both the raid dies and local backup drive dies? That’s when we pack up and go home for the day. We wish , however we have yet another method to make sure your data is safe — we sync our local backup drives to a secondary backup server weekly as well. We hope you understand now how serious we take keeping customer data safe. We have developed custom, in-house scripts to monitor that data and make sure it’s always in sync with each server. Lately, we always encourage our customers to take backups of their own data as well. You can download a full backup of your site by logging into cPanel -> Backup -> Generate Full Backup.

System Failure – Although hard drive failures are most common, an entire system failure can happen as well. Yes — we have solutions for these types of failures as well. In Europe, where we lease dedicated servers, the datacenter will replace the faulty parts for us and we’ll be back up and running in no time. In Phoenix, where we buy all of our own hardware, it’s a bit different. We have spare servers with the exact same specs sitting in our cabinets waiting in case of a failure. We buy premium Dell equipment which is brand new, therefore to-date, we’ve never had a system failure, however does that mean we shouldn’t spend the money to have extra machines sitting there doing nothing? No! If a system does fail, we can swap the drives and the customer will be back up in no time — and we can spend the rest of the day troubleshooting with Dell on what went wrong and fix the bad hardware without having the customer suffer.

Networking Failure – The last part failure we are concerned about is a networking failure. While we don’t run our own network in Europe and therefore don’t need to worry about it there — we do in Phoenix. In Phoenix, our entire network was designed by us and all equipment is owned by us. As we prefer to have complete control over our network, it also is prone to failures. To prevent a networking failure, we have redundant switches and routers sitting in our cabinets, similar to our server setup. In the event one dies, we will move everything to a spare and they will be back up. As far as providers, we currently have 3 network providers and have configured our network that if one has an outage, it will automatically switch to the other one within seconds. We do, unlike our phone company, test our failover and make sure it does work as designed.

We hope you have a better understanding of how we stand by our name, Stable Host. It’s not cheap by any means to have redundant everything, but it’s worth it in an outage. Every company will have outages from time to time — it’s a matter of who’s going to be prepared for it or not.