Slicehost

Server adriana is down at the moment, affecting about 16 Slices. If you cannot reach your Slice, this is likely the reason. We’re working hard to get it back up or onto a backup machine. We’ll keep everyone posted.

UPDATE I

Definitely a hardware failure: either CPU or RAM related. We moved all of the drives to another box and everyone is back online. We’ll keep monitoring the situation. Sorry for the outage to those on this server.

UPDATE II

It was a bad RAM module – and our run of poor memory luck continues. Replacement DIMMS on the way and the box will be worked back into production for new customers.

We had more trouble today with a circuit that blew this evening while we were at the NOC installing new hardware and investigating the previous circuit trouble. Customers on 4 servers were restarted and down approximately 5-10 minutes. The web server was also on this circuit and we didn’t restart Apache for a good half hour by mistake, so this outage may have appeared longer than it actually was.

We are terribly sorry for this happening twice in the same week and are working diligently to solve the problem. Thanks for your support. Please contact us with any questions.

Don’t forget, a bit of downtime in a couple hours while we’re rebooting and installing some network equipment. We’ll keep you posted.

UPDATE – completed around 1145 CST. Total downtime under 3 minutes.

6 servers (one being the web server) were rebooted after a rack outlet we plugged into got fried. We quickly moved everyone to another outlet. Total downtime was about 3-5 minutes depending on how quickly we restarted your Slice. Please contact us if you are having any problems and apologies for the downtime.

The ajax SSH console in SliceManager has been acting funky today, so we’ve taken it down to track squash a few bugs. Hopefully back up in the morning – please use SSH to access your Slice in the meantime.

UPDATE: back online at 2340 CST.

Brief network outage scheduled for next Wednesday, 10-11-2006. Less than 5 minutes total, please let us know if you have objections or requests for specific time windows. Post here are in the forum thread.

A server required an emergency reboot just after 12am CST this morning. This was due to a runaway process and an LVM hang that degraded performance. Slices were gracefully shutdown with a downtime of just over 90 seconds. If your Slice was on this machine, you have already been sent notification. Apologies for the trouble and we’ll work to make sure it’s avoided in the future. Please relay any feedback you might have.

We now handle the RDNS zones for our IP blocks. Turn around should be much faster – if you submitted a previous request, it has been completed.

First – an apology to all of our customers. We experienced 30 minutes of downtime today, around 1pm. That’s unacceptable and we’re working hard to make sure it doesn’t happen again.

• 1245 CST: Slicehoster Matt T. pops into the chatroom and asked if he was the only one who couldn’t reach his box. I sarcastically responded yes, about 15 seconds before our phones started beeping and the alarms sounded. Predictably, all heck broke loose.

• 1247 CST: I call the NOC. We recently doubled bandwidth, but were having trouble seeing the increase on our side. A NOC engineer was plugged into our backbone switch to investigate. I take off in a sprint for the NOC (tall programmers are not the most graceful runners), while Jason and others man the phone/chatroom to relay updates.

• 1252 CST: Huffing and puffing, I arrive at the NOC to discover the backbone router is off. Engineer admits he may have bumped it or maybe it failed. Power is cycled. But this doesn’t look good, a bump shouldn’t put a router on the fritz.

• 1257 CST: After a few failed reboot attempts, it becomes apparent something else is very wrong. More investigation via the console points to a memory problem. We reseat all of the memory in the device.

• 1307 CST: More reboot attempts after reinstalling the router lead nowhere. Cutover to emergency backup device which requires upstream changes.

• 1315 CST: Everything is back online.

• The router did indeed have a bad memory chip. This prevents it from booting. A replacement is on the way.
• HSRP, scheduled for later this month, was not yet in place. This falls on us.

Again, we aplogize for the outage, it should not have happened. Something got fried and we didn’t have the proper failovers in place. This will be remedied soon. In the meantime, everyone is back online and there shouldn’t be any noticeable difference. And the network should feel faster ;). Please contact us with any questions and thanks again for your support and words of encouragement in the chatroom.

Random updates for Slicehosters:

• We’re working on taking control of our Reverse DNS. Currently we have to go upstream for requests which is taking a bit longer than we hoped. The process should be fixed this week (at least we hope).

• We’re working on allowing customers to purchse extra IP’s or storage space.

• The forums now support RSS/Atom feeds.

The referrals program is in effect. We decided on using email addresses for the referral code, to be referenced during the signup process. This seemed good because we figured people might not sign-up right away after reading a site or blog post. However, some are now asking for referral codes to pass via URL, so we’ll get started on that too.

Current customers – check the Accounts tab in SliceManager. Have people reference your email address during the sign-up process and after 90days your account will be credited. Enjoy!

Hurray – we have opened the doors! A big thanks to all of our early testers. Please contact us with any questions – we’re here for you. Enjoy.

Thanks to 37signals, the chartroom is backup. In a geeky brush with fame, I received an email from Jason (the owner) and a comment from the man himself – DHH. We bought their book, we use all of their products and are all aboard with working smarter, not harder. However, the fact that both of these guys were working customer support on a beautiful Sunday afternoon and evening speaks volumes about their success. Thanks guys!

It appears that 37signals’ scheduled maintenance on the Campfire chat application has broken a few things. We had to flush our DNS cache this morning and it took a few hours for us to login. But now the public chatroom seems to be unaccessible. The URLs it generates for us link to error pages. Hopefully they’ll get it back up soon and we’ll see you there.