Slicehost

As I’m sure most people noticed, we had 2 major bouts of network trouble – yesterday afternoon and one today. It appears they were distributed denial of service attacks directed at multiple customers. The amount of traffic was filling both of the datacenter’s incoming pipes. There aren’t any excuses for why this took 2 extended outages to figure out and we’ll be addressing that issue with our provider. All I can offer are our sincere apologies and a promise that this will get better.

During the outages a lot of questions came up regarding multiple carriers, routers, etc. So to answer a couple of those:

• we have redundant transport via XO and Verizon, however this won’t help since traffic will end up filling both.
• we also have redundant routers using VRRP to failover on our side as does the datacenter.

Again, we’re extremely disappointed with how this was handled, but here is some good news. In the next couple of weeks, the datacenter’s core network (not ours, but how we connect to them) will be upgraded. We’re assured this will result in a more stable setup for growth.

Secondly and this was the announcement we referred to earlier this week, we are putting the final touches on a second facility. This space uses separate carriers and we will be managing the network, but everything else stays the same. We hope this expansion allows us to accelerate our growth and offer a higher level of redundancy for customers who need it.

Thanks to everyone for their patience during the past 2 days. For future reference, aside from the chatrooms, the slicehost twitterstream and the network status page are worth bookmarking for quick updates.

Appears to be an upstream issue at the moment, we’ll have more details later in another post. Our apologies for the downtime and flurry of pages/emails/calls you all likely received.

Update 2145 GMT – issue appears to be resolved.

Heads up – we’ll be switching to new versions of our backend software and SliceManager Monday morning. manage.slicehost.com site may experience minor outages as we move things into place and squash any last minute bugs. There will be no service impact to your Slices during this time, but the webconsole, rescue mode, adding slices and upgrades will be unavailable while the site is down.

We’ll be around Monday to keep an eye on things, if after logging into the new site you notice any problems – please let us know. Once things have settled down, we plan to resume new signups and have some exciting news to share.

Background info – Pickled Onion has a new page on the wiki up with more details.

I’m sure all of you Slicers-to-be have noticed the waitlist estimates growing longer over the past few weeks and we feel your pain. The new customers are pouring in and we’re working hard to get people in the door as fast as possible.

Getting the bad news over with: although you can still add yourself to the waitlist, we are temporarily halting new invitations for slices to people on the list. Why? We’re planning to deploy new backend software and an updated version of SliceManager this week. As all of you know, rolling out a new piece of software is a daunting chore and we’ll be focused on this 100%.

Good news – once completed, the new software and some pending announcements will allows us to accelerate growth and hopefully blow through the waitlist quicker than ever.

Ultimately, we didn’t want to give people false time frames for getting a Slice. Our goal is to get everyone aboard ASAP and all of us are working to get it done. We thank everyone for their patience and holding out for us, the notes of encouragement and enthusiasm really pump us up. We plan to make things even better in the coming months!

It was just brought to our attention in the chat room that our form mailer at www.slicehost.com has been queuing mail for a few days instead of delivering it. If you did not receive a response, we apologize and hope to get back to you soon. Feel free to resend a message if needed.

You should have received an email today if any of your Slices need a new IP. The wiki has instructions and examples for various distributions.

One of our biggest requests is multiple IP addresses for a Slice. It’s something we’ve been working toward for sometime. While not quite here yet, we are taking preliminary steps to reach that goal. A required hurdle is the transition away from some of our current IP allocations. We have a process in place to minimize how this impacts our customers, allowing them to use both IPs for a bit to smooth the cutover. If your Slice IP needs an address change, you’ll receive an email soon with detailed instructions.

• a bit of manual work on your Slice to modify the network config
• updating external DNS records pointing to the new IP
• removing the old IP after the cutover

It’s a short-term pain, but ultimately a good thing for all Slicehosters. It grants us portability, more IPs and hopefully multiple IPs for those who need it.

UPDATE: Use this forum thread for comments/questions.

WIKI: DNS updates

We’ve just enabled a new feature that will allow you to resize your Slices.

• via the SliceManager, choose a new Slice size
• resize will begin as soon as resource availability permits
• data is copied from the old slice, preserving your IP
• the synchronization process was designed to minimize downtime
• a notification email is sent on completion and status may be viewed in SliceManager

• upgrades are prorated based on the difference in Slice price and remaining time in the current bill period
• the speed of the process and amount of downtime will vary depending on your system activity and the composition of your filesystem
• the original copy of the Slice is kept for up to a 12 hour verification period

We just sent an email to a batch of customers requesting address and phone number updates and did not BCC. It was a misclicked paste, we truly apologize for this, since privacy is one of our personal pet peeves. Please contact us if that has caused you any trouble.

Major apologies everyone – we just got a heads up from our NOC that a core router software upgrade is taking place tonight from 0230-0500 CST. This is NOT how we typically handle maintenance, but this is coming from upstream.

Cisco has been onsite all weekend (tracking down the issue from the problem last Friday) and has determined it is an IOS bug that requires an upgrade. They are focusing on us specifically to ensure availability and minimize downtime by moving us between the core routers, but there still maybe be network outages of approximately 5 minutes.

We are obviously very frustrated with the entire situation. However, this upgrade is for the best and should prevent future problems. We will likely be onsite during the upgrades, so please contact us with any questions. Thanks everyone for your patience.

chat.slicehost.com will redirect you to the Campfire chatroom for updates.

update – I had the GMT conversion incorrect.

Friday Feb 16 0700-1100 GMT

Upstream Network Maintenance

Impact: a few brief network blips lasting approximately 3 minutes. Please contact us with any questions.

Happy holidays. Just an update for new users joining via the signup codes. We’ve temporarily suspended new users because of some sporadic problems on the new server batch. Once we solve the problem, we’ll start taking customers again. Discussion thread here.

Sorry, we’re sold out until tomorrow afternoon when a new shipment arrives.

UPDATE 11.13.2006 6:11pm, open for business again.

We’ll be starting Xen upgrades in the next few days. You’ll be receiving an email with the date and time we plan to upgrade your server. Ideally, the downtime will be one reboot, as we shutdown and restart the box. If there’s a problem, we’ll roll back. Most importantly, if you compiled anything with our kernel, please get in touch with us, since the kernel will be upgraded in this maintenance. Check the forum discussion thread – as always, we’re open to questions and comments.

We think everyone will be quite pleased with this update, it addresses performance and scheduling on SMP machines. See this post for a benchmark comparing Slices on previous Xen versions.