It’s no big secret that Pastie is hosted at Linode, but from the information I have so far I have no reason to believe anyone has enjoyed root (or even non-root) access to Pastie servers other than myself - despite claims by HTP to the contrary.
I’ll be posting a much longer post about this on my blog in the next few days and I’ll be sure to link to it from here.
Perhaps issues with the load balancer. I’m looking into it. Huge spike in traffic first though - perhaps another attack? In any case it doesn’t seem like the load balancer is doing it’s job properly. The back ends are still up yet the load balancer is shutting down the site.
Hope to know more soon.
Pastie is actually still up if you can visit http://balancer.pastie.org. I’m pointing the main DNS to there now, but will take a bit to update. I’ve been playing with the setup but wasn’t sure I needed to load balance to two front-end caches, but this outage would seem to indicate that if I want 100% uptime that perhaps I do.
Some apparent host disk I/O issues so I’m moving the disk-based Varnish cache that sits in front of Pastie to a different host. Should be back online soon.
Pastie will be down an hour or two today for much needed disk upgrades.
Update: And we’re back online about 38 minutes later. Not so shabby.
First post-Rails Machine DDOS attack. Lots of TCP traffic and connection flooding on port 80 it looked like. Some interruption of service. Only the the front-end was affected. I could even still mostly get to the service if I was willing to wait quite a few seconds. Was just putting some firewall rules in place that seemed to help as the attack died off.
Not sure if varnish has some type of tuning I need to apply or if iptables is where I need to spend most of my effort. Any advice should be appreciated.
Even during the attack http://fastly.pastie.org and http://lb.pastie.org/ continued to work just fine (I don’t think they were targeted, but just saying).
I wouldn’t call it the perfect storm, but a lot of things conspired to keep Pastie down longer than I would have liked. If at the time I had known I was about to get terribly sick, wreck my car, and pretty much be unavailable to work on Pastie for a more than a few days… I might have made a different decision about the course I took getting the service back online.
Ultimately I decided I wanted to completely rewrite the very old deployment scripts (now based on Sprinkle and Capistrano) and rebuild the entire back-end architecture of the service to put us on a more solid foundation for the future.
Then
Previously Pastie was hosted on a single VPS with Rails Machine. Gradually over the years we’ve bumped up the resources as necessary to keep the service running smoothly.
Now
Pastie is now hosted on 4 quad-core Linode VPSs.
This setup already feels much nicer and cleaner than before (not to mention faster). I’m having fun keeping an eye on the resource usage and performance of the individual layers. I’m going to keep an eye on it over the next days and weeks and perhaps expand it or add more redundancy if that seems needed.
This raises hosting costs $140 higher per month than they were before ($0). I’ll be adding a Donate/Membership feature (much like 5by5’s) to Pastie soon for those who would like to either offer a one-time contribution or help contribute a small amount each month. It’s really amazing what $140 a month will buy you these days though.
Thanks for everyone who wrote me and offered suggestions of words of encouragement while we were down. It means a lot to have loyal users who love a service. I’m very glad we’re back up and we can all get back to pasting on the best paste service in the world. :)
DB server is sitting idle. App servers are ready to go. Working on bringing up a few different front-ends and testing things then deciding where to park the main pastie.org domain.
Coming up soon:
It will be interesting to see how these fare during the next DDOS (if there is another). I guess I assume someone who actually cares would try and take ALL of Pastie offline, not just the main URL… so if these stay up and handle attacks I can judge how the actual services compare at mitigating DDOS tracks… of course maybe that is a broken assumption… who knows. Often there isn’t a lot of rhyme or reason to such things.
And at the very least if the attacker only bothers the main domain then anyone in the know can just switch to an alternative and you’re good to go. :-)
These aren’t really designed to be “pick your favorite” but rather as “go tos” in the case of future issues… when I find out what works best of course I’d move the main domain there as well. I’m just putting this out there for anyone who is really really missing the site and wanting it back ASAP. :) These may be up before pastie.org… (or many not). Pastie.org has a lot shorter TTL now than *.pastie.org… we’ll see how tonight goes.
Still a lot to learn about silly DDOS stuff.
First, my apologies. I’m really truly sorry.
I did make some progress setting up new hosting while at RailsConf, but not enough. Unfortunately I felt for what I paid (booked too many things last minute) I really needed to focus on the conference and then deal with Pastie after it was over. Unfortunately I think I picked up the flu or something and it only got worse on my flights home. All I’ve done for the past few days is sleep and be sick. Not to mention I wrecked my car on the drive home from the airport as well (not my fault, huge ass object in the middle of the Interstate). It’s just not been a good last few days at all.
I’m finally starting to really feel better I hope to make some real progress on getting things back online tomorrow (Sunday).
I can’t really say he misrepresents anything… of course I don’t think he’s disagreeing with anything I say… he’s just explaining their rational for dropping the hosting arrangement so quickly. 100s of complaints does sound really high (he’s speaking in aggregate over a few years)… after some tweaks to the site a while back we receive significantly fewer escalated complaints than we used to (many of them from automated bots, not humans).
I’m only going from memory but in my recollection it seems Rails Machine would forward an abuse notice every few weeks lately. It really used to be a huge pain, now it’s just a pain.
