So in our adventures at Leaf, building out our new environment, Will and I ran into a persistent problem – EC2 doesn’t guarantee that an instance will ever be shut down gracefully. No guarantees for
/etc/init.d, whatever. This is particularly problematic for dealing with Chef, where an instance needs to be deleted from the Chef datastore when it goes away. If you don’t,
knife search will happily return tens or hundreds of nodes that no longer exist. Which is great.
No, wait. It’s crap.
asger. Since autoscaling groups can publish notifications to SNS,
asger will watch an SQS queue subscribed to SNS and execute arbitrary tasks based on the up/down pattern. Created nodes invoke up functions, terminated nodes invoke down functions, life is good. Currently there’s a task for deregistering Chef nodes, but
asger is pretty flexible–Route 53 subscriptions, tie-ins to systems like Consul, that sort of thing. You can get asger via RubyGems with
gem install asger or on GitHub; use it in good health.