Random perspectives from Cleveland

25 steps to happiness.

9:30 AM in a random state near you, on a day much like yesterday.


Step 1. A deer decides to cook himself at a substation. Hooray for lack of power!
Step 2. UPS's kick in inside the server room.
Step 3. Heat goes up 5 degrees every 10 minutes in the server room due to lack of air conditioning.
Step 4. Drives start failing on foundry server.
Step 5. Mayhem.
Step 6. Power down foundry server, hoping heat caused the issues.
Step 7. Fire up the generator to power fans for the server room.
Step 8. Power comes back on 30 minutes after initially going out.
Step 9. Power on foundry server.
Step 10. Everything looks fine. That's a big relief.
Step 11. Another drive failure on foundry server.
Step 12. RAID backup kicks in.
Step 13. We're OK.
Step 14. Cascading drive failures on foundry server.
Step 15. We're totally and completely screwed.
Step 16. Rebuild server. Keep in mind that we are 24/7 and the cluster hasn't arrived to replace foundry and dip yet.
Step 17. More cascading drive failures. Yay.
Step 18. Rebuild server using replacement disks.
Step 19. See step 17.
Step 20. Get rid of the damn PERC controller. Bastage.
Step 21. See step 18.
Step 22. Reinstall SQL and restore databases.
Step 23. Go home at 9:30 PM.
Step 24. Get paged in the middle of the night.
Step 25. I'm happy! Look how happy I am.