17
the air does smell sweeter
No comments · Posted by beerorkid in random crap, wilderness park
Cloe was so happy out in wilderness park that day. Tonight I share the intense joy she shows in the above pic.
Feels so good to have the 4 month work nightmare done and over with. Tonights work could of sucked, we prepared to take down around 120 servers for the final fix. The HP guy said it was not necessary and we were done at 6:30. I am so gonna ride my bike tomorrow. Just a bit of moving stuff around and cleaning up in the next few weeks and I can get back to normal work.
tech dork stuff splaining the whole situation below:
We had SAN which housed around 20 LUN’s, the majority being VMware ESX VM’s. While trying to provision two blades to be W2k3 SQL servers we used alteris to image the servers which were still presented and FC attached to the SAN (bad). The LUN that was still presented got overwritten by alteris and wiped the partition table, the VM’s lived in limbo. That LUN was our production environment which contained 60+ VM’s. A few mis fortunate moves wiped them all off and we had to restore all from a very mis configured data protector tape backup system. Right before the incident we were planning on upgrading our EVA 3000 to a EVA 6000 and we decided to just turn on the EVA 6000 to assist with the restore to the EVA 3000. Many VM’s were lost which needed to be created from scratch and many clients needed new SAN space during the clean up month that followed. We placed them on the EVA 3000 since it had the FC switches and got an loaner FC switch from our reseller for the 6000. It took a couple months to get everything on the 3000 to the 6000 which is not easy cuz it involved continuous access licenses between the two SAN’s which actually expired during the transfer time (that sucked big time). We got more temp licenses to achieve it all. There were many bumps in the road going from a basic MPIO driver to a full featured MPIO driver which killed some switchover LUN’s and other assorted nightmares.
Tonight we are all back on one SAN which would be an EVA6000 and all is right with the world. The severe amount of overtime spent by so many of my coworkers, but mostly me and my NA at the Legislature were insane. Downtime cannot happen during regular work hours. I am glowing and will have amazing sleep. Severe kudos to the team I work with and their amazing effort to make things right.
No tags
