Not sure if it was the dreamhost guys moving me to my own instance of apache or me finally upgrading to the newest wordpress, but the site loads quick as hell now.
Oh and to add to the awesomeness. The guy from dreamhost who was helping me uses XGL. he was glad to help.
Good to know the peeps who host my shit, use what I host and Linux. made a little snide remark towards ATI as well
F&#K ATI and their shitty drivers.