The pins are tempered and pretty hard, aluminum is soft and the drill will most likely wander into the aluminum because the pin isn't solid, it is split and that split will pull the drill into the aluminum. At the point you are I think a bigger hammer and a good punch is a better solution than drilling.
While I didn't read the entire thread, check to see if some knothead didn't drive a 1/16" punch into the center of it and wedged it outwards. I had someone do this the other day on a bolt catch, broke a small section of it off inside of the roll pin.
Thanks for the advice everyone
Finally got the pin out, thanks for all the tips guys. I ended up using a small drill bit on the top side to weaken it then knocked it out from the bottom with a punch. Got the new one installed no problem. Thanks again