In my expeience with ARs, if the lower takedown pins are hard to push out, then something else is wrong, usually something as simple as the upper and lower not being porperly aligned with each other.
As far as the hammer or the punch not making contact with the lower, if the pins truly are stuck, the punch can bounce after being hammered as an example.

