I had about 500 rounds worth of residue on my X300 (mounted on an M&P40). A pencil eraser works as I had to use it last time around but requires a bit of elbow grease and also has the risk of scratching.
This time I looked under my sink and found a bottle of "CLR" cleaner I bought to clean my shower. I dabbed a bit on a paper towel and hit the glass with it. The residue came off super fast. I followed up with soap and water in the sink to make sure the stuff didn't attack the annodizing. No problems.
Just a little tid bit of info that worked for me. I don't baby my stuff so I'm not scared.