If you're worried about cosmetics yes, will it matter and effect function? No. You may have a safety that's milled a little to far to one side. I see this on occasion. I find it's a safety problem not a lower problem. In your case your safety and lower are RRA.
Just touch it up with some Birchwood Casey aluminum black or Krylon and rock on.