From my understanding, a factory built Colt upper will have a small proof mark on the upper next to delta ring on the ejection port side. Here's a pict on a A2 upper, in the mark closer to the ejection port door. On the M4 uppers, it should be in a similar location. Check both sides of your upper.
Attachment 24625 Attachment 24626
(Images from Urban_Ops from a post over at ARF)
I've built over 100 uppers using individually sold Colt M4 uppers receivers (meaning not disassembled from complete uppers) and none of them had the VP ("Verified Proof" from a test fire) proof mark. All the factory built Colt uppers that I have (several) and upper receivers that I've built from "pull downs" all had VP proof marks.