It can be very challenging to find a doc to work with on that matter. I don't know your situation or if TRT is indeed indicated for you, so your docs could be right, I don't know.
I couldn't tell you specifically, but acceptance of TRT/HRT in men ranges widely within the medical community. That's changing as evidence continues to grow however. TRT is not without its potential downsides also, which should not be ignored.
Like any relationship, you have to find the right that that works with you vs (seemingly) against your wishes. Me personally (not medical advice) I would not be happy with those T levels, and as one can see from the articles and studies I posted or wrote, the data very much supports the risk/benefit of TRT strongly in favor of TRT in my view.
You could also send them various articles/studies I have posted to see if that helps their position/attitude toward it.
Good luck.