You build speed as your form improves. You need to be able to do the task correctly before you add speed. No one said anything about not training at speed, you need to have developed form to be consistent.
I work off index points and know where I want to be at each stage of the draw.
The thing that trips up some folks may be what you are alluding to, they don't push themselves to failure and then back off to last success as their new start point.
Another thing is accepting less than optimal results in practice which just ingrains bad habits.
EAT - Reference hand speed (since I just digested that part of your post) work it in stages and from different variables - start IPSC surrender to index touch, to hand PROPER retention release and PROPER hand placement on the weapon - user a timer to work to your par focus on driving the hand to the index touch, not just moving it; once you are making par move your start point - arm extended forward at shoulder height, arm extended forward as if opening a doorknob, etc.
But for the complete product, first you need to have the optimal draw grooved.