I always figured primer was just extra work. Last week, I left it out on a shop window install. The sealant peeled away in two days of wind. Now I know primer is needed for a good seal. That redo job was not fun at all.
That "two days of wind" detail is key. People think wind is just a pulling force, but it's also blasting tiny dust and grit against the surface. That grit sticks to the unprimed material and creates a weak layer the sealant bonds to instead of the actual frame. Primer's main job is often just to lock that contamination out.