Primer

[homebild]

With a truly paintable caulk, using a little or a lot of primer would make know difference and either amount would adhere just as readily.

I think you just didn’t use a paintable caulk from your desription of the symptoms….