Paint loves a bit of adhesion to help it stick to the second and third coat, so skipping this prepping step is a big no no.
Not only that, but with a good primer, you may not even have to sand down ... piece. (No spotty cracks or corners over here!) After about half an hour of drying, spray paint your color of choice.