gulash

I actually do the exact opposite with inserts and vectors.  If I'm only getting one good strong band from PCR, I usually just digest the insert, then PCR cleanup only to keep away from the contaminants from a gel extraction (I only gel extract if I'm getting multiple bands).  But I always gel extract the cut vector, since you want to avoid getting uncut, or supercoiled DNA in with your ligation at all costs (you don't want single-cut either, but that's usually hard to spot on a gel).  The contaminants I normally get from gel extraction are bad enough that I've re-done my PCR off my gel extracted insert, then run a PCR cleanup off that to keep the insert I use in the ligation as clean as possible.

As bob1 said, you almost surely just have uncut vector contaminating your ligation.  Without a control vector-alone ligation, you have no way of knowing how many colonies is reasonable to screen for insert.  If you don't see more than 2:1 colonies on your +insert plates versus your control plates, I'd consider it a failed ligation and just repeat the cloning from the "last known good" step (or even from the beginning if I'm feeling paranoid).  If you're only at 1.5:1, then you could probably try 10 colonies or something to try to salvage the work you've done, but I'd simultaneously restart the cloning so I can hit the ground running if none of them have insert.