I wouldn't recommend leaving them off just because you may get extra scratching of the paint. As for the rivets, I too have had one of my mud flaps come loose due to rivets popping off. And as also already mentioned, just get a rivet gun, however, DO NOT use regular sheet metal rivets. The rivets used to hold in the mud flaps from the factory have plastic/rubber washer like ends on them to help the rivet from "going" through the mud flap. They literally cost about 50 cents CDN each and work a heck of a lot better then the regular metal ones without the added piece.