I was just thinking about how I was going to post what sounds like "pun intended" the same problem. In my case it only happens when the engine is cold and as my daughter noted, it sounds like a loose bolt bouncing around inside something that is rotating. It only lasts for less than a minute.

Great diagnostic idea of removing serpentine belt but, will that make it throw check engine light errors ?

Hopefully things will warm up a bit to try it. Currently it's +5F up here in northwestern Vermont.