I bet if you go drive your car you will find that the air is being diverted to the defrost but at a much lower volume. A leak in the floor /defrost diaphragm will allow the vacuum to leak away whenever you go to a throttle position that goes to boost, thus cutting off the source of vacuum. There could be a vaccum leak in the check valve in the supply hose, though that is less likely than the diaphragm leak. Pull the driver's under dash panel and find the yellow hose on the diaphragm unit on the vent case above and to the right of accel pedal. Pull off the hose and plug it. This will eliminat floor position, but I doubt you will notice it, especially compared to the $500 bill you'd get for all the work necessary to replace that unit.