Maybe I need to clarify. Idle is controlled almost exclusively by the IAC...little thing that meters air in going into the intake after the throttle plate using a spinning disc (oversimplification). If it is working correctly, you idle well as long as the ecu gets the throttle-closed signal from the TPS.

A significant vac leak coupled with a significant vac requirement to refill the brake booster temporarily upsets the volume of air going to the motor and the IAC needs a second or two to compensate.

Likely your vac leak is the line to the booster, the check valve on the booster or the booster itself but I did not want you to zero in on that one stream to the exclusion of all others.

If either line to the IAC has a leak, the IAC is unplugged, the line to the flame trap (probably too small), the line to the climate controls or its check valve, the line to the evap canister, the line to the cruise control, etc...any of them are bad and it could be your culprit.

There is also the chance it is something else but your specific series of events screams vac leak.