Could it be that someone has attempted to set the idle speed via the throttle plate stop rather than letting the IAC do its job? It wouldn't be the first time. They may have set the hard stop so that it idled fine when cold but that would surely make it too high when warm. Maybe the throttle body needs a good cleaning along with a readjustment, including the throttle position switch. The following four scans from the Volvo Rex-Regina manual spell out the adjustment procedure. Note, as Art suggests, hearing the "click" isn't proof the switch is actually opening and closing correctly. An ohm meter connected to the contacts should provide verification of both the idle and full throttle switching.

Sorry for the poor image quality. (Second generation scans.)