linkage or throttle shafts just not returning completely when pedal is lifted... A little lubricant would probably fix that. Also check that the throttle return springs are still there... they can and do break... You can also try stronger throttle return springs, but the stronger those are, the faster your throttle shafts will wear.

--
-Matt '70 145s, '65 1800s, '66 122s wagon, others inc. '53 XK120 FHC

I'll toss in another possibility - a sticky vacuum advance in the distributor. Monie was sticky and sluggish, and caused both a very slow return to idle speed and a very sluggish acceleration of the line. Unplug the hose and take it for a spin, see if it is any better.

The other suggestions already offered (vacuum 'leaks' and throttle linkages that don't always close 100%) are more likely though. In particular, HS6's can tend to hang the throttle valve slightly open when the throttle shafts wear and sag slightly. If you do a quick dab on the throttle and it causes the idel to drop to normal that might be the culprit.
--
I'm JohnMc, and I approved this message.

I had a similar problem with my 72 164 D Jet. The throttle return was sluggish and I sensed that the gas pedal was stiff. A squirt of lithium lubricant on the linkage joint where it attaches to a bracket at the fire wall did the trick. Lubricating the various ball joints in the linkage is not needed since lubricant will eventually gum up these joints. Keep these clean.

The intake manifold may be coming loose, especially the rear nuts/washers, and the throttle shafts in the carbs may have play which admits air, or causes the shafts to hang up and not return to the idle stop screws.