I have an '01 V40 and yes there is a trick to this. The good news is that it requires no tools and only takes a second. I was tired of seeing posts that said you had to pull the radio and remove the facia, this is total BS.

First I got the cup holder from eBay. I tried dozen pick and pulls but the cup holders were always gone. Guess 'cause they are all on ebay. Anyway, I got one for $30+shipping.

Here is all you need to do to remove it (reinsert it):
OPen the current cupholder half way.
With your other (non-dominant hand) inverted, push up on the upper housing of the opening. There is a plastic pin that keeps the cup holder from hyper extending. Push hard enough for it to clear. You end up pinching your finger, but it only takes a second. PULL! Forcefully but not too hard. Just 5 lbs of force. Should come right out.

Try on the replacement first and you will see where to push and how easy it is.
Then try it for real in the car. You will have it replaced in 2 minutes Tops.