kernel: do not cleanup what should be already cleaned up

On spec driver remove, do not clean-up VIC stuff. The VIC should be cleaned
by the mezzanine driver when it release all its handlers (see patch f7ee35460).
Just leave a reminder for driver developer when something goes wrong.

Signed-off-by: Federico Vaga <federico.vaga@cern.ch>