KVM: Really remove a slot when a user ask us so