RDMA/cma: Reenable device removal on passive side