mm: reduce atomic use on use_mm fast path