USB: serial: don't call release without attach