Don't delete them if we don't have permission for them, since we may get permission for them later via Update. But to listen for Update we need the connection around.