lookup_server_connector creates a lot of oc_jobs entries, high DB load during cron, and tries to sync even if disabled in options #13396
Comments
alx-tuilmenau
added
0. Needs triage
|
Right. I see what is happening. The lookup server is indeed disabled. But since we do not have the state we try to tell it to remove all the data. @schiessle any idea how to solve this? |
|
Ok I think I have a solution in mind. The moment that the lookupserver uploads get disabled. We should insert a background job. This job should
|
|
@alx-tuilmenau @rullzer One workaround is to set an empty value for the |
yes, this would be a quick workaround to solve the problem. As a more general solution, I wounder if we should introduce a "dsiable_lookup_server" config.php switch which would be:
|
|
In my opinion, if the "global adressbook" is not activated in the admin options, there should be no requests at all. Noone expects the leak of the usernames to an external server if the option is turned off, it's also a GDPR problem. If I set lookup_server to an empty value and re-activate the app, a lot of jobs are created very fast again, it's the same problem (it retries 5 times via cron). Maybe it depends on the configuration here, because outgoing http(s) is done via proxy. It looks like every LDAP sync, maybe on every login, will trigger the lookup_server sync. Setting lookup_server to an empty string is not a solution. |
|
@alx-tuilmenau ah, now I saw that a setting already exists "Allow users to publish their data to a global and public address book", I missed it when I looked at the settings matrix the first time. Yes, we could use this to disable the connection to the lookup server completely which would come with the draw back that already published data would stay at the lookup server. So we would probably need something like suggested from @rullzer #13396 (comment)
With a quick look at the code I confirm this behavior. The connector will try to update the lookup server, will fail and create the background job, the background job will disappear after first try. But of course on every sync with the LDAP we will create them again and again. So setting the lookup_server to '' will at least help in that regard that the background jobs are cleaned up again. I will work on a fix for this part, this should be straight forward. |
|
We can see the same behaviour after upgrading from NC13 to NC14. Unfortunately the enormous job queue kills the whole server after a while. Is there something we can do right now to resolve the issue? (empty vaule for lookup_server parameter didn't help) |
|
@jngeiger See my first post, because I don't use federation, I moved apps/lookup_server_connector folder to a different location (outside nextcloud/docroot) and deleted all entries with class like %LookupServer% from oc_jobs table. |
|
I'm also seeing this on my company server with AD-backed authentication. I noticed CPU load slowly creeping up after upgrading to 14, and finally tracked it back to this issue. Removing lookup_server_connector to mitigate the issue has it under control for now, but I'd like to not have to do that after every upgrade. |
|
Is this fix likely to get a backport to 15.x? |
|
There is a backport for NC 15: |
|
Hi, I also had this bug that generated enormous amount of UPDATE and INSERT each 15 minutes. Each cron job had a duration of >10m with taking 100% CPU. I was running 16.0.1 and I believe that the fix is included in that version. If yes, it didn't work for me. Since we don’t use federation currently, the fix was to delete all those pointless jobs in the DB (14000+) : And removing the offending app Just wanted to report that. |
|
@schiessle shouldn't #13401 handle |
Steps to reproduce
Expected behaviour
There should be no connections to external servers. The system should run stable.
Actual behaviour
A lot of traffic to lookup.nextcloud.com. Increasing, high DB load. Leak of personal information (usernames) to nextcloud.com without permission.
It looks like lookup_server_connector app is trying to update lookup.nextcloud.com on ervery user change, also if users are synced via LDAP (and nothing is changed), even if the global adressbook and all other federation options are disabled.
This often fails (maybe server fail / network / proxy timeout) and creates a entry in oc_jobs for retrying during cronjob.
I got about 300k of lookup retries in the oc_jobs table, The cron job has no chance to get this done, so all other jobs (mails) are not processed.
My temporary solution was to delete the lookup_server_connector app completely (occ app:disable didn't work) and deleting all entries for this class in oc_jobs.
Server configuration
Operating system:
Debian 9
Web server:
Apache 2.4
Database:
MariaDB (10.1.37)
PHP version:
7.2
Nextcloud version: (see Nextcloud admin page)
14.0.4.2
Updated from an older Nextcloud/ownCloud or fresh install:
Updated
Signing status:
Signing status
No errors have been found.
Nextcloud configuration:
Config report
{ "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "***", "c***" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "14.0.4.2", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "dbdriveroptions": { "1002": "SET wait_timeout = 28800" }, "installed": true, "default_language": "de", "updatechecker": true, "has_internet_connection": true, "memcached_servers": [ [ "localhost", 11211 ] ], "appstoreenabled": true, "ldapIgnoreNamingRules": false, "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory", "loglevel": 1, "trusted_proxies": "***REMOVED SENSITIVE VALUE***", "forwarded_for_headers": [ "HTTP_X_FORWARDED_FOR" ], "overwritehost": ***", "overwriteprotocol": "https", "overwrite.cli.url": "***", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_smtpmode": "smtp", "mail_domain": "***REMOVED SENSITIVE VALUE***", "skeletondirectory": "\/data\/nextcloud_skel", "theme": "tui", "activity_expire_days": 14, "integrity.check.disabled": true, "memcache.local": "\\OC\\Memcache\\APCu", "memcache.distributed": "\\OC\\Memcache\\Redis", "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 6379, "password": "***REMOVED SENSITIVE VALUE***" }, "knowledgebaseenabled": false, "defaultapp": "apporder", "sharing.minSearchStringLength": 4, "sharing.maxAutocompleteResults": 5, "maintenance": false, "debug": false, "htaccess.RewriteBase": "\/", "proxy": "****", "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpport": "25", "updater.release.channel": "production" }, "apps": { "QuotaAdmin": { "enabled": "no", "installed_version": "0.0.1", "types": "" }, "accessibility": { "enabled": "yes", "installed_version": "1.0.1", "types": "" }, "activity": { "enabled": "yes", "installed_version": "2.7.0", "notify_email_announcementcenter": "1", "notify_email_calendar": "0", "notify_email_calendar_event": "0", "notify_email_calendar_todo": "0", "notify_email_circles_as_member": "0", "notify_email_circles_as_moderator": "1", "notify_email_comments": "0", "notify_email_favorite": "0", "notify_email_file_changed": "0", "notify_email_file_created": "0", "notify_email_file_deleted": "0", "notify_email_file_downloaded": "0", "notify_email_file_restored": "0", "notify_email_public_links": "0", "notify_email_remote_share": "0", "notify_email_shared": "0", "notify_email_systemtags": "0", "notify_setting_batchtime": "86400", "notify_setting_self": "1", "notify_setting_selfemail": "0", "notify_stream_calendar": "1", "notify_stream_calendar_event": "1", "notify_stream_calendar_todo": "1", "notify_stream_circles_as_member": "1", "notify_stream_circles_as_moderator": "1", "notify_stream_comments": "1", "notify_stream_favorite": "1", "notify_stream_file_changed": "1", "notify_stream_file_created": "1", "notify_stream_file_deleted": "1", "notify_stream_file_downloaded": "1", "notify_stream_file_favorite": "0", "notify_stream_file_restored": "1", "notify_stream_public_links": "1", "notify_stream_remote_share": "1", "notify_stream_shared": "1", "notify_stream_systemtags": "1", "types": "filesystem" }, "admin_audit": { "enabled": "no", "installed_version": "1.2.0", "types": "logging" }, "announcementcenter": { "enabled": "yes", "installed_version": "3.3.1", "types": "logging" }, "apporder": { "enabled": "yes", "hidden": "[]", "installed_version": "0.6.0", "order": "[\"\/apps\/files\/\",\"\/apps\/activity\/\",\"\/apps\/calendar\/\",\"\/apps\/tasks\/\",\"\/apps\/circles\/\",\"\/apps\/spreed\/\",\"\/apps\/notes\/\",\"\/apps\/contacts\/\",\"\/apps\/gallery\/\",\"\/apps\/announcementcenter\/\",\"\/apps\/gpxedit\/\",\"\/apps\/quotaadmin\/\"]", "types": "" }, "audioplayer": { "enabled": "no", "installed_version": "2.1.0", "types": "filesystem" }, "backgroundjob": { "lastjob": "515531" }, "bookmarks": { "enabled": "no", "installed_version": "0.10.1", "types": "" }, "bruteForce": { "whitelist_1": "***" }, "bruteforcesettings": { "enabled": "yes", "installed_version": "1.3.0", "types": "" }, "calendar": { "enabled": "yes", "installed_version": "1.6.4", "types": "" }, "circles": { "allow_federated": "0", "allow_linked_groups": "0", "enabled": "yes", "installed_version": "0.15.2", "members_limit": "1000", "types": "" }, "cloud_federation_api": { "enabled": "yes", "installed_version": "0.0.1", "types": "filesystem" }, "comments": { "enabled": "yes", "installed_version": "1.4.0", "types": "logging" }, "contacts": { "enabled": "yes", "installed_version": "2.1.8", "types": "" }, "core": { "backgroundjobs_mode": "cron", "installed.bundles": "[\"CoreBundle\"]", "installedat": "1506428024.7302", "lastcron": "1546800408", "lastupdateResult": "[]", "lastupdatedat": "1546799081", "moveavatarsdone": "yes", "previewsCleanedUp": "1", "public_files": "files_sharing\/public.php", "public_webdav": "dav\/appinfo\/v1\/publicwebdav.php", "scss.variables": "385ed5e08336aefb25c8b98f0028dbe6", "shareapi_allow_group_sharing": "yes", "shareapi_default_permission_canshare": "no", "shareapi_default_permissions": "15", "umgmt_show_backend": "false", "umgmt_show_email": "false", "umgmt_show_last_login": "false", "updater.secret.created": "1545907978", "vendor": "nextcloud" }, "dashboard": { "enabled": "no", "installed_version": "4.0.5", "types": "" }, "dav": { "buildCalendarSearchIndex": "yes", "enabled": "yes", "installed_version": "1.6.0", "types": "filesystem" }, "drop_account": { "enabled": "yes", "installed_version": "0.0.13", "types": "" }, "federatedfilesharing": { "enabled": "yes", "installed_version": "1.4.0", "types": "" }, "federation": { "enabled": "yes", "installed_version": "1.4.0", "types": "authentication" }, "files": { "cronjob_scan_files": "500", "default_quota": "10 GB", "enabled": "yes", "installed_version": "1.9.0", "types": "filesystem" }, "files_clipboard": { "enabled": "no", "installed_version": "0.7.1", "types": "" }, "files_downloadactivity": { "enabled": "yes", "installed_version": "1.3.0", "types": "filesystem" }, "files_external": { "allow_user_mounting": "no", "enabled": "no", "installed_version": "1.3.0", "types": "filesystem", "user_mounting_backends": "ftp,dav,owncloud,sftp,amazons3,dropbox,googledrive,swift,\\OC\\Files\\Storage\\SFTP_Key" }, "files_fulltextsearch": { "enabled": "yes", "files_audio": "0", "files_encrypted": "0", "files_external": "0", "files_federated": "0", "files_group_folders": "0", "files_image": "0", "files_local": "1", "files_office": "1", "files_pdf": "1", "files_size": "5", "installed_version": "1.1.1", "types": "filesystem" }, "files_markdown": { "enabled": "yes", "installed_version": "2.0.5", "types": "" }, "files_pdfviewer": { "enabled": "yes", "installed_version": "1.3.2", "types": "" }, "files_retention": { "enabled": "yes", "installed_version": "1.3.1", "types": "filesystem" }, "files_rightclick": { "enabled": "yes", "installed_version": "0.8.4", "types": "" }, "files_sharing": { "enabled": "yes", "incoming_server2server_share_enabled": "no", "installed_version": "1.6.2", "lookupServerUploadEnabled": "no", "outgoing_server2server_share_enabled": "no", "types": "filesystem" }, "files_texteditor": { "enabled": "yes", "installed_version": "2.6.0", "types": "" }, "files_trashbin": { "enabled": "yes", "installed_version": "1.4.1", "types": "filesystem,dav" }, "files_versions": { "enabled": "yes", "installed_version": "1.7.1", "types": "filesystem,dav" }, "files_videoplayer": { "enabled": "yes", "installed_version": "1.3.0", "types": "" }, "firstrunwizard": { "enabled": "no", "installed_version": "2.1", "types": "logging" }, "fulltextsearch": { "app_navigation": "0", "cron_err_reset": "1546796703", "enabled": "yes", "installed_version": "1.1.0", "provider_indexed": "{\"files\":\"1\",\"test_provider\":\"0\"}", "search_platform": "OCA\\FullTextSearch_ElasticSearch\\Platform\\ElasticSearchPlatform", "types": "" }, "fulltextsearch_elasticsearch": { "analyzer_tokenizer": "standard", "elastic_host": "***", "elastic_index": "my_index", "enabled": "yes", "installed_version": "1.0.2", "types": "" }, "gallery": { "enabled": "yes", "installed_version": "18.1.0", "types": "" }, "gpxedit": { "enabled": "yes", "installed_version": "0.0.10", "types": "" }, "groupfolders": { "enabled": "no", "installed_version": "1.1.0", "types": "filesystem" }, "guests": { "allow_external_storage": "false", "enabled": "yes", "group": "guests", "installed_version": "1.0.0", "types": "authentication", "usewhitelist": "true", "whitelist": "activity,apporder,circles,files_markdown,files_sharing,files_texteditor,files_trashbin,files_versions,gallery,notifications,spreed" }, "logreader": { "enabled": "yes", "installed_version": "2.0.0", "levels": "00111", "ocsid": "170871", "relativedates": "1", "types": "" }, "lookup_server_connector": { "enabled": "yes", "installed_version": "1.2.0", "types": "authentication" }, "metadata": { "enabled": "yes", "installed_version": "0.8.0", "types": "" }, "nextcloud_announcements": { "enabled": "no", "installed_version": "1.1", "types": "logging" }, "notes": { "enabled": "yes", "installed_version": "2.5.1", "types": "" }, "notifications": { "enabled": "yes", "installed_version": "2.2.1", "types": "logging" }, "oauth2": { "enabled": "yes", "installed_version": "1.2.1", "types": "authentication" }, "password_policy": { "enabled": "no", "installed_version": "1.2.2", "types": "" }, "provisioning_api": { "enabled": "yes", "installed_version": "1.4.0", "types": "prevent_group_restriction" }, "quota_warning": { "alert_email": "yes", "enabled": "yes", "installed_version": "1.3.0", "repeat_warning": "14", "types": "filesystem", "warning_email": "yes" }, "quotaadmin": { "enabled": "yes", "installed_version": "0.0.1", "types": "" }, "serverinfo": { "enabled": "yes", "installed_version": "1.4.0", "types": "" }, "sharebymail": { "enabled": "no", "installed_version": "1.2.0", "types": "filesystem" }, "spreed": { "enabled": "yes", "installed_version": "4.0.2", "signaling_ticket_secret": *** "stun_servers": "[\"stun.nextcloud.com:443\"]", "types": "prevent_group_restriction" }, "support": { "enabled": "no", "installed_version": "1.0.0", "types": "" }, "survey_client": { "enabled": "no", "installed_version": "1.0.0", "types": "" }, "systemtags": { "enabled": "yes", "installed_version": "1.4.0", "types": "logging" }, "tasks": { "enabled": "yes", "installed_version": "0.9.8", "types": "" }, "theming": { "enabled": "no", "installed_version": "1.3.0", "types": "logging" }, "twofactor_backupcodes": { "enabled": "yes", "installed_version": "1.3.1", "types": "" }, "twofactor_totp": { "enabled": "yes", "installed_version": "1.5.0", "types": "" }, "updatenotification": { "enabled": "yes", "installed_version": "1.4.1", "types": "", "update_check_errors": "0" }, "user_ldap": { "_lastChange": "1545915889", "background_sync_interval": "1800", "background_sync_offset": "0", "background_sync_prefix": "", "cleanUpJobOffset": "4600", "enabled": "yes", "has_memberof_filter_support": "0", "home_folder_naming_rule": "attr:uid", "installed_version": "1.4.0", "last_jpegPhoto_lookup": "0", "ldap_agent_password": "***REMOVED SENSITIVE VALUE***", "ldap_attributes_for_group_search": "", "ldap_attributes_for_user_search": "sn\nuid\nmail", "ldap_backup_host": "", "ldap_backup_port": "", "ldap_base": "ou=user,o=uni", "ldap_base_groups": "ou=user,o=uni", "ldap_base_users": "ou=user,o=uni", "ldap_cache_ttl": "3600", "ldap_configuration_active": "1", "ldap_default_ppolicy_dn": "", "ldap_display_name": "sn", "ldap_dn": "cn=proxy-nextcloud,***i", "ldap_dynamic_group_member_url": "", "ldap_email_attr": "mail", "ldap_experienced_admin": "1", "ldap_expert_username_attr": "uid", "ldap_expert_uuid_group_attr": "", "ldap_expert_uuid_user_attr": "uid", "ldap_gid_number": "gidNumber", "ldap_group_display_name": "cn", "ldap_group_filter": "", "ldap_group_filter_mode": "0", "ldap_group_member_assoc_attribute": "uniqueMember", "ldap_groupfilter_groups": "", "ldap_groupfilter_objectclass": "", "ldap_host": "ldap:\/\/localhost", "ldap_login_filter": "***", "ldap_login_filter_mode": "0", "ldap_loginfilter_attributes": "mail", "ldap_loginfilter_email": "0", "ldap_loginfilter_username": "1", "ldap_nested_groups": "0", "ldap_override_main_server": "", "ldap_paging_size": "0", "ldap_port": "389", "ldap_quota_attr": "", "ldap_quota_def": "", "ldap_tls": "0", "ldap_turn_off_cert_check": "0", "ldap_turn_on_pwd_change": "0", "ldap_user_display_name_2": "mail", "ldap_user_filter_mode": "0", "ldap_userfilter_groups": "", "ldap_userfilter_objectclass": "inetOrgPerson", "ldap_userlist_filter": "***", "types": "authentication", "use_memberof_to_detect_membership": "1" }, "workflowengine": { "enabled": "yes", "installed_version": "1.4.0", "types": "filesystem" } } }Are you using external storage, if yes which one: local/smb/sftp/...
no
Are you using encryption: yes/no
no
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
LDAP
LDAP configuration (delete this part if not used)
LDAP config
hasMemberOfFilterSupport | 0 | | hasPagedResultSupport | | | homeFolderNamingRule | attr:uid | | lastJpegPhotoLookup | 0 | | ldapAgentName | cn=proxy-nextcloud | | ldapAgentPassword | *** | | ldapAttributesForGroupSearch | | | ldapAttributesForUserSearch | sn;uid;mail | | ldapBackupHost | | | ldapBackupPort | | | ldapBase | ou=user,o=uni | | ldapBaseGroups | ou=user,o=uni | | ldapBaseUsers | ou=user,o=uni | | ldapCacheTTL | 3600 | | ldapConfigurationActive | 1 | | ldapDefaultPPolicyDN | | | ldapDynamicGroupMemberURL | | | ldapEmailAttribute | mail | | ldapExperiencedAdmin | 1 | | ldapExpertUUIDGroupAttr | | | ldapExpertUUIDUserAttr | uid | | ldapExpertUsernameAttr | uid | | ldapGidNumber | gidNumber | | ldapGroupDisplayName | cn | | ldapGroupFilter | | | ldapGroupFilterGroups | | | ldapGroupFilterMode | 0 | | ldapGroupFilterObjectclass | | | ldapGroupMemberAssocAttr | uniqueMember | | ldapHost | ldap://localhost | | ldapIgnoreNamingRules | | | ldapLoginFilter |(objectclass=inetOrgPerson) [modified to hide internal attributes] | ldapLoginFilterAttributes | mail | | ldapLoginFilterEmail | 0 | | ldapLoginFilterMode | 0 | | ldapLoginFilterUsername | 1 | | ldapNestedGroups | 0 | | ldapOverrideMainServer | | | ldapPagingSize | 0 | | ldapPort | 389 | | ldapQuotaAttribute | | | ldapQuotaDefault | | | ldapTLS | 0 | | ldapUserAvatarRule | default | | ldapUserDisplayName | sn | | ldapUserDisplayName2 | mail | | ldapUserFilter (objectclass=inetOrgPerson) [modified to hide internal attributes] | ldapUserFilterGroups | | | ldapUserFilterMode | 0 | | ldapUserFilterObjectclass | inetOrgPerson | | ldapUuidGroupAttribute | auto | | ldapUuidUserAttribute | auto | | turnOffCertCheck | 0 | | turnOnPasswordChange | 0 | | useMemberOfToDetectMembership | 1 |Insert your browser log here, this could for example include:
a) The javascript console log
b) The network log
c) ...
The text was updated successfully, but these errors were encountered: