Nextcloud 18.0.2 uses a lot of ressources #300

Thatoo · 2020-05-02T09:32:46Z

I guess its glances screenshot will summarize better than I would.
Before I update to 18.0.2, nextcloud wasn't taking much ressources, only synapse and the whole server was running below 50% charge of CPU.
Since I updated to 18.0.2 (2 days ago), I experienced a lot of problem reaching synapse or nextcloud services and I discovered why this morning.

lapineige · 2020-05-02T09:41:53Z

Could you try to upgrade to 18.0.3 ? (or maybe testing with 18.0.4 is you wish)
Just in case it's a Nextcloud bug.

Thatoo · 2020-05-02T09:52:13Z

Thank you @lapineige
Master branch is on 18.0.2, not 18.0.3? Is there a branch on 18.0.3?
Should I update from testing (18.0.4)? On this server, I prefer going with stable branches.. but I trust you so if you tell me that #274 is about to be merged, ok. Well, I can also wait for it to be merged.
Is it possible to stop an app like
sudo yunohost app nextcloud stop ?
I could do that till it get merged.

lapineige · 2020-05-02T10:13:45Z

Oh that's correct, sorry, the PR wasn't merged yet. It's almost about to be merged, yet it's maybe safer to first check if the error comes from somewhere else than a 18.0.2 bug.

You can't stop an app like that. Yep you can stop php7.3, if nextcloud is the only one to use it.

Did you take a look at the logs ? (from Nextcloud admin interface for instance)

maniackcrudelis · 2020-05-02T10:23:51Z

@Thatoo a few question

Are you running on an ARM board ?
Do you have many users on your Nextcloud ?
And especially do you have third party applications connecting often to your nextcloud ? Like an Android app or anything that maintain a connection on your server.
Is this overload of your server is constant or only when using nextcloud ?

This PR fix a "regression" that actually overload the CPU when an user connect to Nextcloud.
If you have the load average in glance, I would like to see that info.

Thatoo · 2020-05-02T10:30:42Z

How can I know if nextcloud is the only one using php7.3 when I don't even know that it is itself using php7.3? :-)

When I go to yunohost's admin > applications > nextcloud > debug, I have logs for php7.0-fpm not php7.3-fpm is not normal?

However, in Services, I have a php7.3-fpm service running.

You were right, there are a lot of debug coming in the nextcloud logs, mostly this one:
Deprecated event type for OCA\User_LDAP\User\User::postLDAPBackendAdded: null
and
initializing paged search for Filter (&(|(objectclass=posixAccount))(uid=thatoo)) base Array ( [0] => ou=users,dc=yunohost,dc=org ) attr Array ( [0] => entryuuid [1] => nsuniqueid [2] => objectguid [3] => guid [4] => ipauniqueid [5] => dn [6] => uid [7] => samaccountname [8] => memberof [9] => userquota [10] => mail [11] => displayname [12] => jpegphoto [13] => thumbnailphoto ) limit 500 offset 0

Thatoo · 2020-05-02T10:39:03Z

@maniackcrudelis

* Are you running on an ARM board ?

No, a Core2Duo

* Do you have many users on your Nextcloud ?

No, about 5 I guess that use time to time (mostly syncing)

* And especially do you have third party applications connecting often to your nextcloud ? Like an Android app or anything that maintain a connection on your server.

Yes

* Is this overload of your server is constant or only when using nextcloud ?

Constant, I actually didn't notice it at first. I thought it was coming from Synapse as I was experimenting a lot of disconnection with matrix but that was not making sense as synapse has not been updating for weeks. When I saw the result of glances, and I saw all this 8 nextcloud processus going crazy, it then made sense as I've just updated nextcloud.

This PR fix a "regression" that actually overload the CPU when an user connect to Nextcloud.
If you have the load average in glance, I would like to see that info.
The load average is

1 min | 10.31
5 min | 9.42
15 min | 9.05

maniackcrudelis · 2020-05-02T10:43:19Z

Indeed, you're quite overloaded...
Maybe you should try to tail -F nextcloud log to see what it's currently doing.
And maybe also fail2ban log, just in case.

To put a core2duo on its knee, nextcloud should really be overwhelmed !

Thatoo · 2020-05-02T10:43:21Z

Deprecated event type for OCA\User_LDAP\User\User::postLDAPBackendAdded: null
gives me
{"reqId":"YxmFL2WtgYUGNAw9CUE8","level":0,"time":"2020-05-02T10:38:38+00:00","remoteAddr":"2.10.1.1","user":"--","app":"no app in context","method":"PROPFIND","url":"/nuage-nextcloud/remote.php/dav/calendars/ **_name_of_user_**/contact_birthdays/","message":"Deprecated event type for OCA\\User_LDAP\\User\\User::postLDAPBackendAdded: null","userAgent":"DAVx5/2.6.5-ose (2020/03/10; dav4jvm; okhttp/3.12.10) Android/8.1.0","version":"18.0.2.2","id":"5ead4daf41e3b"}

Ready for a paged search
gives me
{"reqId":"BOUrRGRr12dgRjS0yPRG","level":0,"time":"2020-05-02T10:38:38+00:00","remoteAddr":"2.10.1.1","user":"--","app":"user_ldap","method":"PROPFIND","url":"/nuage-nextcloud/remote.php/dav/calendars/**_name_of_user_**/contact_birthdays/","message":"Ready for a paged search","userAgent":"DAVx5/2.6.5-ose (2020/03/10; dav4jvm; okhttp/3.12.10) Android/8.1.0","version":"18.0.2.2","id":"5ead4daf41e68"}

and
{"reqId":"tJ7FjY74GxbeEiwwwaGd","level":0,"time":"2020-05-02T10:42:24+00:00","remoteAddr":"2.10.1.1","user":"--","app":"user_ldap","method":"PROPFIND","url":"/nuage-nextcloud/remote.php/dav/calendars/**_name_of_user_**//contact_birthdays/","message":"initializing paged search for Filter (&(|(objectclass=posixAccount))(uid=titouan)) base Array\n(\n [0] => ou=users,dc=yunohost,dc=org\n)\n attr Array\n(\n [0] => entryuuid\n [1] => nsuniqueid\n [2] => objectguid\n [3] => guid\n [4] => ipauniqueid\n [5] => dn\n [6] => uid\n [7] => samaccountname\n [8] => memberof\n [9] => userquota\n [10] => mail\n [11] => displayname\n [12] => jpegphoto\n [13] => thumbnailphoto\n)\n limit 500 offset 0","userAgent":"DAVx5/2.6.5-ose (2020/03/10; dav4jvm; okhttp/3.12.10) Android/8.1.0","version":"18.0.2.2","id":"5ead4e916c4fd"}

Thatoo · 2020-05-02T10:46:12Z

$ tail -F nextcloud log
tail: impossible d'ouvrir 'nextcloud' en lecture: Aucun fichier ou dossier de ce type
tail: impossible d'ouvrir 'log' en lecture: Aucun fichier ou dossier de ce type

sudo doesn't help.

Thatoo · 2020-05-02T10:48:43Z

Should I stop name_of_user 's Davx5 sync?

maniackcrudelis · 2020-05-02T10:49:06Z

Huh yep sorry I mean the log of nextcloud ;)
/home/yunohost.app/nextcloud/data/nextcloud.log

But, do your 5 users sync their calendars on nextcloud ?

Thatoo · 2020-05-02T10:52:59Z

yes they do and it has never been a problem in the last 2 years...

In all the log I checked, it was always the same user mention. Could it come from his android phone? his Davx5? Should I remake the sync from start on his phone?

Thatoo · 2020-05-02T10:56:00Z

Same with this log, USER_1 is the same user all the time and the same I mentioned earlier. It is like something is wrong for him

:~$ sudo tail /home/yunohost.app/nextcloud/data/nextcloud.log
{"reqId":"WMfQraAPzVTt1LspCtp6","level":0,"time":"2020-05-02T10:53:29+00:00","remoteAddr":"2.10.1.1","user":"--","app":"user_ldap","method":"PROPFIND","url":"/nuage-nextcloud/remote.php/dav/calendars/USER_1/contact_birthdays/","message":"initializing paged search for  Filter (&(|(objectclass=posixAccount))(uid=USER_1)) base Array\n(\n    [0] => ou=users,dc=yunohost,dc=org\n)\n attr Array\n(\n    [0] => entryuuid\n    [1] => nsuniqueid\n    [2] => objectguid\n    [3] => guid\n    [4] => ipauniqueid\n    [5] => dn\n    [6] => uid\n    [7] => samaccountname\n    [8] => memberof\n    [9] => userquota\n    [10] => mail\n    [11] => displayname\n    [12] => jpegphoto\n    [13] => thumbnailphoto\n)\n limit 500 offset 0","userAgent":"DAVx5/2.6.5-ose (2020/03/10; dav4jvm; okhttp/3.12.10) Android/8.1.0","version":"18.0.2.2"}
{"reqId":"WMfQraAPzVTt1LspCtp6","level":0,"time":"2020-05-02T10:53:29+00:00","remoteAddr":"2.10.1.1","user":"--","app":"user_ldap","method":"PROPFIND","url":"/nuage-nextcloud/remote.php/dav/calendars/USER_1/contact_birthdays/","message":"Ready for a paged search","userAgent":"DAVx5/2.6.5-ose (2020/03/10; dav4jvm; okhttp/3.12.10) Android/8.1.0","version":"18.0.2.2"}
{"reqId":"3hGA2wf7WWN8RTi7ZkG7","level":0,"time":"2020-05-02T10:53:29+00:00","remoteAddr":"2.10.1.1","user":"--","app":"user_ldap","method":"PROPFIND","url":"/nuage-nextcloud/remote.php/dav/calendars/USER_1/contact_birthdays/","message":"initializing paged search for  Filter (&(|(objectclass=posixAccount))(uid=USER_1)) base Array\n(\n    [0] => ou=users,dc=yunohost,dc=org\n)\n attr Array\n(\n    [0] => entryuuid\n    [1] => nsuniqueid\n    [2] => objectguid\n    [3] => guid\n    [4] => ipauniqueid\n    [5] => dn\n    [6] => uid\n    [7] => samaccountname\n    [8] => memberof\n    [9] => userquota\n    [10] => mail\n    [11] => displayname\n    [12] => jpegphoto\n    [13] => thumbnailphoto\n)\n limit 500 offset 0","userAgent":"DAVx5/2.6.5-ose (2020/03/10; dav4jvm; okhttp/3.12.10) Android/8.1.0","version":"18.0.2.2"}
{"reqId":"3hGA2wf7WWN8RTi7ZkG7","level":0,"time":"2020-05-02T10:53:29+00:00","remoteAddr":"2.10.1.1","user":"--","app":"user_ldap","method":"PROPFIND","url":"/nuage-nextcloud/remote.php/dav/calendars/USER_1/contact_birthdays/","message":"Ready for a paged search","userAgent":"DAVx5/2.6.5-ose (2020/03/10; dav4jvm; okhttp/3.12.10) Android/8.1.0","version":"18.0.2.2"}
{"reqId":"bZ5oBGAV5PrDiflK8zSx","level":0,"time":"2020-05-02T10:53:31+00:00","remoteAddr":"2.10.1.1","user":"--","app":"no app in context","method":"PROPFIND","url":"/nuage-nextcloud/remote.php/dav/calendars/USER_1/contact_birthdays/","message":"Deprecated event type for OCA\\User_LDAP\\User\\User::postLDAPBackendAdded: null","userAgent":"DAVx5/2.6.5-ose (2020/03/10; dav4jvm; okhttp/3.12.10) Android/8.1.0","version":"18.0.2.2"}
{"reqId":"JE8alFIdBoDY5tV79Pyl","level":0,"time":"2020-05-02T10:53:31+00:00","remoteAddr":"89.234.177.91","user":"administrateur","app":"no app in context","method":"GET","url":"/nuage-nextcloud/apps/logreader/poll?lastReqId=uppYcGyJPTrudqg9wSmO","message":"Deprecated event type for OCA\\User_LDAP\\User\\User::postLDAPBackendAdded: null","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0","version":"18.0.2.2"}
{"reqId":"VCTcFAbw5154oo3jx8BS","level":0,"time":"2020-05-02T10:53:32+00:00","remoteAddr":"2.10.1.1","user":"--","app":"no app in context","method":"PROPFIND","url":"/nuage-nextcloud/remote.php/dav/calendars/USER_1/contact_birthdays/","message":"Deprecated event type for OCA\\User_LDAP\\User\\User::postLDAPBackendAdded: null","userAgent":"DAVx5/2.6.5-ose (2020/03/10; dav4jvm; okhttp/3.12.10) Android/8.1.0","version":"18.0.2.2"}
{"reqId":"lZyjDhSX3scWsvJmS8gq","level":0,"time":"2020-05-02T10:53:33+00:00","remoteAddr":"2.10.1.1","user":"--","app":"user_ldap","method":"PROPFIND","url":"/nuage-nextcloud/remote.php/dav/calendars/USER_1/contact_birthdays/","message":"initializing paged search for  Filter (&(|(objectclass=posixAccount))(uid=USER_1)) base Array\n(\n    [0] => ou=users,dc=yunohost,dc=org\n)\n attr Array\n(\n    [0] => entryuuid\n    [1] => nsuniqueid\n    [2] => objectguid\n    [3] => guid\n    [4] => ipauniqueid\n    [5] => dn\n    [6] => uid\n    [7] => samaccountname\n    [8] => memberof\n    [9] => userquota\n    [10] => mail\n    [11] => displayname\n    [12] => jpegphoto\n    [13] => thumbnailphoto\n)\n limit 500 offset 0","userAgent":"DAVx5/2.6.5-ose (2020/03/10; dav4jvm; okhttp/3.12.10) Android/8.1.0","version":"18.0.2.2"}
{"reqId":"lZyjDhSX3scWsvJmS8gq","level":0,"time":"2020-05-02T10:53:33+00:00","remoteAddr":"2.10.1.1","user":"--","app":"user_ldap","method":"PROPFIND","url":"/nuage-nextcloud/remote.php/dav/calendars/USER_1/contact_birthdays/","message":"Ready for a paged search","userAgent":"DAVx5/2.6.5-ose (2020/03/10; dav4jvm; okhttp/3.12.10) Android/8.1.0","version":"18.0.2.2"}
{"reqId":"LzI2yOnUsNdrIVOF3pSP","level":0,"time":"2020-05-02T10:53:33+00:00","remoteAddr":"2.10.1.1","user":"--","app":"no app in context","method":"PROPFIND","url":"/nuage-nextcloud/remote.php/dav/calendars/USER_1/contact_birthdays/","message":"Deprecated event type for OCA\\User_LDAP\\User\\User::postLDAPBackendAdded: null","userAgent":"DAVx5/2.6.5-ose (2020/03/10; dav4jvm; okhttp/3.12.10) Android/8.1.0","version":"18.0.2.2"}
:~$

maniackcrudelis · 2020-05-02T11:00:30Z

Not sure about how caldav protocol works, but I know that it does connect with the credentials.
So I think you could try to apply the fix about argon2

It's quite easy, you just have to edit /var/www/nextcloud/config/config.php and add 'hashing_default_password' => true after the line memcache.locking
A reboot of php7.3-fpm to be sure, and we'll see if it's better.

EDIT:
FUCKING USELESS QUOTES ARE THAT ARE NOT QUOTES !!!
Use that syntax, which is actually completely different... Because the following one use REAL quotes...

'hashing_default_password' => true

Thatoo · 2020-05-02T11:23:34Z

I have updated Davx5 on the phone of USER_1 and

Charge CPU
1 min | 1.09
5 min | 1.98
15 min | 3.98

Let's see if it lasts with time, if not, I'll try the fix about argon2, what do you think?

maniackcrudelis · 2020-05-02T11:37:37Z

I think you're free to do as you prefer 😉

I don't know if that fix will solve anything for you. Now you know how do to it, feel free to try it if you want.
Meanwhile, if things are better, that's for the best.

Thatoo · 2020-05-02T12:21:24Z

Thank you very much @maniackcrudelis and @lapineige for your help. I'm not sure I would have find out that my issue was coming from an android phone without your guidance on where to look.

Thatoo · 2020-05-02T17:52:35Z

Too bad, so sad, problem isn't solved apparently...
I tried to edit /var/www/nextcloud/config/config.php and add
‘hashing_default_password’ => true,
after the line memcache.locking
then I restarted the server but still I have several
php-fpm : pool nextcloud going on with 20% CPU usage....

Thatoo · 2020-05-02T18:05:47Z

I really hope this #274 , updating to 18.0.4 will solve my issue.

Thatoo · 2020-05-02T18:11:31Z

I still have pbm with "hashing_default_password"

{"reqId":"sBGGS2D3sP6F5urKKvhr","level":3,"time":"2020-05-02T18:10:09+00:00","remoteAddr":"89.234.177.91","user":"titouan","app":"PHP","method":"PROPFIND","url":"/nuage-nextcloud/remote.php/dav/calendars/USER_1/personal/","message":"Use of undefined constant ‘hashing_default_password’ - assumed '‘hashing_default_password’' (this will throw an Error in a future version of PHP) at /var/www/nextcloud/config/config.php#29","userAgent":"DAVx5/3.0-ose (2020/04/24; dav4jvm; okhttp/4.5.0) Android/8.1.0","version":"18.0.2.2","id":"5eadb782763f3"}

Thatoo · 2020-05-02T18:16:29Z

My mistake, I wrote
‘hashing_default_password’ => true,
instead of
‘hashing_default_password’ => true
I saw , on all other lines and I thought it was needed...
Could be that?

Thatoo · 2020-05-02T18:24:24Z

I guess it was that and only that because it suddenly drop to a load below 0 and I don't have any new log for more than 10 minutes. Thanks again.

Thatoo · 2020-05-02T18:33:56Z

Nope, sorry for closing/opening but after I remove the "," I didn't get any log or heavy load because nextcloud was down....
I added back the "," so nextcloud is on again but logs are flooding back with errors and debug... and CPU load has gone up to 4 already...
We agree that config.php should look something like


  'instanceid' => 'oc8i76jor9qm',
  'ldapIgnoreNamingRules' => false,
  'ldapProviderFactory' => '\\OCA\\User_LDAP\\LDAPProviderFactory',
  'updatechecker' => false,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'integrity.check.disabled' => true,
  'filelocking.enabled' => true,
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  ‘hashing_default_password’ => true,
  'redis' => 
  array (
    'host' => 'localhost',
    'port' => '6379',
    'timeout' => '0.0',
    'password' => '',
  ),

right?

Thatoo · 2020-05-02T19:05:40Z

So I decided to do a
sudo service php7.3-fpm stop
to get a nice server back.
I guess I'll wait for the nextcloud update to 18.0.4 and work without nextcloud till then.

maniackcrudelis · 2020-05-02T19:09:47Z

@Thatoo, I updated my previous comment. A fucking fake quote sneaked its way into my comment...
Use real simple quote, it will work...

maniackcrudelis · 2020-05-02T19:10:25Z

And sorry to hear that it wasn't actually fixed...

Thatoo · 2020-05-02T19:14:47Z

and no , after true?

maniackcrudelis · 2020-05-02T19:16:07Z

Yep there's a , at the end, like the others line in the file.

Thatoo · 2020-05-02T19:16:37Z

ok. I'll try and let you know.

Thatoo · 2020-05-03T17:45:55Z

Bingo, 24h has passed and no more problem!
Thanks @maniackcrudelis

Thatoo closed this as completed May 2, 2020

Thatoo mentioned this issue May 2, 2020

Connection and federation issue YunoHost-Apps/synapse_ynh#194

Closed

Thatoo reopened this May 2, 2020

Thatoo closed this as completed May 2, 2020

Thatoo reopened this May 2, 2020

Thatoo closed this as completed May 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nextcloud 18.0.2 uses a lot of ressources #300

Nextcloud 18.0.2 uses a lot of ressources #300

Thatoo commented May 2, 2020

lapineige commented May 2, 2020

Thatoo commented May 2, 2020 •

edited

Loading

lapineige commented May 2, 2020

maniackcrudelis commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020

maniackcrudelis commented May 2, 2020

Thatoo commented May 2, 2020 •

edited

Loading

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020

maniackcrudelis commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020 •

edited

Loading

maniackcrudelis commented May 2, 2020 •

edited

Loading

Thatoo commented May 2, 2020

maniackcrudelis commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020 •

edited

Loading

Thatoo commented May 2, 2020

maniackcrudelis commented May 2, 2020

maniackcrudelis commented May 2, 2020

Thatoo commented May 2, 2020

maniackcrudelis commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 3, 2020

Nextcloud 18.0.2 uses a lot of ressources #300

Nextcloud 18.0.2 uses a lot of ressources #300

Comments

Thatoo commented May 2, 2020

lapineige commented May 2, 2020

Thatoo commented May 2, 2020 • edited Loading

lapineige commented May 2, 2020

maniackcrudelis commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020

maniackcrudelis commented May 2, 2020

Thatoo commented May 2, 2020 • edited Loading

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020

maniackcrudelis commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020 • edited Loading

maniackcrudelis commented May 2, 2020 • edited Loading

Thatoo commented May 2, 2020

maniackcrudelis commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 2, 2020 • edited Loading

Thatoo commented May 2, 2020

maniackcrudelis commented May 2, 2020

maniackcrudelis commented May 2, 2020

Thatoo commented May 2, 2020

maniackcrudelis commented May 2, 2020

Thatoo commented May 2, 2020

Thatoo commented May 3, 2020

Thatoo commented May 2, 2020 •

edited

Loading

Thatoo commented May 2, 2020 •

edited

Loading

Thatoo commented May 2, 2020 •

edited

Loading

maniackcrudelis commented May 2, 2020 •

edited

Loading

Thatoo commented May 2, 2020 •

edited

Loading