MediaTracker import no progress?

[phrogg]

Hey, I have set up your docker container, and I'm currently trying to import my MediaTracker stuff. It indicates that the import is running but for days nothing has been imported yet. I tried to ping my MediaTracker instance from ryot as well as restarting both containers. Is there anything where I can see the progress of the import?

Thanks in advance also for RYOT it is great! Is there a way to donate btw?

[IgnisDa]

Can you set RUST_LOG: ryot=trace in the env section of your docker compose and restart the import?

Thanks for the offer! I have a donation link on my profile page.

[phrogg]

You meant RUST_LOG=ryot=trace correct? I already set this but in docker-compose logs -f nothing appears after adding the import.

[IgnisDa]

docker exec -it ryot bash and then cat tmp/ryot will show you the backend logs.

[phrogg]

docker exec -it ryot bash and then cat tmp/ryot will show you the backend logs.

After starting the import and waiting 2 minutes I get this:

2024-01-10T08:23:25.160847Z TRACE ryot::background: Started job: "UpdateExerciseJob"
2024-01-10T08:23:25.359875Z TRACE update_exercise: ryot::fitness::resolver: Created new exercise with id: Zottman Preacher Curl
2024-01-10T08:23:25.359981Z TRACE ryot::background: Job: "UpdateExerciseJob", Time Taken: 251ms, Successful = true
2024-01-10T08:23:25.732099Z TRACE update_exercise: ryot::fitness::resolver: Created new exercise with id: Wrist Roller
2024-01-10T08:23:25.732223Z TRACE ryot::background: Job: "UpdateExerciseJob", Time Taken: 571ms, Successful = true
2024-01-10T08:23:25.732763Z TRACE update_exercise: ryot::fitness::resolver: Created new exercise with id: Wrist Rotations with Straight Bar
2024-01-10T08:23:25.732859Z TRACE ryot::background: Job: "UpdateExerciseJob", Time Taken: 624ms, Successful = true
2024-01-10T09:00:00.020426Z TRACE ryot::background: Getting data from yanked integrations for all users
2024-01-10T09:13:30.033690Z TRACE ryot::background: Started job: "ImportFromExternalSource"
2024-01-10T09:13:30.284910Z TRACE import_media{user_id=1}: ryot::importer: Started import job with id = 1

Further waiting does still not show up additional logs.

[IgnisDa]

Do you have a lot of data on your media tracker instance? Can you try curl https://yourmediatracker.app/api/items --header 'Access-Token: <here>' and see how big the response is?

I tried with the public media tracker (https://mediatracker.app) instance (which has ~630 items) and it works fine.

[phrogg]

Do you have a lot of data on your media tracker instance? Can you try curl https://yourmediatracker.app/api/items --header 'Access-Token: <here>' and see how big the response is?

I tried with the public media tracker (https://mediatracker.app) instance (which has ~630 items) and it works fine.

I have quite a lot there, but I doubt that it is much more than from the public instance. In fact, I also have about 600 items in my instance. I get a response that looks good from manually curling the address. The response size is: 1014.52 KB

Thanks for your assistance so far!

[IgnisDa]

Then it's probably a network issue. Can you curl the same endpoint from the Ryot container and show me the screenshot?

[phrogg]

Then it's probably a network issue. Can you curl the same endpoint from the Ryot container and show me the screenshot?

Seems to work same reply (part of it):

[IgnisDa]

That's very weird. I already have sufficient logging setup in the media tracker importer.

The only other possible solution I have is that you dump your media tracker instance and I debug the issue locally.

If you're okay with that, you can mail it to me (email id in my gh profile).

[phrogg]

That's very weird. I already have sufficient logging setup in the media tracker importer.

The only other possible solution I have is that you dump your media tracker instance and I debug the issue locally.

If you're okay with that, you can mail it to me (email id in my gh profile).

If your up for that that sounds good. Do you want the sqlite database directly or what dump?

[IgnisDa]

Sqlite sounds fine. Also I'll need the login credentials for the account you're trying to import.

[phrogg]

Sqlite sounds fine. Also I'll need the login credentials for the account you're trying to import.

Wilco, send.

[IgnisDa]

@phrogg I have found the issue, the fix will be deployed in the next release (which is waiting for remix-run/remix#8309 to be released).

Though I still find it weird that the error was not visible in the logs.

[phrogg]

@phrogg I have found the issue, the fix will be deployed in the next release (which is waiting for remix-run/remix#8309 to be released).

Though I still find it weird that the error was not visible in the logs.

Thanks a lot!

[IgnisDa]

Please upgrade to v4.1.0 and let me know if the import works fine. And thanks for the donation!

[phrogg]

Yes I can confirm it works now. But I don't know if this happens only to me. But when I setup the container fresh the first start always fails with:

ryot-db  | The files belonging to this database system will be owned by user "postgres".
ryot-db  | This user must also own the server process.
ryot-db  | 
ryot-db  | The database cluster will be initialized with locale "en_US.utf8".
ryot-db  | The default database encoding has accordingly been set to "UTF8".
ryot-db  | The default text search configuration will be set to "english".
ryot-db  | 
ryot-db  | Data page checksums are disabled.
ryot-db  | 
ryot-db  | fixing permissions on existing directory /var/lib/postgresql/data ... ok
ryot-db  | creating subdirectories ... ok
ryot-db  | selecting dynamic shared memory implementation ... posix
ryot-db  | selecting default max_connections ... 100
ryot-db  | selecting default shared_buffers ... 128MB
ryot-db  | selecting default time zone ... UTC
ryot-db  | creating configuration files ... ok
ryot     | [backend] 2024-01-15T09:42:48.061669Z  INFO ryot: Running version: 4.1.3
ryot-db  | running bootstrap script ... ok
ryot     | [proxy] {"level":"info","ts":1705311768.3350623,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
ryot     | [proxy] {"level":"warn","ts":1705311768.3389487,"logger":"admin","msg":"admin endpoint disabled"}
ryot     | [proxy] {"level":"warn","ts":1705311768.3407485,"logger":"http.auto_https","msg":"automatic HTTPS is completely disabled for server","server_name":"srv0"}
ryot     | [proxy] {"level":"info","ts":1705311768.3415296,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000344f80"}
ryot     | [proxy] {"level":"info","ts":1705311768.3446884,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/home/ryot/.local/share/caddy"}
ryot     | [proxy] {"level":"info","ts":1705311768.3448226,"logger":"tls","msg":"finished cleaning storage units"}
ryot     | [proxy] {"level":"info","ts":1705311768.359072,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
ryot     | [proxy] {"level":"info","ts":1705311768.361297,"msg":"autosaved config (load with --resume flag)","file":"/home/ryot/.config/caddy/autosave.json"}
ryot     | [proxy] {"level":"info","ts":1705311768.36134,"msg":"serving initial configuration"}
ryot-db  | sh: locale: not found
ryot-db  | 2024-01-15 09:42:49.000 UTC [31] WARNING:  no usable system locales were found
ryot-db  | performing post-bootstrap initialization ... ok
ryot     | [frontend] [remix-serve] http://localhost:3000 (http://192.168.208.2:3000)
ryot     | [backend] thread 'main' panicked at apps/backend/src/main.rs:128:10:
ryot     | [backend] Database connection failed: Conn(SqlxError(PoolTimedOut))
ryot     | [backend] note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
ryot     | [backend] ryot exited with code 101
ryot     | --> Sending SIGTERM to other processes..
ryot     | node:events:492
ryot     |       throw er; // Unhandled 'error' event
ryot     |       ^
ryot     | 
ryot     | Error: spawn ps ENOENT
ryot     |     at ChildProcess._handle.onexit (node:internal/child_process:286:19)
ryot     |     at onErrorNT (node:internal/child_process:484:16)
ryot     |     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
ryot     | Emitted 'error' event on ChildProcess instance at:
ryot     |     at ChildProcess._handle.onexit (node:internal/child_process:292:12)
ryot     |     at onErrorNT (node:internal/child_process:484:16)
ryot     |     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
ryot     |   errno: -2,
ryot     |   code: 'ENOENT',
ryot     |   syscall: 'spawn ps',
ryot     |   path: 'ps',
ryot     |   spawnargs: [ '-o', 'pid', '--no-headers', '--ppid', 17 ]
ryot     | }
ryot     | 
ryot     | Node.js v20.10.0
ryot exited with code 1
ryot-db  | syncing data to disk ... ok
ryot-db  | 
ryot-db  | 
ryot-db  | Success. You can now start the database server using:
ryot-db  | 
ryot-db  |     pg_ctl -D /var/lib/postgresql/data -l logfile start
ryot-db  | 
ryot-db  | initdb: warning: enabling "trust" authentication for local connections
ryot-db  | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
ryot-db  | waiting for server to start....2024-01-15 09:43:07.844 UTC [37] LOG:  starting PostgreSQL 16.1 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20231014) 13.2.1 20231014, 64-bit
ryot-db  | 2024-01-15 09:43:07.850 UTC [37] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
ryot-db  | .2024-01-15 09:43:08.565 UTC [40] LOG:  database system was shut down at 2024-01-15 09:42:50 UTC
ryot-db  | 2024-01-15 09:43:08.781 UTC [37] LOG:  database system is ready to accept connections
ryot-db  |  done
ryot-db  | server started
ryot-db  | 
ryot-db  | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
ryot-db  | 
ryot-db  | waiting for server to shut down...2024-01-15 09:43:09.062 UTC [37] LOG:  received fast shutdown request
ryot-db  | .2024-01-15 09:43:09.352 UTC [37] LOG:  aborting any active transactions
ryot-db  | 2024-01-15 09:43:09.363 UTC [37] LOG:  background worker "logical replication launcher" (PID 43) exited with exit code 1
ryot-db  | 2024-01-15 09:43:09.364 UTC [38] LOG:  shutting down
ryot-db  | 2024-01-15 09:43:09.656 UTC [38] LOG:  checkpoint starting: shutdown immediate
ryot-db  | ..2024-01-15 09:43:11.566 UTC [38] LOG:  checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.002 s, sync=0.679 s, total=2.202 s; sync files=2, longest=0.433 s, average=0.340 s; distance=0 kB, estimate=0 kB; lsn=0/14F0498, redo lsn=0/14F0498
ryot-db  | 2024-01-15 09:43:11.590 UTC [37] LOG:  database system is shut down
ryot-db  |  done
ryot-db  | server stopped
ryot-db  | 
ryot-db  | PostgreSQL init process complete; ready for start up.
ryot-db  | 
ryot-db  | 2024-01-15 09:43:12.273 UTC [1] LOG:  starting PostgreSQL 16.1 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20231014) 13.2.1 20231014, 64-bit
ryot-db  | 2024-01-15 09:43:12.273 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
ryot-db  | 2024-01-15 09:43:12.273 UTC [1] LOG:  listening on IPv6 address "::", port 5432
ryot-db  | 2024-01-15 09:43:12.480 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
ryot-db  | 2024-01-15 09:43:12.823 UTC [51] LOG:  database system was shut down at 2024-01-15 09:43:11 UTC
ryot-db  | 2024-01-15 09:43:13.026 UTC [1] LOG:  database system is ready to accept connections

After then rebooting the container everything works.

[IgnisDa]

Yep this is probably because the database has not finished initializing by the time Ryot attempts the DB connection. I don't think there is a need to fix it since its a one-time issue.

[phrogg]

Yep this is probably because the database has not finished initializing by the time Ryot attempts the DB connection. I don't think there is a need to fix it since its a one-time issue.

Probably not, but since this happens every time you fresh installation. It's maybe something that new users wouldn't want.

[IgnisDa]

Alright, I will keep that in mind.