podman instructions do not work.

[hanwen-flow]

Description

the instructions at https://criu.org/Podman do not work. I tested on Ubuntu 22.04 and Fedora 39. The symptoms are the same,

hanwen@hanwen-flow:~/vc/ci-experiments$ podman run -d --name looper2 busybox /bin/sh -c \
         'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
6efa431f914bf306432284e9b052105153ea6eef1f1b531a79366c7825bd2412
hanwen@hanwen-flow:~/vc/ci-experiments$ podman logs -l
0
1
2
3
4
5
6
7
8
9
10
11
12
hanwen@hanwen-flow:~/vc/ci-experiments$ sudo podman container checkpoint -l
Error: no such container
hanwen@hanwen-flow:~/vc/ci-experiments$ sudo podman container checkpoint looper2
Error: no container with name or ID "looper2" found: no such container
hanwen@hanwen-flow:~/vc/ci-experiments$ podman ps
CONTAINER ID  IMAGE                             COMMAND               CREATED         STATUS             PORTS       NAMES
a96ca5ad012d  docker.io/library/busybox:latest  /bin/sh -c i=0; w...  16 hours ago    Up 16 hours ago                looper
6efa431f914b  docker.io/library/busybox:latest  /bin/sh -c i=0; w...  42 seconds ago  Up 42 seconds ago              looper2
hanwen@hanwen-flow:~/vc/ci-experiments$ podman -v
podman version 3.4.4

Fedora 39 runs podman 5.2.3 but fails in the same way.

[hanwen-flow]

I suspect this is due to

Checkpoints currently work with root containers only. Therefore, you have to run the example container as root.

https://podman.io/docs/checkpoint

this could be more obvious in the criu docs.

[adrianreber]

Why are you running some commands with sudo and some commands without? That is the problem. You have to run everything as root.

[hanwen]

Right. The page only implies this through the '#' prompt. I tried to update the docs to be more explicit it, but I the confirmation email doesn't get through to my gmail account, so I can't edit anything.

The instructions from the podman docs (https://podman.io/docs/checkpoint) seem to work, but the first time I ran them it took 45 seconds for the container to be taken down (while the command was running, 8080 kept responding). I can't reproduce the slowness, and the logs in journalctl doesn't show anything odd. Now, cp/restore takes a second or so.

hanwen@fedora:~$ sudo podman ps 
CONTAINER ID  IMAGE                           COMMAND           CREATED        STATUS        PORTS                 NAMES
1d1c9fbba250  docker.io/library/httpd:latest  httpd-foreground  5 seconds ago  Up 5 seconds  0.0.0.0:8080->80/tcp  funny_keller
hanwen@fedora:~$ sudo podman container checkpoint -l 

^CError: `/usr/bin/crun checkpoint --image-path /var/lib/containers/storage/overlay-containers/1d1c9fbba250f4f826f3a6ce0d1db98ad0ec2a8de31d70e17069a571fa219b25/userdata/checkpoint --work-path /var/lib/containers/storage/overlay-containers/1d1c9fbba250f4f826f3a6ce0d1db98ad0ec2a8de31d70e17069a571fa219b25/userdata 1d1c9fbba250f4f826f3a6ce0d1db98ad0ec2a8de31d70e17069a571fa219b25` failed: signal: interrupt
hanwen@fedora:~$ sudo /usr/bin/time -v podman container checkpoint funny_keller
funny_keller
   Command being timed: "podman container checkpoint funny_keller"
   User time (seconds): 0.22
   System time (seconds): 0.44
   Percent of CPU this job got: 1%
   Elapsed (wall clock) time (h:mm:ss or m:ss): 0:46.97
   Average shared text size (kbytes): 0
   Average unshared data size (kbytes): 0
   Average stack size (kbytes): 0
   Average total size (kbytes): 0
   Maximum resident set size (kbytes): 45284
   Average resident set size (kbytes): 0
   Major (requiring I/O) page faults: 0
   Minor (reclaiming a frame) page faults: 17562
   Voluntary context switches: 3969
   Involuntary context switches: 47
   Swaps: 0
   File system inputs: 0
   File system outputs: 19096
   Socket messages sent: 0
   Socket messages received: 0
   Signals delivered: 0
   Page size (bytes): 4096
   Exit status: 0

[hanwen]

note that the slowness I referred to in #2523 (comment) was on my Fedora 40 personal machine.

[rst0git]

@hanwen-flow Thank you for opening this issue. I have updated the criu docs with your suggestions.

The following option may help with obtaining more detailed information on the performance of CRIU and Podman:

podman container checkpoint --print-stats ...

[hanwen-flow]

👍

[hanwen]

I tried repro'ing the wait in #2523 (comment), by clearing out /var/lib/containers and reinstalling podman, but the problem did not reproduce.

[hanwen-flow]

The slowness looks suspiciously like #2032