Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[jail profile] pkg: Repository Synth has a wrong packagesite #237

Open
nuclight opened this issue Feb 16, 2025 · 25 comments
Open

[jail profile] pkg: Repository Synth has a wrong packagesite #237

nuclight opened this issue Feb 16, 2025 · 25 comments

Comments

@nuclight
Copy link

Created a profile for a fresh clean jail, switched to it and did synth build www/firefox. At the end (non-first run, Ctrl-Q used on earlier incarnation) synth asked about upgrading system (the jail specified as root as I expect) with built pkgs, but that failed. Not critical as can be done manually, but still...

The task is complete.  Final tally:
Initial queue size: 1
    packages built: 1
           ignored: 0
           skipped: 0
            failed: 0

Duration: 02:56:45
The build logs can be found at: /zp/bsd/repo1/synth/log
Would you like to rebuild the local repository (Y/N)? y
Stand by, recursively scanning 1 port serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Would you like to upgrade your system with the new packages now (Y/N)? y
Updating Synth repository catalogue...
pkg: Repository Synth has a wrong packagesite, need to re-create database
Fetching meta.conf: 100%    178 B   0.2kB/s    00:01    
Fetching data.pkg: 100%   81 KiB  82.8kB/s    00:01    
Processing entries: 100%
Synth repository update completed. 304 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The most recent versions of packages are already installed

synth build www/firefox  29751.41s user 1049.80s system 283% cpu 3:01:20.79 total
@jrmarino
Copy link
Owner

"Repository Synth has a wrong packagesite, need to re-create database" is a message from pkg, not synth.
Presumably it's configured to prefetch.

I'm not an expert on pkg but presumably this is all normal based on your specific setup

@nuclight
Copy link
Author

Yeah, but isn't it configured by Synth? There is no /usr/local/etc/pkg/repos/00_synth.conf inside jail, what did I miss?

@jrmarino
Copy link
Owner

does it happen every time or just that one time?

@nuclight
Copy link
Author

I switched profile back to LiveSystem for trying to build older xterm (has some locale bug in vim) and got same message after answering Y to question - so probably that's "every time"

@nuclight
Copy link
Author

Porters in our Telegram supposed that's new unexpected changes in pkg as a side-effect fro new pkgbase shlib tracking etc.

@jrmarino
Copy link
Owner

you need to repeat in the same profile before you can determine if it's repeatable.
If it's a pkg change, it should only happen once

@nuclight
Copy link
Author

I looked at pkg sources and found in libpkg/repo/binary/init.c:

        /* Check package site */
        char *req = sqlite3_mprintf("select count(key) from repodata "
                "WHERE key = \"packagesite\" and value = '%q'", pkg_repo_url(repo));

        res = 0;
        get_pragma(sqlite, req, &res, true);
        sqlite3_free(req);
        if (res != 1) {
                pkg_emit_error("Repository %s has a wrong packagesite, need to "
                        "re-create database", repo->name);
                sqlite3_close(sqlite);
                return (EPKG_FATAL);
        }

i.e. it means if several records found for packagesite.

So I opened /var/db/pkg/repo-Synth.sqlite manually but found only one row in repodata table, exactly packagesite with path to directory from LiveSystem profile (I switched back to it since). Then I looked into jail's /usr/obj/jclassic/var/db/pkg and discovered it is empty directory.

Now I suspect that what happened is: synth on profile for jail did record to system's /var/db/pkg/ so it got two lines - for two profiles. Am I missing something for profile settings in synth?

BTW I did synth rebuild-repository and it said

Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Note: This system's pkg(8) is configured with CONSERVATIVE_UPGRADE = true
      You may wish to toggle that setting if this is a local repository.

but that's probably OK.

@jrmarino
Copy link
Owner

so the majority of pkg actions occur inside the builder where the environment is specified, including several pkg settings.
However, if I remember correctly, there are some pkg operations that are done with the hosts pkg.
The question is whether something like PKG_DBDIR (the source of "/var/db/pkg") is set in the environment for those operations. I thnk repository generation is one of those host pkg operations.

i don't know - sounds like PKG_DBDIR might need to be overridden for edge usage like yours for those host pkg operations. I need to check the code.

@jrmarino
Copy link
Owner

FYI - I haven't done this yet, crazy week at work

@jrmarino
Copy link
Owner

jrmarino commented Mar 4, 2025

I confirm the host pkg is the one that executes the "repo" command.

@jrmarino
Copy link
Owner

jrmarino commented Mar 4, 2025

I also confirm PKG_DBDIR is not being set in the environment OR the command line. That's probably the issue.

@jrmarino
Copy link
Owner

jrmarino commented Mar 4, 2025

can you post the contentions of /usr/local/etc/synth/synth.ini ? I want to see how you've set up the jail profile.

@nuclight
Copy link
Author

nuclight commented Mar 4, 2025 via email

@jrmarino
Copy link
Owner

jrmarino commented Mar 4, 2025

My theory is that if directory system == "/" do not modify the repo command (use PKG_DBDIR setting that pkg is configured for), otherwise insert this fragment in the host pkg repo command:

-o pkg_dbdir=<value of directory_system>/var/db/pkg

I can create a branch with this change. Could you then build synth to that git hash by modifying synth Makefile and running "make makesum", install it and test it?

@nuclight
Copy link
Author

nuclight commented Mar 4, 2025

Yes, I can build/install/test (though it make take time for another jail) branch. But may be a separate setting for profile needed - can we break here something for existing "profiles" feature users, or it's safe to insert?

Also, for case profile is of thick jail, probably synth's repo should be added to pkg config inside jail, just as synth does it for live system. Though not sure were it would point regarding (typically) nullfs mount points (probably yet another setting).

@jrmarino
Copy link
Owner

jrmarino commented Mar 4, 2025

I don't think you need to create a new jail. You could even build the branch on the LiveSystem profile, install it, and then test it on the Repo1 profile that you are using. It would be a minor change that shouldn't break anything.

@jrmarino
Copy link
Owner

jrmarino commented Mar 4, 2025

hmm, I misread the code. it's only using the host pkg when the signing server is being used, but normal repo generations use the builder's pkg8.

@jrmarino
Copy link
Owner

jrmarino commented Mar 4, 2025

can you run "pkg -vv" and show the output after "Repositories:", e.g.

Repositories:
  FreeBSD: {
    url             : "pkg+http://pkg.FreeBSD.org/FreeBSD:13:amd64/quarterly",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/share/keys/pkg"
  }
  Synth: {
    url             : "file:///usr/home/marino/issue-224/live_packages",
    enabled         : yes,
    priority        : 0
  }

@nuclight
Copy link
Author

nuclight commented Mar 5, 2025

That's what on host currently (was busy with work, didn't build anything yet):

Repositories:
  FreeBSD: { 
    url             : "pkg+http://pkg.FreeBSD.org/FreeBSD:13:amd64/quarterly",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/share/keys/pkg"
  }
  Synth: { 
    url             : "file:///zp/bsd/synth/live_packages",
    enabled         : yes,
    priority        : 0
  }

@nuclight
Copy link
Author

nuclight commented Mar 5, 2025

it's only using the host pkg when the signing server is being used, but normal repo generations use the builder's pkg8.

but the discussed problem is when it tries to update (live) system, so it's some third variant of pkg run?

@jrmarino
Copy link
Owner

jrmarino commented Mar 5, 2025

I'm trying to figure out which package database is being used that could be getting multiple definitions of the packagesite. Obviously your use of prefetch is a factor here. I am struggling to reproduce this in my head.

Can you clarify does this error ONLY happen in profile Repo1, or does it happen in the LiveSystem profile too?

@jrmarino
Copy link
Owner

jrmarino commented Mar 5, 2025

here's something I don't understand. You show in the Repo1 profile that you are building packages here: /zp/bsd/repo1/synth/packages

but your pkg repository configuration is something else: file:///zp/bsd/synth/live_packages

shouldn't it be file:///zp/bsd/repo1/synth/packages ?

is there some kind of move task I don't know about?

@nuclight
Copy link
Author

nuclight commented Mar 5, 2025

Can you clarify does this error ONLY happen in profile Repo1, or does it happen in the LiveSystem profile too?

Currently I can't reproduce on LiveSystem. It was switched to, according to synth.ini timestamp, at 20 Feb and not touched since, so now I did git checkout ... x11/xterm (I did not finish bisecting minor title breakage in it) and synth build x11/xterm finished with:

Duration: 00:00:35
The build logs can be found at: /var/log/synth
Would you like to rebuild the local repository (Y/N)? y
Stand by, recursively scanning 1 port serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Would you like to upgrade your system with the new packages now (Y/N)? y
Updating Synth repository catalogue...
Fetching meta.conf: 100%    178 B   0.2kB/s    00:01    
Fetching data.pkg: 100%  368 KiB 376.7kB/s    00:01    
Processing entries: 100%
Synth repository update completed. 1505 packages processed.
All repositories are up to date.
pkg: vim-gtk2: duplicate dependency listing: xxd
pkg: vim-gtk2: duplicate dependency listing: xxd
pkg: xorg-apps: duplicate dependency listing: xterm
pkg: xorg-apps: duplicate dependency listing: xterm
Checking integrity... done (0 conflicting)
The following 2 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
	xterm: 388_1 -> 389_1 [Synth]

Installed packages to be REINSTALLED:
	boost-libs-1.86.0_1 [Synth]

Number of packages to be upgraded: 1
Number of packages to be reinstalled: 1
[1/2] Upgrading xterm from 388_1 to 389_1...
[1/2] Extracting xterm-389_1: 100%
[2/2] Reinstalling boost-libs-1.86.0_1...
[2/2] Extracting boost-libs-1.86.0_1: 100%
==> Running trigger: desktop-file-utils.ucl
Building cache database of MIME types

Obviously your use of prefetch is a factor here. I am struggling to reproduce this in my head.

I wish some debug of spawned processes would exist, at least pwd, root dir and argv, but ktrace on 13.4 can't do it yet, and I don't have skills in audit or dtrace...

@nuclight
Copy link
Author

nuclight commented Mar 5, 2025

here's something I don't understand. You show in the Repo1 profile that you are building packages here: /zp/bsd/repo1/synth/packages

Yes, that profile was planned for brand new classic jail to put firefox in it.

but your pkg repository configuration is something else: file:///zp/bsd/synth/live_packages

Yes, it is that on host since 20 Feb when I switched profile back to LiveSystem.

shouldn't it be file:///zp/bsd/repo1/synth/packages ?

I was expecting synth to create such pkg configuration entry inside that jail, just like entry was automatically created by synth for LiveSystem on host years ago... but seems that code do not exists yet (or current practices differ, I did not touched jail for dozen years, so this February went simplest manual case for fat jail in Handbook "untar base.txz there and adjust this and that").

is there some kind of move task I don't know about?

Just for completeness, here's tail of history -E -m '*synth*' from my root zsh:

 3005  15.2.2025 02:32  synth prepare-system; echo "\a"
 3006  15.2.2025 04:33  synth configure
 3008  15.2.2025 04:39  mkdir /zp/bsd/repo1/{uports,synth}
 3009  15.2.2025 04:39  mkdir /zp/bsd/repo1/synth/{log,packages}
 3015  15.2.2025 04:47  synth configure
 3016  15.2.2025 04:51  mkdir /usr/obj/synth-repo1
 3017  15.2.2025 04:51  synth configure
 3018  15.2.2025 04:52  synth build www/firefox
 3020  15.2.2025 05:09  synth configure
 3021  15.2.2025 05:09  synth build www/firefox
 3035  16.2.2025 02:27  synth build www/firefox
 3038  16.2.2025 02:43  gdb `which synth` synth.68699.core
 3039  16.2.2025 02:44  rm synth*core
 3040  16.2.2025 03:13  synth build www/firefox
 3042  16.2.2025 06:18  cat /usr/local/etc/synth/synth.ini
 3101* 20.2.2025 01:57  synth build x11/xterm
 3103* 20.2.2025 01:58  synth build x11/xterm
 3110* 20.2.2025 02:13  synth build x11/xterm
 3111* 20.2.2025 02:13  rm /zp/bsd/repo1/synth/packages/All/xterm-391.pkg
 3112* 20.2.2025 02:13  synth build x11/xterm
 3120* 20.2.2025 02:19  synth configure
 3124* 20.2.2025 02:20  synth configure
 3125* 20.2.2025 02:20  synth build x11/xterm
 3129* 20.2.2025 02:32  synth build x11/xterm
 3133* 20.2.2025 02:34  synth build x11/xterm
 3138  20.2.2025 15:14  pkg add /zp/bsd/synth/live_packages/All/vim-gtk2-9.1.0015_2.pkg
 3141  20.2.2025 15:19  pkg add /zp/bsd/synth/live_packages/All/xterm-388_1.pkg
 3147* 23.2.2025 21:15  synth rebuild-repository
 3148* 23.2.2025 22:04  synth install x11/xterm
 3182  6.3.2025 01:25  ll /usr/local/etc/synth/synth.ini
 3183  6.3.2025 01:30  chflags sunlnk /zp/bsd/synth/live_packages/All/xterm-388_1.pkg
 3188* 6.3.2025 01:33  synth build x11/xterm

@jrmarino
Copy link
Owner

jrmarino commented Mar 7, 2025

so when you say yes, you want to upgrade now, this command is run:

/usr/local/sbin/pkg install --yes --repository Synth

which means when you build with profile Repo1, you are not installing with the packages you just built.
Those are located in /zp/bsd/repo1/synth/packages
but the synth repository is defined at /zp/bsd/synth/live_packages

so if you want to install from packages built by Repo1, either

  1. update the synth repository definition to the repo1 location
  2. a) create another repository definition pointing to repo1 location
    b) Always answer "No" when asked to upgrade
    c) manually run "pkg install --yes repository < new repository definition >" yourself

unless I'm missing something, this repository definition is your main issue for upgrading the packages with a local repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants