NIX_CFLAGS_COMPILE and NIX_LDFLAGS heavy flag multiplication (up to x5 clones per flag) due to setup-hooks.

[liarokapisv]

It appears that cc-wrapper packages at different *buildInputs in the dependency hierarchy actually multiply the header and library paths added to NIX_CFLAGS_COMPILE and NIX_LDFLAGS.
We can get up to x5 copies for each, leading to #41340 in larger projects.

Minimal repro:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/23.05";
  };

  outputs = { self, nixpkgs }: {
    packages.x86_64-linux =
      let
        pkgs = nixpkgs.legacyPackages.x86_64-linux;
      in
      {
        default =
          (with pkgs; let
            dep1 =
              runCommand "dep1"
                {
                  buildInputs = [ gcc ];
                  nativeBuildInputs = [ gcc ];
                } "";
            dep2 =
              runCommand "dep2"
                {
                  buildInputs = [ libpng ];
                } "";
          in
          pkgs.mkShell
            {
              inputsFrom = [
                dep1
                dep2
              ];
              shellHook = ''
                NIX_DEBUG=1 g++ <(echo "int main() { return 0; }") 2>&1 | grep /nix/store
              '';
            });
      };
  };
}

Entering the shell outputs:

extra flags before to /nix/store/dcd1zhv56rk0d2z7akzfjgzr076c4jl9-gcc-12.2.0/bin/g++:
  -Wl\,-dynamic-linker=/nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/ld-linux-x86-64.so.2
original flags to /nix/store/dcd1zhv56rk0d2z7akzfjgzr076c4jl9-gcc-12.2.0/bin/g++:
extra flags after to /nix/store/dcd1zhv56rk0d2z7akzfjgzr076c4jl9-gcc-12.2.0/bin/g++:
  -B/nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/
  /nix/store/rfw51dqr3qn7b6fjy8hmx6f0x3hfwbx6-glibc-2.37-8-dev/include
  /nix/store/dcd1zhv56rk0d2z7akzfjgzr076c4jl9-gcc-12.2.0/lib/gcc/x86_64-unknown-linux-gnu/12.2.0/include-fixed
  -B/nix/store/yazs3bdl481s2kyffgsa825ihy1adn8f-gcc-12.2.0-lib/lib
  -B/nix/store/7wkshj58fcsl1f3zyi67qsxgl1p8nki1-gcc-wrapper-12.2.0/bin/
  /nix/store/bw6xcpqzfw3n7zfz7g0cpqgx2y2lac9j-libpng-apng-1.6.39-dev/include
  /nix/store/bw6xcpqzfw3n7zfz7g0cpqgx2y2lac9j-libpng-apng-1.6.39-dev/include
  /nix/store/qwmhvny4in8134s96ssfy92w5acbwc4c-zlib-1.2.13-dev/include
  /nix/store/qwmhvny4in8134s96ssfy92w5acbwc4c-zlib-1.2.13-dev/include
  /nix/store/bw6xcpqzfw3n7zfz7g0cpqgx2y2lac9j-libpng-apng-1.6.39-dev/include
  /nix/store/bw6xcpqzfw3n7zfz7g0cpqgx2y2lac9j-libpng-apng-1.6.39-dev/include
  /nix/store/qwmhvny4in8134s96ssfy92w5acbwc4c-zlib-1.2.13-dev/include
  /nix/store/qwmhvny4in8134s96ssfy92w5acbwc4c-zlib-1.2.13-dev/include
  /nix/store/bw6xcpqzfw3n7zfz7g0cpqgx2y2lac9j-libpng-apng-1.6.39-dev/include
  /nix/store/qwmhvny4in8134s96ssfy92w5acbwc4c-zlib-1.2.13-dev/include
  -L/nix/store/hr3m53r0nhyqx80sg0bz9xjgk6jg009k-zlib-1.2.13/lib
  -L/nix/store/yszpxj5q3j8lfxw3df0q0k2c4zmfgyg1-libpng-apng-1.6.39/lib
  -L/nix/store/hr3m53r0nhyqx80sg0bz9xjgk6jg009k-zlib-1.2.13/lib
  -L/nix/store/yszpxj5q3j8lfxw3df0q0k2c4zmfgyg1-libpng-apng-1.6.39/lib
  -L/nix/store/hr3m53r0nhyqx80sg0bz9xjgk6jg009k-zlib-1.2.13/lib
  -L/nix/store/yszpxj5q3j8lfxw3df0q0k2c4zmfgyg1-libpng-apng-1.6.39/lib
  -L/nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib
  -L/nix/store/dcd1zhv56rk0d2z7akzfjgzr076c4jl9-gcc-12.2.0/lib/gcc/x86_64-unknown-linux-gnu/12.2.0
  -L/nix/store/yazs3bdl481s2kyffgsa825ihy1adn8f-gcc-12.2.0-lib/x86_64-unknown-linux-gnu/lib
  -L/nix/store/yazs3bdl481s2kyffgsa825ihy1adn8f-gcc-12.2.0-lib/lib
extra flags before to /nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld:
original flags to /nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld:
  /nix/store/dcd1zhv56rk0d2z7akzfjgzr076c4jl9-gcc-12.2.0/libexec/gcc/x86_64-unknown-linux-gnu/12.2.0/liblto_plugin.so
  -plugin-opt=/nix/store/dcd1zhv56rk0d2z7akzfjgzr076c4jl9-gcc-12.2.0/libexec/gcc/x86_64-unknown-linux-gnu/12.2.0/lto-wrapper
  /nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib64/ld-linux-x86-64.so.2
  /nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/crt1.o
  /nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/crti.o
  /nix/store/dcd1zhv56rk0d2z7akzfjgzr076c4jl9-gcc-12.2.0/lib/gcc/x86_64-unknown-linux-gnu/12.2.0/crtbegin.o
  -L/nix/store/hr3m53r0nhyqx80sg0bz9xjgk6jg009k-zlib-1.2.13/lib
  -L/nix/store/yszpxj5q3j8lfxw3df0q0k2c4zmfgyg1-libpng-apng-1.6.39/lib
  -L/nix/store/hr3m53r0nhyqx80sg0bz9xjgk6jg009k-zlib-1.2.13/lib
  -L/nix/store/yszpxj5q3j8lfxw3df0q0k2c4zmfgyg1-libpng-apng-1.6.39/lib
  -L/nix/store/hr3m53r0nhyqx80sg0bz9xjgk6jg009k-zlib-1.2.13/lib
  -L/nix/store/yszpxj5q3j8lfxw3df0q0k2c4zmfgyg1-libpng-apng-1.6.39/lib
  -L/nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib
  -L/nix/store/dcd1zhv56rk0d2z7akzfjgzr076c4jl9-gcc-12.2.0/lib/gcc/x86_64-unknown-linux-gnu/12.2.0
  -L/nix/store/yazs3bdl481s2kyffgsa825ihy1adn8f-gcc-12.2.0-lib/x86_64-unknown-linux-gnu/lib
  -L/nix/store/yazs3bdl481s2kyffgsa825ihy1adn8f-gcc-12.2.0-lib/lib
  -L/nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib
  -L/nix/store/yazs3bdl481s2kyffgsa825ihy1adn8f-gcc-12.2.0-lib/lib
  -L/nix/store/7wkshj58fcsl1f3zyi67qsxgl1p8nki1-gcc-wrapper-12.2.0/bin
  -L/nix/store/dcd1zhv56rk0d2z7akzfjgzr076c4jl9-gcc-12.2.0/lib/gcc/x86_64-unknown-linux-gnu/12.2.0
  -L/nix/store/dcd1zhv56rk0d2z7akzfjgzr076c4jl9-gcc-12.2.0/lib/gcc/x86_64-unknown-linux-gnu/12.2.0/../../../../lib64
  -L/nix/store/dcd1zhv56rk0d2z7akzfjgzr076c4jl9-gcc-12.2.0/lib/gcc/x86_64-unknown-linux-gnu/12.2.0/../../..
  -dynamic-linker=/nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/ld-linux-x86-64.so.2
  /nix/store/dcd1zhv56rk0d2z7akzfjgzr076c4jl9-gcc-12.2.0/lib/gcc/x86_64-unknown-linux-gnu/12.2.0/crtend.o
  /nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/crtn.o
extra flags after to /nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld:
  /nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib
  /nix/store/yazs3bdl481s2kyffgsa825ihy1adn8f-gcc-12.2.0-lib/lib
/nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld: /dev/fd/63: file not recognized: Illegal seek

[liarokapisv]

@Ericson2314 I have been working on a PR for the past few days, main idea is that at the point where the setup-hooks append flags, we become aware of the newly appended flags as an atom. At this point we can search the relevant var for matches and avoid appending if they are already present. Doing this post-hook is not possible because it's not easy to distinguish single flags vs flag sets (eg -isystem may be recognized as a single flag).
One place where this is especially annoying is when mangling all post_role variables. This uses the appended string env vars so distinguishing between flags is not possible, we would need to maintain and use corresponding associated array helper-vars for each in order for this to be done properly.

All of this is a bit tricky to get right especially if we want to abstract this away using dedicated helpers because bash is a terrible medium to maintaing a proper c/c++ build model through env variables, many flag sets contain spaces/quotes and dealing with all of the bash string rules is tricky.

Do you have any suggestions on how to best approach this issue? I can also post my PR and get some discussion going over there. I am at a point where I get a huge reduction of duplicate flags but when building huge overlays I get some issues with some packages, probably something wrong with my append/prepend helpers.

Maybe another approach like taking advantage of gcc itself to deduplicate the flags would be better.