[BUG] fd -X don't process all files found

[kkew3]

Checks

• I have read the troubleshooting section and still think this is a bug.

Describe the bug you encountered:

fd -X doesn't process all found files when the result list is very long; long filenames makes the issue worse. For example, when there are tens of thousands of txt files in a directory and I run fd -etxt -X du -sh, the total size added up will be smaller than the ground truth.

I'm not sure, but this may be related to #410 and #441.

Describe what you expected to happen:

As this section, I expect -X function equivalently with xargs and/or parallel, that is, to process all files even if processing them all at once exceeds the command line length limit.

What version of fd are you using?

fd 8.6.0

Which operating system / distribution are you on?

Darwin 21.6.0 x86_64

zsh: command not found: lsb_release

[tavianator]

Are you sure that fd without -X is finding all the relevant files? Did you try with -u to make sure hidden and gitignored files are also included?

[tmccombs]

Is your list of files long enough it is exceeding the maximum command line size? If so there isn't much we can do about it. For du specifically it would probably work better to pipe the output of fd, with the -0 flag to du with the --files0-from flag (reading from stdin).

[kkew3]

Thank you so much for your advice. I'll close the issue if it's not possible to solve. Thanks again.

[sharkdp]

No, wait. We spent a lot of time on this actually (as described in #410). This should work since fd 8.4 (see https://github.com/sharkdp/fd/releases/tag/v8.4.0 => bugfixes). If the argument list is too long, we will launch the process again with another batch of arguments.

@kkew3 Did you try the suggestion by @tavianator?

[tmccombs]

Yes, if it is exceeding the max length it should result in multiple invocations. In the case of du, that would result in multiple summaries. So if you are using du and want a single summary, you'll probably want to pipe to du instead. And in that case, xarga would have a similar problem. Did you get multiple summary lines @kkew3?

[kkew3]

Sorry, it's my fault. Surly it produces multiple summary lines. I did tail -n1 to get the expected summary line before. Thank you so much for your kind explanation.