Symlinks are not found when loading configuration files

[ElenaKhaustova]

Description

When having parameters in conf/base/sub_folder/parameters_A.yml structure where sub_folder is symlinked, the pipeline gives the following error, however, it is able to read the parameters from the following structure: conf/base/parameters_A.yml

ValueError: Pipeline input(s) {'params:length', 'params:width'} not found in the DataCatalog

Context

The issue is that fsspec.filesystem.glob() which we use to find the paths recursively doesn’t find symlinks.

kedro/kedro/config/omegaconf_config.py

Line 295 in adfc593

for each in self._fs.glob(Path(f"{str(conf_path)}/{pattern}").as_posix()):

Steps to Reproduce

For a default spaceflights-pandas project create a symlink folder and place parameters_data_science.yml in the linked folder.

Run the pipeline.

Expected Result

Symlinks are found when loading configuration files.

Your Environment

• Kedro version used (pip show kedro or kedro -V): kedro, version 0.19.6
• Python version used (python -V): Python 3.11.9
• Operating system and version: macOS Sonoma version 14.5

[ElenaKhaustova]

Additional context from the user side:

• they use symlinks in their assembler system on CX;
• inability to use symlinks blocks them from migrating to the latest kedro version and or away from kedro glass.

[noklam]

Python doc

If recursive is true, the pattern “**” will match any files and zero or more directories, subdirectories and symbolic links to directories. If the pattern is followed by an os.sep or os.altsep then files will not match.

[noklam]

I created a PoC fix that seems to work for me: https://github.com/kedro-org/kedro/compare/noklam/glob-tmp-fix?expand=1