Rename doesn't work with files that don't have a .py extension

[muhzii]

Describe the bug
Attempting to rename a variable in a file that doesn't end with a .py extension doesn't seem to do anything, the issue seems to occur only with renaming, where other features such as go to definition, references, etc.. seem to work properly.

To Reproduce
Create an extensionless file with a shebang (so it is is detected as a Python file), then try to rename a variable.

Expected behavior
It should be possible to change variable names in files identified as Python which don't have an extension.

[erictraut]

Which client (editor) are you using?

[muhzii]

I'm using a coc-pyright extension with Vim

[erictraut]

I'm able to repro the problem. It's due to a check that prevents renaming symbols in files that are not part of your current project. I think that logic requires some fixes. We've received a number of related bug reports and requests for rename, and we're tracking them all in the pylance-release project, since this is a language server feature and not a core type checking feature. I'm going to transfer this bug report to the pylance-release project so it is tracked and fixed along with the other rename bugs.

[heejaechang]

@muhzii does that happen even if you open the file as standalone? or only if you open it inside of workspace folder?

[muhzii]

@heejaechang I performed the above steps on a file created inside my home directory, and it works. I think it happens only if the file belongs to a project root, in my case it is a git repo directory.

[heejaechang]

@muhzii thank you for the reply!. can I ask 2 more questions.

1. does the file under project root use something from the project root? (such as venv), in another word, must it exist under the project root to run properly?

2. if you put the file in "include" (https://github.com/microsoft/pyright/blob/main/docs/configuration.md), does it work?

I am asking to determine whether 1. there is a workaround 2. fix to put the file under standalone mode is enough.

[muhzii]

1. It's not actually a Python project, just scripts that exist under a project root, so there is no venv etc.. just uses the system's interpreter/ libs.
2. Tried that but it doesn't seem to work.

As per the issue encountered on my end, it would seem that putting the file in "standalone mode" would suffice, but there might also be use-cases where such files have dependencies from the project itself, in which case I think suggestion no. 2 would not work then?

[heejaechang]

@muhzii 2 might not be working due to the file extension since we have some code path that checks file extension to be py and pyi. for now, my thinking is if rename is happened on an open file for local symbol, allow renaming.

[muhzii]

@heejaechang So, basically, with such allowance in place, renames on symbols that are shared between more than one file in the same project (where the rename action is performed from the extensionless file) will just not work?

Hm, sorry if I ask, but I don't understand how it works right now, can't such check simply be extended so it is wired with shebang detection? i.e. the same way the other features do it.

[heejaechang]

@muhzii so, figuring out what files depend on other files requires building dependency graph which require parsing/binding and etc which are expensive. that is why there are scoping to limit how much files we process such as checking file extensions or project root.

if we just allow any files under project root, then we need to read every files under the root and try to parse for any rename. so it will be making 99% case slower to make 1% work.

can't such check simply be extended so it is wired with shebang detection? i.e. the same way the other features do it.

I am not sure what you mean. can you provide a bit more explanation?

thank you.

[erictraut]

@heejaechang, the problem here is that the file in question is not considered part of the project. IMO, we shouldn't prevent rename in such files. We should simply do a local (single-file) rename in this case since we don't know what other files might be associated with it. Currently, the logic refuses to do perform even a local (single-file) rename if the file isn't part of the project. I suspect this is the root of most of the rename bugs that have been filed — including those related to notebooks.

[heejaechang]

@erictraut ya, that is the fix I am making. @muhzii is asking what about dependent files.

[muhzii]

if we just allow any files under project root, then we need to read every files under the root and try to parse for any rename.

I'm not suggesting to blanket allow all the files in a project, although e.g. if a file starts with a #!/usr/bin/env python3 then it should probably behave just the same as other files that have a .py extension, and as such allow it to do project-wide renames which take effect in the relevant files in a project etc..

My suggestion, basically, was to extend the allowance for the renaming action beyond that for files with .py/ .pyi extensions to also detect Python shebangs.. I think such thing would be sensible to do seeing as how other features like resolving definitions and symbol usages already work across different files currently regardless of the extension limitation.

[erictraut]

As @heejaechang mentioned, that would be incredibly expensive because we'd need to open and read every file to look for the shebangs. Your use case (where you're using Python files without ".py" suffixes) is very atypical. We wouldn't want to impose such a large overhead on the millions of other pyright and pylance users to accommodate this very unusual case. Out of curiosity, why do your python source files not end in ".py"?

[muhzii]

@erictraut Totally agree! Looking for symbols in other extensionless files to rename is certainly not desirable (sorry for not clarifying on this). However, It should probably allow renames in other .py files in the project. Specifically, it would be desirable to have this shebang-based thingy work on the currently open file for renames that is.

Out of curiosity, why do your python source files not end in ".py"?

I only use it for convenient scripts, which could also wrap other functionalities provided by the project etc.

[heejaechang]

@muhzii are you saying

1. if one is importing symbols from "*.py" file in the project root (ex, from foo import bar) in shebang-based file under the same project root.
2. and rename "bar" from file "foo"
3. we should check the shebang-based file (if it is opened in the editor) whether it is using "bar" from the "foo" and do rename?

I think that scenario, we should open a new issue and gather some feedbacks before adding support.

[erictraut]

We determine which files are part of the project at the time the language server starts. This is done by scanning the file system. Dynamically adding files to the project based on shebangs discovered at the time that a file is open would add a bunch of complexity and would lead to inconsistent behaviors, so I think we'd be pretty reluctant to do this.

If you want multi-file rename to work in this case, I recommend that you rename your scripts to end in ".py". Your use case is highly atypical.

[heejaechang]

the original scenario should be fixed in the next release.

[muhzii]

@heejaechang Thanks, yes, that is correct. Such scenario would merely be a nice-to-have here, though.

@erictraut I understand, thanks for your clarification. Well, this is not really a huge annoyance, albeit something that seemed simply odd compared to how everything else works nicely.

[bschnurr]

This issue has been fixed in version 2022.3.2, which we've just released. You can find the changelog here: CHANGELOG.md