Want way to specify alternative leafname to replace Cargo.toml

[ijackson]

Suppose for some reason (eg #6713) it is necessary to edit the Cargo.toml of one or more crates one is buidling. it would be much nicer if this could be done by creating an alternative, massaged, file, for each Cargo.toml.

The alternative filename could be non-version-controlled, so one wouldn't need to dirty the git tree.

To support this, and other unanticipated requirements, I suggest that cargo support an option or configuration parameter which replaces the leafname Cargo.toml everywhere. Ideally the option would be a list of values, so that it is possible to specify fallback to Cargo.toml

[joshtriplett]

First, if you need to edit the Cargo.toml of a whole dependency tree of crates, that suggests a fix needed in Cargo itself, such as a mechanism to replace crates deeper in the dependency tree (which exists), or other dependency-tree-wide options (which may or may not already exist). Please file issues for any such things if you encounter them, so we can fix them in Cargo rather than hacking around them.

Apart from that, this filename is intentionally non-configurable and not a list of searchable filenames, for quite a few reasons. Many tools other than cargo parse Cargo.toml, and expect to find it in that path. I also don't think having a "substitute this filename in place of Cargo.toml" mechanism is the right way to make a systematic change throughout a tree of dependencies; after all, that still requires you to unpack all the dependencies and edit their sources.

On the other hand, I do think it makes sense to have a mechanism to substitute a completely different Cargo.toml file in place of the one of an arbitrary crate in your dependency graph. How would you feel about a way to crate a mapping similar to [replace], but instead of replacing the whole crate source, it just says "get the crate source from the registry but replace Cargo.toml with the file at this specified path before building"? Cargo would then unpack the crate source to a temporary location, replace Cargo.toml, then build.

[ijackson]

(I deleted this comment because I misdirected it to the wrong issue. Sorry.)

[maxcountryman]

@joshtriplett I think the [replace] functionality would be quite handy.

[joshtriplett]

@maxcountryman That functionality exists today, in .cargo/config. (We've talked about adding it to Cargo.toml as well.)

[joshtriplett]

I don't think this issue is likely to be implemented in the future; Cargo.toml being a fixed name is an intentional design choice to simplify tooling, and the underlying problem this was proposed to address sounds like something better fixed via specific functionality within Cargo.

@rfcbot close

[rfcbot]

Team member @joshtriplett has proposed to close this. The next step is review by the rest of the tagged team members:

• @Eh2406
• @alexcrichton
• @ehuss
• @joshtriplett

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[ijackson]

tl;dr: This is still needed (i) as part of the workaround for #6713 (ii) to provide flexibility in general.

Hi. Please can you reconsider.

This proposed feature has numerous possible uses. I mentioned one of them - the unpublished local crate problem: #6713.

The .cargo/config [replace] feature doesn't solve that problem because it doesn't work for totally unpublished crates. See the issue. Indeed in #6713 @kankri already asked for precisely "change Cargo.toml leafname" as an alternative approach!

The proposed partial [replace] that replaces only Cargo.toml would sometimes be able to do the same thing. But it has some downsides: it involves providing .cargo/config, and it is currently not so easy to have cargo merge multiple .cargo/config so complex massaging would be needed. If what is needed is to override Cargo.toml, it is an additional layer of indirection which hides the replacement away. Also it is a lot more complicated to implement. The Cargo.toml leafname change, with fallback to Cargo.toml, would be straightforward.

Stepping back a bit: it is very disappointing to read that cargo is intentionally inflexible.

Cargo's mission includes interoperating with pre-existing build systems, and that includes both calling other build systems (eg via build.rs) and being called by other build systems. Other build systems vary a great deal, particularly in their management of dependencies. Dependency management also has security and policy implications so local decisions must be allowed to take precedence. All this means it is vital that Cargo be flexible enough to cope with unanticipated or minority use cases.

It is a poor answer to say that problems that might be solved by additional flexibility on Cargo's part should be dealt with "via specific functionality within Cargo".

Sometimes the requirement is sufficiently unusual that it does not make sense to spend the effort to make it a first-class feature in cargo, nor to to expect every cargo user to deal with its specifics. Sometimes the requirement is urgent, and patching cargo is too slow.

And sometimes, as in the case of #6713, even a very sorely needed and extremely general requirement remains unsatisfied for a long period of time.

I have mentioned #6713 several times but I don't think that is the only reason why someone might want to use an alternative Cargo.toml without dirtying their source tree. Other possibilities would seem to include wanting to change which features are enabled deep in the dependency stack, or overriding dependency version specifications.

I appreciate that there is resistance to adding features willy-nilly; it seems to me that much of this is influenced by the Rust language's quite proper resistance to features and complexity. But languages are not build systems.

Build systems which try to maintain simplicity by restricting the ability of their users to override things just make life difficult for those users. The user's requirements are not typically very negotiable (and nor should the user be expected to compromise). The result of trying to make cargo "simple" by making it inflexible is that whole systems built on top of cargo become more horrific, not less.

Thanks for your attention.

[ijackson]

Hi. As I understand the governance process, this request will be closed by rfcbot unless someone on the team acts to stop that. @Eh2406 @alexcrichton @ehuss @joshtriplett Would someone from the team please acknowledge my comment of the 28th of May. Obviously I would prefer it if you would do that by tagging a concern with the rfcbot. However, if you feel that my comments have no merit, or that my concerns are outweighed by the contrary points that are already mentioned in this MR, then please say so explicitly. Ian.

…

-- Ian Jackson <ijackson@chiark.greenend.org.uk> These opinions are my own. Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter.

[Eh2406]

Stepping back a bit: it is very disappointing to read that cargo is intentionally inflexible.

There are maximally flexible build systems. makefiles and setup.py come to mind. Why does Cargo make you specify the dependencies before the project is compiled, why can't you add them as part of the Build.rs as setup.py lets you do? Because we have other nice things that we prioritize over that use case.

In my understanding the priority for cargos support for "upstream changes" is that it should be on the path to making a PR to share the fix with the rest of the community. We do not require copy-left, but the tooling is intended to make it easier to contribute it back then to keep a private fork.

I have reread the conversations linked several times. Your post rehashes the the arguments for this in one place, thank you for that. I seem to have missed what new arguments your post adds that were not made before closing was proposed. Can you help me see?

[ijackson]

Jacob Finkelman writes ("Re: [rust-lang/cargo] Want way to specify alternative leafname to replace Cargo.toml (#6715)"):

There are maximally flexible build systems. makefiles and setup.py come to mind. Why does Cargo make you specify the dependencies before the project is compiled, why can't you add them as part of the Build.rs as setup.py lets you do? Because we have other nice things that we prioritize over that use case.

tl;dr: Yes, offering flexibility to packages is hazardous. But, flexibility for users (those who run cargo) is simply good. There is a difference between (i) flexibility for the person who runs the build system (let me call them users), and (ii) flexibility for the person who writes the package (let me call them authors). The build system mediates the interface between authors and users (obviously the same person might perform either or both of these roles with respect to different packages at different times). Flexibility for users is essential because like any software, it is the purpose of the build system to serve the person who runs it. Flexibility for the user is not a problem because if the user does something strange that breaks things, the pain is felt locally by the user. The user gets to live with the consequences of what they did. Conversely, flexibility for authors is a very mixed blessing. Hazardous, even. If an author does something strange in their package, which breaks on some user's system, the author is unaffected and the user (perhaps many users and downstreams) have to fix it. Inflexibility for package authors is part part of making packages build everywhere, automatically generate docs, run CI, and soon. Dependencies computed at runtime are indeed hazardous. They can make life easier for authors but they make life harder for everyone who consumes the package. I agree with cargo's decision to rule that out. Conversely, inflexibility for users is simply an inconvenience and leads to users using workarounds. Back to this specific PR: everything that can be achieved by my requested feature could be achieved instead by linkfarming. It's just that linkfarming is complex and tiresome and has a lot more edge cases and cleanup work to do. We should to to avoid creating a situation where the best way for a user to solve their problem is linkfarming.

I have reread the conversations linked several times. Your post rehashes the the arguments for this in one place, thank you for that. I seem to have missed what new arguments your post adds that were not made before closing was proposed. Can you help me see?

As I see it thei things my comment added were were: * Contrary to the implication in the first comment, which seemed to be the main explanation for why to close: the `[replace]' feature does not solve one of the main motiviating use cases. * I saw few stated reasons for `this filename is intentionally non-configurable'. The comment says `quite a few reasons' but mentions only one very weak reason. [1] The main impression I get from the comments from the team in this PR is simply that this is an intentional inflexibility in Cargo. So I tried to address that by providing reasons for offering flexibility (to *users who run cargo*, not to package authors). This discussion about the merits of flexibility is new and I think actually goes to the heart of the matter. Thank you for engaging with that. See my response above. * Also, the statement that this was `better fixed via specific functionality within Cargo' lacked reasons. I aimed to explain why I felt that it not a useful response to a request for a general escape hatch feature. [1] The one reason actually given here for why this feature would be a bad idea is this:

Many tools other than cargo parse Cargo.toml, and expect to find it in that path

That some other tools that read Cargo.toml might not be affected by this option just means that a user who is using this option might need to adjust the way they work with those tools, or simply not use them. For the avoidance of doubt I am definitely not suggesting that the alternative leafname should be specifiable by crates. That would be very bad.

…

-- Ian Jackson <ijackson@chiark.greenend.org.uk> These opinions are my own. Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter.

[joshtriplett]

@ijackson

This proposed feature has numerous possible uses. I mentioned one of them - the unpublished local crate problem: #6713.

#6713 is an issue I don't think we should close, and I agree it's a real problem we should solve. In general, though, we prefer to add solutions to problems rather than adding features whose primary use case seems to be to work around problems, even if that means that it takes longer for the solution to become available. And in particular, adding something whose primary use is working around a problem is the kind of thing we'd prefer not to do.

It's also worth pointing out that we have compatibility requirements that mean anything we add is something we will maintain compatibility with forever. (Those compatibility requirements apply both to the Rust compiler and to the Cargo build system.)

I'm aware of the general problem of making the perfect the enemy of the good, but there are also downsides to adding surface area. And especially, we usually try to have one primary recommended way to solve any given problem, rather than multiple different ways.

Ironically, this exact feature makes it harder for other kinds of external build systems and tools to integrate with cargo; any such build system would break if it weren't aware of this override mechanism, whereas today such a build system can count on looking for Cargo.toml. (That's one example of many possible concerns and complexity that this would introduce.)

The .cargo/config [replace] feature doesn't solve that problem because it doesn't work for totally unpublished crates. See the issue.

I've read through the crate, and I'm not sure I quite understand why [replace] or [patch] would not be able to handle an unpublished crate. It should be able to do so, so either we need to improve the documentation (perhaps a new use-case example in https://doc.rust-lang.org/cargo/reference/overriding-dependencies.html ) or fix a bug if this doesn't work.

For clarity: this absolutely should be supported, so we should fix it if it isn't, and/or document it if it's not easy to find or use. Note that several aspects of registries are a little less clear than they should be.

Cargo's mission includes interoperating with pre-existing build systems, and that includes both calling other build systems (eg via build.rs) and being called by other build systems. Other build systems vary a great deal, particularly in their management of dependencies. Dependency management also has security and policy implications so local decisions must be allowed to take precedence. All this means it is vital that Cargo be flexible enough to cope with unanticipated or minority use cases.

It is a poor answer to say that problems that might be solved by additional flexibility on Cargo's part should be dealt with "via specific functionality within Cargo".

Cargo is not a "here's all the mechanism, add the policy yourself" build system; it's intentionally a build system that includes a substantial amount of policy, with which to curate an ecosystem. You are also free to put arbitrary build systems either underneath or on top of Cargo, and many people do; for instance, it's common to use Cargo to build a static or dynamic library and then link that library from C.

Cargo also goes to a great deal of effort (including ongoing development work) to provide facilities to integrate into other build systems, including large policy-filled build systems such as those commonly used in large monorepos, or those used in Linux distributions.

Other possibilities would seem to include wanting to change which features are enabled deep in the dependency stack, or overriding dependency version specifications.

Both of those are also reasonable to control via .cargo/config, for local use.

I appreciate that there is resistance to adding features willy-nilly; it seems to me that much of this is influenced by the Rust language's quite proper resistance to features and complexity. But languages are not build systems.

The same principles that apply to the Rust language apply to the Cargo build system; they're part of the same ecosystem. The same stability and compatibility guarantees apply; the same desires to maintain consistency, orthogonality, and various policies apply; the same careful approach towards new features applies. In general, we consider Cargo to be part of the overall Rust story.

Build systems which try to maintain simplicity by restricting the ability of their users to override things just make life difficult for those users. The user's requirements are not typically very negotiable (and nor should the user be expected to compromise). The result of trying to make cargo "simple" by making it inflexible is that whole systems built on top of cargo become more horrific, not less.

I think it's generally reasonable for a build system to steer people towards preferred/conventional/idiomatic ways of doing things. That doesn't mean it should be impossible to do other things, but rather that there's an intentional "path of least resistance".

Cargo also needs to balance between the complexity of Cargo itself and the complexity of the overall system including Cargo. We push back on things that would add substantial complexity to Cargo, but we also push back on things that are relatively simple in Cargo but on balance may result in more overall complexity for a use case. In this case, you're arguing that you don't want more complexity pushed into the build system outside of Cargo, and I'm suggesting that the way we'd prefer to solve your use case would result in even less complexity in the build system around Cargo.

To summarize, I think it's a good thing that Cargo asks "what is your use case, what problem are you trying to solve?", and then provides specific functionality for that use case. There are also various general mechanisms for overrides, such as .cargo/config files, alternate registries, build plans, and others. In general, we're likely to follow roughly this process for many feature requests:

• What's the underlying use case? (Our normal answer to "could you add feature X so that I can do use case Y" is to evaluate the best solution to Y, which may or may not be X.)
• Does one of the existing mechanisms in Cargo support this? (If not, could we improve such a mechanism, rather than introducing a new one, to maintain orthogonality?)
• Could we (and should we) add a mechanism in Cargo to support this kind of use case?
  • If we shouldn't, are there enough facilities in Cargo that it's possible to implement this outside of Cargo?
    • If there aren't, should we add them, and will doing so complicate Cargo for the common case?

In this case, I think that process points towards mechanisms like [replace] or [patch], and if those mechanisms don't suffice for the use case, then I think we should improve those mechanisms so that they do.

[joshtriplett]

@ijackson One further clarification on something:

Flexibility for the user is not a problem because if the user does something strange that breaks things, the pain is felt locally by the user. The user gets to live with the consequences of what they did.

Features added to Cargo aren't just used by end-users, they're also used by build systems consuming Cargo, which may in turn be used by end-users who aren't the ones working on such build systems, and thus are bearing the consequences of something they didn't do. This is a major reason why we focus on solving use cases directly and not just via "extension points" that work around those use cases.

(We do still provide some fully general extension points, notably build plans, but those are for the full-on "I want to do all of Cargo's job" use cases, not for the "I want to change one small thing" use cases. And we provide other extension points, but we think about how those extension points may be used, and if they're solving problems or working around them.)

Also:

We should to to avoid creating a situation where the best way for a user to solve their problem is linkfarming.

I'd certainly agree with that, but I don't think there's a dichotomy here where the only two ways to address your use case are this exact mechanism or linkfarming.

[rfcbot]

The final comment period, with a disposition to close, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.