Conan 2 workspaces

[memsharded]

This ticket purpose is to gather and centralize discussion of all the other opened tickets, now that we are starting to resume work on workspaces for Conan 2.

• The idea of using CMakeDeps in [question] Can an editable package be build using only cmake commands #15880 (comment) by @mattangus needs to be explored
• Adding the capability of having local configuration (custom commands) that compose with the global cache custom commands [feature] Folder specific custom commands #16023
• Adding the capability of having local config (global.conf) in a project, and compose it with the global one: [feature] On-the-fly configuration for 2.0 #6761
• Being able to have "light" workspaces which are mostly just a list of editables: Lightweight workspaces aka Temporary editables #5566
• CMake necessary changes to make the old 1.X workspace work: [workspaces] update documentation #7206
• [question] Shared code and workspaces #5939 seems it would have been solved with new layout() definition
• The issue [workspaces] Workspaces should be Super Builds #5762 contains a large discussion, the globalness of editables need to be addressed (probably the first issue a workspace 2.X will solve)
• Issue [workspaces] Decoupled Workspaces #6844 suggests more decoupled workspaces
• Ticket [feature] Extend .conanrc for profiles information #15209 suggest to be able to have profiles locally defined by .conanrc. If considered, this would be a workspace feature.

[aander80]

I am very keen on seeing workspaces being introduced!

Currently, I can think of two features that I would like to have.

Single build command for building multiple packages in a workspace

I have created a fork of your own fork and implemented a very hacky solution of this: https://github.com/aander80/memsharded-conan/tree/feature/workspace. I believe you should be able to clone my fork, checkout feature/workspace, cd into the workspace folder and run ./demo.sh (I ran it in WSL, have not tested it in Windows).

The idea is to have a command like conan workspace build <targets> (conan ws:build <targets> in my fork) and have conan build all the targets and editable dependencies in the workspace given the build order graph (note: I was not sure how to get this correctly using the API so my solution is probably not fully correct, but it seems to work for my demo). If <targets> is empty, the entire workspace is built. Of course, this would require efficient caching in each build.

On-the-fly editables and overriding dependencies

I have been looking into Cargo workspaces for Rust. There you have the wonderful feature of setting dependencies for the workspace, and in each package/crate re-use the workspace dependencies. Example here:

• Workspace: https://github.com/astral-sh/uv/blob/main/Cargo.toml#L19
• Package/crate: https://github.com/astral-sh/uv/blob/main/crates/uv-cli/Cargo.toml#L16

I imagine something similar would be very powerful for conan, where you can select to override dependencies in a workspace. When you are working with large projects with many commits being merged every day and with dynamic versioning where each commit gets a unique version, managing static editables will be tough to manage (especially since just creating a new commit during local development would force you to change your editables).

To solve this, it would be very useful if you have multiple repos, add them to a workspace, and the dependencies would be automatically injected in the dependency graph as editables, without having to configure them statically as editables in the conan cache.

In my example, let's assume I have two repos: say and hello. Here, I likely always want to make sure I use my local copy of say, and not the latest version in the cache (which might override my local copy/repo). I have currently thought of these different behaviors:

• You always inject/override dependencies in the graph. If hello points to say/[~1.0] but I have say/1.1.0 in my local repo, Conan would still select my local say/1.1.0 version.
• Alternatively, this could be an option/additional parameter that you set, similar to include_prerelease. I am not creative when it comes to names, but it could be something along the lines of allow_workspace_override. This could also be a core configuration that you can set to apply to all dependencies, similar to core.version_ranges:resolve_prereleases.
• It could also be that you must conform to the version ranges in the version ranges. However, Conan could prioritize the workspace versions ahead of the cache, so that if I have say/1.1.0 in my local repo but there is also say/1.1.1 in my cache, the dependency say/[~1.1] would still resolve to my workspace say/1.1.0. (I guess this is identical to the previous item in this list, with the allow_workspace_override feature turned off)

I understand that this could be very tricky to solve since you might have multiple versions and contexts of packages; some might require say/1.0, another say/2.0 etc., some might want to use say as a require, other as a tool-require and so on. I imagine this could be managed by a combination of the second and third items in the behavior list above, but I am probably missing loads here.

For me I think a key feature for simplicity is to have on-the-fly editables that are only injected in the workspace/for the current command.

I hope I managed to make myself understood in my ramblings here! 😄

[memsharded]

Hi @aander80

Thanks for your feedback.

Regarding your first point:

The idea is to have a command like conan workspace build (conan ws:build in my fork) and have conan build all the targets and editable dependencies in the workspace given the build order graph (note: I was not sure how to get this correctly using the API so my solution is probably not fully correct, but it seems to work for my demo). If is empty, the entire workspace is built. Of course, this would require efficient caching in each build.

This can be done today with the conan install <path-to-root> --build=editable. This will call the build() of every package in editable mode in the current dependency graph of <path-to-root> in the right order.

This will be most likely improved with a conan workspace build or similar, but that command will very likely just be a --build=editable under the hood, leveraging some definition in the workspace of the "path-to-root".

About your second point, I think it is very aligned with our current vision, and I have already started to work on it, I have some working code in a branch. Keep tuned, I will try to share when possible to start gathering feedback, I will share it in this thread.

[aander80]

Thanks for your reply! I am happy to hear about your vision, looking forward to seeing this implemented some day! 😄 It is also great to hear building multiple editables is possible already today, that should make multirepo development easier already now, although it might require a few extra commands!

[memsharded]

Hi everyone,

This is finally happening, we have the first PR for next Conan 2.10: #17272

Feedback is very welcome, if you can run from my source branch, new conan workspace commands, and if you have any issues or questions, please open new tickets, one for each of them. Having early feedback, even before the Conan 2.10 release can be helpful, otherwise, it will be released in 2.10 for dev-testing only, not for production, but looking forward your feedback in any case. Thanks!

[memsharded]

#17272 has been merged, it will be in next Conan 2.10.

It needs env-var defined CONAN_WORKSPACE_ENABLE=will_break_next, as for this iteration it is intended for dev-testing, not for production. Really looking forward to hearing your feedback!

[mattangus]

I've just done some poking around the new command and I have a few initial thoughts

1. It would be good to have a conan workspace new command similar to conan new ... in order to generate the boilerplate workspace
   • Ideally with a few templates, e.g. monorepo where it searches each subfolder for a conanfile.py, basic where it just has an empty list of folders
2. It might be useful to have a way to clean the editable build folders. This is sometimes required when a cached variable isn't set properly or is changed. Going into each individual editable package and removing the build folder/cache is tedious.
3. conan workspace open might need a more clear description in the help message, it's not clear what it's for
4. This might be difficult but ideally once workspace is set up the user could just use cmake to rebuild the editable packages
   • i.e. app depends on project1, I make a change in project1 and run cmake --build --preset conan-release in the app folder, the change in project1 is detected and recompiled.
   • The main thought behind this is that with more complicated projects the dependencies can take a while to enumerate so running conan build ... each time you want to compile could increase incremental build times by a lot.
5. I'm not sure if it's my misunderstanding or something missing in workspaces but IDE integration seems like it will be difficult with the current implementation. e.g. clangd expects a single compilation database. But a monorepo with a workspace has a compilation database for each editable package.

[mattangus]

I think I've been able to come up with a reasonable solution for local development for point 5 (and probably point 4) above. You can add each project in the mono repo using fetch content with OVERRIDE_FIND_PACKAGE on, then you don't need to change your cmake code for all 3 cases: editable mode, normal conan build, and mono cmake build. You just have to make sure that the targets defined in your conanfiles match the targets defined by your cmake (which is probably best practice anyway).

Here is some cmake that got the job done in my little test repo:

cmake_minimum_required(VERSION 3.15)
project(monorepo CXX)

include(FetchContent)

function(add_project PROJECT_PATH_AND_NAME)
    FetchContent_Declare(
        ${PROJECT_PATH_AND_NAME}
        SOURCE_DIR ${CMAKE_CURRENT_LIST_DIR}/${PROJECT_PATH_AND_NAME}
        SYSTEM
        OVERRIDE_FIND_PACKAGE
    )
    FetchContent_MakeAvailable(${PROJECT_PATH_AND_NAME})
endfunction()

add_project(project1)
add_project(project2)
add_project(app)

You have to set the output directory for each sub project as well like so:

set_target_properties(project2
    PROPERTIES
    ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}"
    LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}"
    RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}"
)

Ideally they would be set to the following, but I can't figure out how to get conan to play nice with this.

set_target_properties(project2
    PROPERTIES
    ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/lib"
    LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/lib"
    RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
)

[memsharded]

Thanks very much for the feedback @mattangus.
I am adding a conan new workspace template to the existing conan new command in #17538, if you want to check it. I have started by adding some libs and one exe, and a static workspace, not a dynamic one. Not sure if the dynamic one would be too much, but lets follow up in the PR #17538, thanks!

I'll follow up on the other items when possible, but looking good, so thanks very much!

[memsharded]

Hi everyone, we keep doing progress in workpaces:

• New conan new workspace template will be in next Conan 2.12
• New conan workspace build command, with definition of "products" in the workspace will also be in next Conan 2.12.

This is still an "incubating feature", but doing progress, please keep the feedback coming, so we can continue maturing it.

@mattangus

As quick feedback about the workspace clean feature: I have tried it, but it is far from being easy. It has a bunch of edge cases and challenges, like the "build-folder" is not really defined without fully evaluating the graph for one configuration, as build folders can be different per configuration. Then in multi-config environments one Release configuration "build-folder" will be common to Debug, so it is not possible to simply wipe build folders without affecting different configurations than the currently defined one.

It might be worth to create a dedicated ticket for it, and discuss it in detail there.

[memsharded]

Proposed some very early stage poc for the monolithic CMake build in: #17675

Running from that PR branch:

# create an external dependency to the workspace first
mkdir myproj && cd myproj
conan new cmake_lib -d name=mymath
conan create .
rm -rf *

# now the actual workspace
conan new  workspace -d requires=mymath/0.1
conan workspace install
cmake --preset conan-default # conan-release in Linux

[memsharded]

Update: PR #17688 has proposed a new syntax for the conanws.py file, closer to the typical conanfile.py syntax, something like:

            import os
            from conan import Workspace
            class MyWorkspace(Workspace):
                def name(self):
                    return "myws"
                def add(self, ref, path, *args, **kwargs):
                    self.output.info(f"Adding {ref} at {path}")
                    super().add(ref, path, *args, **kwargs)
                def remove(self, path, *args, **kwargs):
                    self.output.info(f"Removing {path}")
                    return super().remove(path, *args, **kwargs)

And with access to the underlying conanws.yml via self.conan_data.
It has already been merged to develop2 branch, so it can be tested (still under the CONAN_WORKSPACE_ENABLE=will_break_next gate)

[Artalus]

May I please ask for something like CONAN_WORKSPACE_SILENT=yes_i_understand_the_consequences to get rid of the nagging yellow warnings? Or at least contain them to the conan workspace commands themselves?
I believe I understand the intent behind the omnipresent reminder that things will break, but currently the output is produced from the API class constructor when a conanws.py-like file is present. Since this one is called on every Conan command - even before Conan even checks that such command exists - the warnings sometimes get.... tiresome 🙃

[memsharded]

Thanks for the feedback @Artalus

The warnings will not be silented, because what is even worse than the inconvenience of yellow warnings in the output is users complaining because the release broke something.

But I agree those seems too many, it is not intended that there are those too many, how is that happening? Can you share some code or something that reproduce that? The idea is that you get only one per conan xxxx command, and only when using workspaces.

[Artalus]

users complaining because the release broke something

... which is why the silencing would be hidden not by one, but by two "i agree" envvars 😉
But I agree with the notion.

how is that happening? Can you share some code or something that reproduce that?

This one is mostly on me; we have a conan info compatibility method in our code, that calls the command for v1, but for v2 does

    from conan.api.conan_api import ConanAPI
    api = ConanAPI()
    ...

So it will get better once I refactor out the API initialization.
Still it feels slightly weird to keep the warning it in the WorkspaceAPI constructor - because same thing would happen to anyone who is using invokes multiple conan commands in their build wrappers.

[memsharded]

Still it feels slightly weird to keep the warning it in the WorkspaceAPI constructor - because same thing would happen to anyone who is using invokes multiple conan commands in their build wrappers.

We know. The problem is that the Workspace intercepts things very early. So early that it can define its own CONAN_HOME for the current workspace, which in turns affect to all the configuration loaded from global.conf, the remotes.json that are loaded, etc. So the activation of the workspace affects a lot of things without even calling the first conan workspace command.

[memsharded]

Important update: #17675 was merged, it will be in next Conan 2.13 (still as incubating)

This is a major improvement in workspaces. Feedback to get it out of incubating very necessary, please give it a try and report (we will update the docs "incubating" section)

[Artalus]

I am trying to apply conan workspace install approach to our monorepo, having Conan installed from 2.13.0dev at a21c0cf .
Not all packages from our dependency tree have been adapted vor v2, so self.load_conanfileing everything from glob("*/conanfile.py") yields all sorts of errors - from invalid imports to 3rdparty deps missing in our conanv2 remote.
I expected this and was going to filter out through dependency tree manually, somewhat like this:

def editables(self):
  editables_result = {}
  todos = [TARGET_PROJECT]
  while todos:
    cf = Path(todos.pop()).as_posix()
    conanfile = self.load_conanfile(cf)
    editables_result[f"{conanfile.name}/editable"] = {"path": f}
    todos.extend(__something_to_get_direct_dependencies_of(conanfile))

...But the only two interfaces I could think of seem to be unavailable during editables() evaluation?

• conanfile.dependencies() raises an exception, which is kinda expected since the graph is being populated right now:

  File "/home/igor/git/KdMonoRepo/conanws.py", line 44, in editables
    print(conanfile.dependencies)
  File "/home/igor/git/KdMonoRepo/.me/venv/lib/python3.8/site-packages/conan/internal/model/conan_file.py", line 200, in dependencies
    self._conan_dependencies = ConanFileDependencies.from_node(self._conan_node)
  File "/home/igor/git/KdMonoRepo/.me/venv/lib/python3.8/site-packages/conan/internal/model/dependencies.py", line 93, in from_node
    for require, transitive in node.transitive_deps.items())
AttributeError: 'NoneType' object has no attribute 'transitive_deps'

• conanfile.requires is constructed but empty, since apparently def requirements() has not been called by this time:

            print(type(conanfile.requires))
            print(conanfile.requires)

<class 'conan.internal.model.requires.Requirements'>
odict_values([])

I could probably call conanfile.requirements() manually to populate the requirements dictionary, but seeing how requires.py: class Requirements: def __call__ contains a check for if self._requires.get(req): raise ConanException, I do not think this is a good idea.

What are my options @memsharded ?

[memsharded]

Following up in a dedicated issue: #17844