Add SIG Interoperability proposal #62
Conversation
|
This is exciting - I'm really interested in participating. |
|
Awesome, this looks great and happy to get involved! |
|
Yes, definitely excited to see end-user organizations being involved in this initiative! |
|
Thank you all for the comments. I updated the proposal and included @tracymiranda and Priyanka Sharma as members to the SIG. |
|
Hi there! Google Tekton Lead here 👋 I am very interested in participating in this as well :D |
|
I'm not personally familiar with how SIGs are scoped for other LF funds, but the proposal here seems extremely broad and its target problem space is not particularly clear to me. That said, I'm thrilled to see opportunities for fruitful collaboration between maintainers and users of a variety of testing and deployment automation systems, and look forward to seeing what comes of it. If there's a way I can be useful, I'm happy to lend a hand. My biggest concern, which we also covered at the OpenDev Conference in Vancouver last year, is that the broad differences in feature sets between various popular CI/CD and project gating platforms means that the way their jobs are defined and the sorts of data they can report under which circumstances is very disjoint. Any commonalities between them are likely to be woefully contextless, and I'm really not sure that anything can (or should) be done to "fix" it. These are (in most cases I'm aware of) Turing-complete automation frameworks, so coming up with protocol or data interchange format standards isn't likely to address the underlying challenges I think you're trying to get at with regard to bespoke glue between disparate systems. |
|
Great! Very happy to be involved as well. |
|
Thanks for the comments @fungi. With regards to the scope of the SIG - if and when the SIG is approved, this is one of the things we need to work and perhaps come up with a structure similar to CDF Security SIG with subgroups focusing on certain areas. Apart from that and as you might have noticed, we have broad participation from the users of CI/CD tools/technologies in the SIG. The user community could provide valuable input with regards to the challenges they face which could then help us to identify the list of areas we can focus under the SIG and have the scope clarified. About your concern regarding the differences between various popular CI/CD and project gating platforms - the SIG could be a forum to discuss questions like these to see if there are ways to address the challenges and if so how. |
|
I am interested in participating in this. One thing I'd like to see as someone who's managed and used a few different CI systems is that it's not easy to migrate from one CI platform to another. It would be cool to have some standard like for example Ansible (doesn't have to be but this is my top of head example) for defining the build code in such a way that it can easily be moved around between multiple CI systems. Typically CI's might have different trigger mechanisms and post build publishing mechanisms and other unique features that make the CI system itself unique and cool but when it comes to actually running a code build, typically we need to be able to say "where to run the build" and "how to run the build". I'd love to see the "how" be more streamlined so that the possibility of using multiple CI systems is easier, even possibly in parallel. |
I agree @fungi that it's a broad concern - and I think it's something that the CDF is in a uniquely positioned to tackle! Like @fdegir said, having folks from all over the CI/CD space come together to discuss this really gives us a chance to do something cool. One interesting thing that @kimsterv points out a lot, and that is motivating the Tekton project here at Google is that when you squint at a lot of these systems, they do end up having the same basic components deep down, just with slightly different names and shapes.
Interfaces are everything imo, and if we add some clarity around what interfaces users should expect as they move between CI systems, that could make a huge difference! |
|
This looks great. From a process standpoint, it looks like you need to identify a TOC sponsor and a chair for the SIG. I'm happy to serve as the TOC sponsor. @fdegir are you willing to chair this SIG? If not, I think @bobcatfish might be willing. |
|
Thanks for sponsoring the SIG @dlorenc. I'll update the proposal and list you as the TOC sponsor. Regarding the chair; yes, I am willing to chair the SIG and we can perhaps follow the example set by Security SIG with a chair and co-chair setup. I suppose co-chairs can be identified if/once the SIG gets approved by the TOC and when the SIG meets first time. |
As someone who helps write and maintain CI software which actually already uses Ansible for its job description language, I can say that the challenge there is once again context-related. The scheduling system in this case supplies a variety of metadata about build context (in the form of Ansible variables) which job definitions can consume at runtime to make decisions on what needs to happen in a given build. Transpiling between, say, shell scripts and Ansible playbooks isn't really the hard part, it's coming up with a superset context representation which can be ported between different kinds of CI/CD and project gating implementations which have their own World views on what sort of information is relevant based on the types of activities they're designed to automate and the environments in which they're meant to operate. For example, I'm struggling to see how you would consume jobs written to perform cross-project upgrades between proposed changes for like-named branches of a related set of Git repositories (this is not a made-up example by the way, as I'm sure you're painfully aware) in a CI system which doesn't actually have any idea about how to provide multiple Git repository states and works from the perspective of a single source tree for one project. The solution is likely going to be reimplementing a number of features from the first CI system within additional routines for the version of the equivalent job you're going to run in the second. Though at least if you use a modular job definition language you can reuse and share that reimplemented logic with others who might face the same challenges. So (setting aside for the moment the fact that some software authors may consider their choice of job description language an integral feature) while I find the idea of a universal job description language compelling, standardizing a job description language is not really a solution to the harder problem of mapping contextual metadata for disparate systems. These are ultimately the sorts of questions I think such a SIG will have to be able to identify and explain, at the very least, even if it can't necessarily devise a reasonable solution to them. |
|
This is great. I am very much interested in participating. Thanks! |
|
Thanks for the interest @dhgautam. You might have noticed that the vote on the SIG is in progress and will soon be concluded hopefully. Please keep an eye on CDF maillists and join to the CDF TOC meetings to get the updates. We will start setting up the logistics as soon as the SIG is approved to collaborate on this. (repos, meetings, maillists, etc.) |
|
Thanks @danlopez00 for merging the PR. As many follow the PR, I would like to share some logistics here as well since maillist and slack channel created for the SIG recently and not many are subscribed/joined to them yet. First of all, we need to decide on the meeting details. Please respond to the poll from the link below in order for us to decide the meeting frequency and day/time. The survey will be closed on Sunday so we can have our first meeting scheduled for next week. https://www.surveymonkey.com/r/5X6GXPM Secondly, here are the details of the repo, maillist, and slack channel. Please subscribe to the maillist and join slack channel so we can start collaborating. maillist: https://lists.cd.foundation/g/sig-interoperability |
|
@fdegir I tried searching for CDF Slack details but wasn't able to find it, can you share a link? |
|
@zxiiro CDF Communication details are available here. https://github.com/cdfoundation/toc#communication And here is the link to CDF Slack. |
Inital proposal.