Broadcast Framework: Create Multi-client Broadcast Framework

[0xTheProDev]

Work In Progress: Proof of Concept for VS Liveshare equivalent in Theia

The broadcast framework is required to perform operation on collaborative editing. To create similar functionalities as VS Liveshare, all kind of events and event metadata need to be sent to every client connected to same server workspace.

Signed-off-by: Progyan Bhattacharya bprogyan@gmail.com

[paul-marechal]

Thanks for the Pull Request!

I am a bit excited to see that you are working on the ground-work for some live-share feature!
What is the scope of this, do you plan to actually have pair programming in Theia?

I also didn't really try to understand what you are trying to do yet, but overall you seem to have understood how to make a Theia extension, so that's cool. Few comments though:

• Some files need to be moved to their respective folders (not everything is common)
• Code formatting seems off, relative to the project's standards (4 space tabs, I see 2 here).
  https://github.com/theia-ide/theia/wiki/Coding-Guidelines#indentation

[paul-marechal]

I did not realize, but it looks like this file contains an implementation, but you defined it in the common folder?

Usually concrete implementation aren't shared, not that it cannot be done, it is just that it requires rather specific components. In this case you should move most backend/frontend implementations to their respective folders (not in the common one).

Good candidates that should be moved:
-broadcast-client.ts
-broadcast-server.ts
-broadcast-watcher.ts

Unless maybe one of them really should stay in the common place, I think most should be moved.

Ah, and the servicePath too, might be better inside the broadcast-protocol.ts file, it is still common data :)

[0xTheProDev]

I have moved servicePath to broadcast-protocol.ts. Regarding other files, they are being used by both frontend and backend module.

[akosyakov]

First, we should make vscode Live Share to run. Second, Theia is designed in a way that it should be decided per a service where it should share some state with other client or not. I don't think we need a general broadcasting framework. Which services do you want to share?

[0xTheProDev]

This broadcast framework will be used to transfer state to all clients connected to it. We need to further refine a bit if we are having different access levels.
The idea is that, whenever a user opens a new file tab, or terminal session, it must be shared across all clients connected to same workspace server.

[akosyakov]

I don't understand a need for a framework here. Terminals already shared via singleton process manager on the backend. They are just not visible. For other extensions the same should be done, i.e. there should be a singleton service serving shared data for this extension.

[akosyakov]

File Tree is synching is already implemented as well, we are watching fs changes and reflecting them for all clients.

[0xTheProDev]

But Tab Syncing is not there, also in File Tree, if I click on a directory, it should open on all client instance.

[akosyakov]

For live-collaboration we should either pursue #1470 or what would be easier for now is to implement a new Theia extension with atom teletype. It does not need to belong to this repo, but could go under Theia org.

[0xTheProDev]

Can you explain the purpose served by #1470 in liveshare? I didn't quite get it.

[akosyakov]

But Tab Syncing is not there, also in File Tree, if I click on a directory, it should open on all client instance.

It's very questionable feature. I doubt a user wants to see it, the same for popping up terminals and editors. It could be fine to have a view which displays all running terminal processes on the backend and a user can open them when he wants (which is already possible). But changing user layout is too much for me.

Can you explain the purpose served by #1470 in liveshare? I didn't quite get it.

An idea was to sync all opened documents only once to backend from each user and allow LS and VS Code extensions fast access to them, since they seat on backend. Later such backend service could be reimplemented with something like atom-teletype to apply changes from all users to one state. But it's big arch change and it's better just implement live collaboration as an external Theia extension.

[0xTheProDev]

It's very questionable feature. I doubt a user wants to see it, the same for popping up terminals and editors. It could be fine to have a view which displays all running terminal processes on the backend and a user can open them when he wants (which is already possible). But changing user layout is too much for me.

I have a very specific need in mind to use this extension, and I agree that tab-syncing may not make sense for the general IDE usecase. I am building tab syncing for a specific internal use extension for my project.

With this PR, I'd like to get a general purpose client-to-client communication service in place for Theia, so any extension or service authors can also utilize this to emit and catch events across all connected clients and server (including handling of disconnections, new clients joining, etc). I think this PR should be considered only from this aspect.

Whether any future liveshare/teletype plugin will use this communication framework is left to the extension authors. But this general client-server-client communication framework in itself will be helpful for many many more general use-cases.

If this is acceptable, what would you recommend for this PR?

[akosyakov]

With this PR, I'd like to get a general purpose client-to-client communication service in place for Theia, so any extension or service authors can also utilize this to emit and catch events across all connected clients and server (including handling of disconnections, new clients joining, etc). I think this PR should be considered only from this aspect.

We have it already and use when appropriate. You create a singleton JSON-RPC service on the backend which collects clients on setClient and dispatch to them on other JSON-RPC requests. Such service have APIs suitable for use case.

I don't think spamming from one client to another by default is a good idea, especially if clients don't want sharing.

I have a very specific need in mind to use this extension, and I agree that tab-syncing may not make sense for the general IDE usecase. I am building tab syncing for a specific internal use extension for my project.

You could create a custom extension which introduces a special service for this use case and overrides some frontend service to share information about tabs. Who wants such features can install this extension.

[0xTheProDev]

We have it already and use when appropriate. You create a singleton JSON-RPC service on the backend which collects clients on setClient and dispatch to them on other JSON-RPC requests. Such service have APIs suitable for use case.

Can you specify which extension you're talking about?

I don't think spamming from one client to another by default is a good idea, especially if clients don't want sharing.

We can add access level and workspace id as such to avoid such scenario and have pleasant experience.

You could create a custom extension which introduces a special service for this use case and overrides some frontend service to share information about tabs. Who wants such features can install this extension.

That's what I probably be end doing about the special use cases I have, this PR just consists of Broadcasting framework.

[marcdumais-work]

hi @Progyan1997

Can you specify which extension you're talking about?

I think ATM it's not one extension that provides this, but rather each extension that needs this re-uses a common pattern. Here is one example from the Task extension, but there are others (search for "JsonRpcConnectionHandler"). In this case it's used to collect all clients and later send them notifications about tasks starting and terminating:

https://github.com/theia-ide/theia/blob/master/packages/task/src/node/task-backend-module.ts#L34-L44
https://github.com/theia-ide/theia/blob/master/packages/task/src/node/task-server.ts#L77-L79

[0xTheProDev]

@marcdumais-work

but rather each extension that needs this re-uses a common pattern.

Then should not it serve the best use case for such scenario, using just one extension for this in particular?

https://github.com/theia-ide/theia/blob/master/packages/task/src/node/task-backend-module.ts#L34-L44
https://github.com/theia-ide/theia/blob/master/packages/task/src/node/task-server.ts#L77-L79

And from what I've seen in this code is that it can easily be integrated into the service I built.

[akosyakov]

We can add access level and workspace id as such to avoid such scenario and have pleasant experience.

We want to keep access management out of scope of Theia, products using it can integrate they own user management. What they already do check out how sharing is implemented in Gitpod: https://www.gitpod.io/docs/33_Sharing_and_Collaboration/

I'm fine with improving individual services, i.e. making debug session shareable or improving terminal sharing. But within the established approach, I don't see why we need to have generic API, JSON-RPC services are generic to me enough already.

The really missing part is live collaboration, it would be cool if someone look into building it.

[akosyakov]

Thank you for the interest, but such concern will be handled from a case to a case as for now.