Add site channels for realtime

[unDemian]

Changes proposed in this Pull Request

The goal is to have real time comments appear in all Reader sections while keeping the number of joined channels per socket to a minimum. We are using one lasagna channel per site instead of having a channel for each post, that should help us reduce the number of joined channels and join/leave actions.

Concepts

Lasagna is built using Phoenix which gives us multiplexing capabilities allowing one socket connection to handle multiple concerns. At the moment we are having two types of channels (concerns) that are handled via the same websocket.

Sockets

At the moment we are connecting to the Lasagna socket only inside Reader, which means whenever a user navigates to the Reader (or one of its subsections) a socket is opened. Leaving the reader will close that socket. This happens in the lasagna middleware and it will make sure we only have one socket open at a time.

Lifecycle
We keep track of the socket connection status with the following states

• init socket was code was initiliazed
• opening socket is in the process of connecting
• open_error there was an error during connection step
• opened socket is opened for communication
• closed socket is closed for communication

Channels

There are currently two different channel types (topics) that are supported for realtime communication.

User Channels

User specific channel using a topic like user:wpcom:<userId> it will be joined when the socket is connected and will be left on socket disconnect. At this point this is channel is not used but it could be easily handling any realtime user specific concerns.

Site Channels

Site specific channel using a topic like public:push:site:<siteId>, it is used to transmit per site updates. A user can join multiple site channels depending on how many sites are of interested. Currently these are used only in the Reader section, to update comments in realtime.

These are rules for when a user can join or leave a site channel:

• A site channel can be joined once
• A site channel will be kept active for one hour after the join, to make sure we avoid unnecessary join/leave as the user is scrolling through articles
• A site channel can be left after 45 minutes of inactivity (no new messages)
• We can have a maximum of 20 site channels open at the same time, if a new channel has to be joined, the oldest (first joined) will be removed (left)
• Site channels that have posts that are currently viewed by the user will be kept active even if they don't meet the above requirements

We have a special management for these channels to make sure we don't have too many "opened" (joined) at the same time.

Feed full post view

Whenever the user clicks on a post from Following Sites it will be sent to a full post view for that feed item (url like /read/feeds/<feedId>/posts/<itemId>) at this point if the socket it is not connected we will attempt that and then join the site channel of this particular post.

Site full post view

Whenever the user clicks on a post from Conversations it will be sent to a full post view for that blog item (url like /read/blog/<blogId>/posts/<postId>) at this point if the socket it is not connected we will attempt that and then join the blog channel of this particular post.

Conversation

For the Conversations sections we will connect to the socket as soon as possible and join the channels for the visible site posts, while scrolling we will detect visible site posts and join their channel. We keep the currently viewed site and post ids in redux/reader/viewing.

To do

• We currently don't update the comment counts in non conversational feeds like Following Sites](http://calypso.localhost:3000/read), to be fixed in a future PR
• We don't have any visual cues to help the user notice a new comment was added, to be fixed in a future PR
• Check backend new comment filters to make sure we are sending the new comments after they pass the spam filter
• Polling for new posts would show a New Posts sticker which on click would populate the feed with a new duplicated post containing new comments
• Handle comment updates and deletes in realtime

Testing instructions

• Apply patch D40881-code
• Update line 81 in wp-content/mu-plugins/lasagna.php to

'ns'         => 'site' . ':' . $blog_id,

• Sandbox one of your sites, follow it and start a conversation (multiple comments and replies)
• npm install
• Go to http://calypso.localhost:3000/read/conversations
• After you see the conversation go to your site, add another comment
• It should appear in the reader

Edge cases

In development we expose window.SOCKET and window.CHANNELS to help us see what is the current state of our socket and joined channels.

We also have lots of debug logs that can be followed to make sure channel management works correctly, you can enable those by running localStorage.debug = "lasagna:*" in the browser console.

Site channel conditions for join/leave should be tested carefully, we could update the constants for easier testability, decreasing the number of max channel or the lifetime of a channel.

Feed Full Post

• Hard refreshing a feed full post https://d.pr/v/sloTzj
• Navigating to feed full post from the feed list https://d.pr/v/Ic33MK
• Related redux state https://d.pr/v/Nddnc4

Feed Site Post

• Hard refreshing the site full post + leave https://d.pr/v/yjoNoa
• Navigating to site full post from Conversations https://d.pr/v/Dl73J8
• Related redux state https://d.pr/v/YnoZXd

Conversations

When first joining this there is a blip when more posts are visible until all comments load, that is why there is a lot of attempts to join/leave in the console on hard refresh.

• Hard refreshing conversations https://d.pr/v/M7TAS2

[matticbot]

Test live: https://calypso.live/?branch=add/realtime-conversations

[lezama]

Works great! I see comments appear on both sides :) what can I do to debug this better?

After submitting a comment Reader side I see it double for a couple of seconds.

[matticbot]

Here is how your PR affects size of JS and CSS bundles shipped to the user's browser:

Webpack Runtime (~218 bytes removed 📉 [gzipped])

name      parsed_size           gzip_size
manifest      -1818 B  (-1.0%)     -218 B  (-0.7%)

Webpack runtime for loading modules. It is included in the HTML page as an inline script. Is downloaded and parsed every time the app is loaded.

App Entrypoints (~12608 bytes added 📈 [gzipped])

name                 parsed_size           gzip_size
entry-main              +65421 B  (+4.3%)   +12677 B  (+3.3%)
entry-gutenboarding       +120 B  (+0.0%)      -69 B  (-0.0%)

Common code that is always downloaded and parsed every time the app is loaded, no matter which route is used.

Sections (~92183 bytes removed 📉 [gzipped])

name              parsed_size            gzip_size
help                 -63733 B  (-13.4%)   -12465 B  (-10.6%)
concierge            -63733 B  (-22.0%)   -12411 B  (-17.6%)
comments             -63705 B  (-13.6%)   -12036 B  (-11.1%)
stats                -63620 B   (-8.3%)   -11664 B   (-6.1%)
backups              -63620 B  (-19.1%)   -11679 B  (-14.5%)
activity             -63620 B  (-14.6%)   -11652 B  (-10.5%)
settings-writing     -63596 B  (-15.1%)   -11568 B  (-10.8%)
post-editor          -63550 B   (-3.4%)   -10417 B   (-2.0%)
reader                +5428 B   (+1.3%)    +1781 B   (+1.6%)
signup                 -448 B   (-0.3%)      -46 B   (-0.1%)
posts-custom           -117 B   (-0.0%)      -28 B   (-0.0%)
posts                  -117 B   (-0.0%)      -28 B   (-0.0%)
media                   +51 B   (+0.0%)      +20 B   (+0.0%)
settings                +24 B   (+0.0%)       +4 B   (+0.0%)
marketing               +12 B   (+0.0%)       +3 B   (+0.0%)
gutenberg-editor        +12 B   (+0.0%)       +3 B   (+0.0%)

Sections contain code specific for a given set of routes. Is downloaded and parsed only when a particular route is navigated to.

Async-loaded Components (~43246 bytes removed 📉 [gzipped])

name                                             parsed_size            gzip_size
async-load-blocks-inline-help-popover               -63733 B  (-26.7%)   -12465 B  (-21.6%)
async-load-signup-steps-clone-point                 -63620 B  (-40.7%)   -11720 B  (-32.3%)
async-load-blocks-calendar-popover                  -63620 B  (-26.5%)   -11726 B  (-25.1%)
async-load-design-blocks                            -57861 B   (-2.1%)    -9287 B   (-1.4%)
async-load-components-web-preview-component          +5769 B   (+1.5%)    +1736 B   (+1.8%)
async-load-blocks-support-article-dialog-dialog       +845 B   (+1.4%)     +214 B   (+1.2%)
async-load-blocks-inline-help                         -354 B   (-0.6%)      -35 B   (-0.3%)
async-load-blocks-reader-full-post                    +116 B   (+0.3%)      +34 B   (+0.3%)
async-load-post-editor-media-modal                     +12 B   (+0.0%)       +3 B   (+0.0%)

React components that are loaded lazily, when a certain part of UI is displayed for the first time.

Legend

What is parsed and gzip size?

Parsed Size: Uncompressed size of the JS and CSS files. This much code needs to be parsed and stored in memory.
Gzip Size: Compressed size of the JS and CSS files. This much data needs to be downloaded over network.

Generated by performance advisor bot at iscalypsofastyet.com.

[unDemian]

This should be ready for review, unit tests on the way 😟

@hewsut

• for now I replaced the post channels with the more generic blog ones, let me know how you feel about it.
• the fullViewPost redux entity was merged as part of the redux/reader/viewing logic

[unDemian]

The blog manager is hardcoded to public. We're going to need private too. In fact, our first usage is likely to be heavily private (a8c-only streams).

Good point, forgot to add it back.

We seem to have lost the private channel behavior of single join attempt stop on error. When I hacked in private namespacing to the joins and watched the network inspector it kept hammering away on join attempts. I'm not yet sure why. The code looked ok on skim (if error, leave() && delete ref). I think we want it to just stop. Or maybe retry once or twice and then stop if we want to be fancy.

I'm not quite sure where that code was, refreshing the stream will have some attempts because of the flickering of the visible posts on reload but a channel should be actually joined once.

So, circling back here, I'm furiously cleaning up private content handling serverside. I think you'll be really close here if you is_private check the site as the action triggering the join comes in.

Added.

One other tiny request. Could we s/blog/site? It's confusing no matter what we do, heh, but Calypso's UI has shifted toward the latter so I guess it's most forward oriented.

Updated.

[unDemian]

I'm still looking into some performance issues around requesting comments which breaks my A8C Conversations feed. Can you please create a diff that has the wp-site-auth check, thanks

[unDemian]

We seem to have lost the private channel behavior of single join attempt stop on error. When I hacked in private namespacing to the joins and watched the network inspector it kept hammering away on join attempts. I'm not yet sure why. The code looked ok on skim (if error, leave() && delete ref). I think we want it to just stop. Or maybe retry once or twice and then stop if we want to be fancy.

Managed to reproduce, fixed it.

[unDemian]

Should be ready for another round, will investigate the performance issues in another PR.

[unDemian]

Closed in favor of using lasagna.js