[RFC] Netty: add Netty 4 as a new Netty backend and make it default #25

xabolcs · 2021-01-10T23:39:01Z

For Play! 1 @tomparle did a great work already. It's still work in progress, but it's a great milestone.
I tried my luck for Play! 1, based on his work, but failed too. 😅

Now I tried my luck for RePlay because it's simpler than Play! 1 ... and it looks good. 👍

Ideas, thoughts:

ByteRange and ByteRangeInput (and FileService?) could be simplified IMHO
PlayHandler.LazyChunkedInput also needs more love (Raw use of parameterized class ChunkedInput)
it would be nice if one could add two parameters to PlayHandler:
- for inbound message type: FullHttpRequest
- for outbound message type: FullHttpResponse
- sadly Netty 4 doesn't like public class PlayHandler<I extends FullHttpRequest, O extends FullHttpResponse> extends SimpleChannelInboundHandler<I> { ... }
- there are a few static method too
PlayHandler is too big ... it would be nice to separate the Netty handler parts from the Play! internals
- for e.g. a nice re-introduce of WebSockets.

I didn't tested this in production

Therefore this is just a draft.

Tests were green, both here and at asolntsev / criminals.

ghost · 2021-01-10T23:39:13Z

DeepCode's analysis on #e858dd found:

ℹ️ 6 minor issues. 👇
✔️ 1 issue was fixed.

Top issues

Description	Example fixes
Catching java.lang.Throwable will also catch JVM internal errors like OutOfMemoryErrors or StackOverflowErrors. This is most likely not the intended behavior and considered a bad practice. Consider catching a less general exception. Occurrences: FileService.java:46 FileService.java:70 PlayHandler.java:199	🔧 Example fixes
Use org.apache.commons.lang3.StringUtils.isEmpty instead of org.apache.commons.lang.StringUtils.isEmpty. Occurrences: PlayHandler.java:784	🔧 Example fixes
No abstract method which means that the keyword is most likely used to prevent instantiation. Use a private or protected constructor instead. Occurrences: NettyInvocation.java:5	🔧 Example fixes

👉 View analysis in DeepCode’s Dashboard | Configure the bot

asolntsev · 2021-01-11T08:23:14Z

Just... WOW!

xabolcs · 2021-01-11T16:57:50Z

~~I could split up the commits to help git detect the changes.~~ I can't. 😅

Just use git diff --find-copies=1 master on this branch to show a relevant diff.
Diffstat is:

$ git diff --find-copies=1 master --stat
 framework/build.gradle                                                                          |   1 +
 framework/src/play/Invoker.java                                                                 |   4 +-
 framework/src/play/server/NettyInvocation.java                                                  |   6 +++
 framework/src/play/server/Server.java                                                           |  71 ++----------------------
 framework/src/play/server/{ => netty3}/ByteRange.java                                           |   2 +-
 framework/src/play/server/{ => netty3}/ByteRangeInput.java                                      |   2 +-
 framework/src/play/server/{ => netty3}/FileChannelBuffer.java                                   |   2 +-
 framework/src/play/server/{ => netty3}/FileService.java                                         |   2 +-
 framework/src/play/server/{ => netty3}/FlashPolicyHandler.java                                  |   2 +-
 framework/src/play/server/{ => netty3}/HttpServerPipelineFactory.java                           |   2 +-
 framework/src/play/server/{ => netty3}/PlayHandler.java                                         |  10 ++--
 framework/src/play/server/{ => netty3}/Server.java                                              |   2 +-
 framework/src/play/server/{ => netty3}/StreamChunkAggregator.java                               |   2 +-
 framework/src/play/server/{ => netty4}/ByteRange.java                                           |   2 +-
 framework/src/play/server/{ => netty4}/ByteRangeInput.java                                      |  61 +++++++++++++--------
 framework/src/play/server/{ => netty4}/FileService.java                                         |  63 +++++++++++-----------
 framework/src/play/server/{ => netty4}/FlashPolicyHandler.java                                  |  43 +++++++--------
 framework/src/play/server/{HttpServerPipelineFactory.java => netty4/HttpServerInitializer.java} |  64 +++++++++++-----------
 framework/src/play/server/{ => netty4}/PlayHandler.java                                         | 314 +++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------------------------
 framework/src/play/server/{ => netty4}/Server.java                                              |  57 +++++++++++++-------
 20 files changed, 342 insertions(+), 370 deletions(-)

asolntsev · 2021-01-16T20:28:56Z

@xabolcs Hi! Thank you for trying to solve the hard problem with upgrading to Netty 4.

I have tried your PR and found one problem: serving static resources

Namely, I am trying to open the following URL in a browser:
http://localhost:9126/public/images/logo-bspb.svg

and it stays indefinitely in "loading" state. :(

xabolcs · 2021-01-16T22:45:01Z

I have tried your PR and found one problem: serving static resources

May I ask a test for it? 😉
Here or in criminals.

... and it stays indefinitely in "loading" state. :(

That was the problem for Play! 1 too. Let me try to fix it.

xabolcs · 2021-01-16T23:14:46Z

May I ask a test for it? 😉

There are RouterTest and RouteTest in play.mvc. ~~What~~ How are they testing? 🤔

xabolcs · 2021-01-17T01:43:41Z

Looks like FileService is still not Netty 4 compatible. 😕
Using HttpChunkedInput in FileService.getChunkedInput doesn't help.

Hmmm. @tomparle wanted to get rid of StreamChunkAggregator:

I also tried to get rid of the StreamChunkAggregator class, because not only it seems to require a lot of work to migrate, but also because it may not be necessary anymore if we can use the Netty built-in HttpObjectAggregator handler (see https://netty.io/4.1/api/io/netty/handler/codec/http/HttpObjectAggregator.html).

I'll try to just @Deprecated it, instead dropping. 😀

xabolcs · 2021-01-25T14:53:56Z

At dCache they implemented ReusableChunkedNioFile, which is used by HttpPoolRequestHandler.
Looks promising.

(Btw. I inserted HttpStaticFileServerHandler into RePlay, and works nicely. Something is wrong with RePlay's implementation. 🙃)

cies · 2022-03-15T11:48:20Z

Could a conclusion be added to this issue detailing why the port to netty4 was unsuccessful? This to inform those (who consider) making a similar attempt in the future.

I have questions like:

Is there a fundamental incompatibility?
The work required to complete the transition to Netty4: (1) is too much, (2) is known/unknown, (3) requires very specific expertise, (4) specific to a particular sub-part (like static file serving)?

For us dropping the Netty3 dependency would help as we find some incompatibility with libraries (mostly one for Redis) that uses Netty4 in newer versions. We just use an old version now which works fine, so there is no urgency.

tomparle · 2022-03-15T18:20:27Z

@cies as far as I'm concerned :

there is fundamental incompatibility in a few classes only, mainly PlayHandler
I lacked some expertise to understand the core part of Play with Netty3's code and how to migrate it to Netty 4

I cannot tell a lot more since I'm not (yet?) actually contributing to replay, but still on Play1 codebase so that most users can benefit of our contributions.

xabolcs · 2022-03-21T16:06:56Z

Hi @cies ,

as you can see, this is a draft PR with an [RFC] tag. I filed it because all tests were green and I want to let know the people of this, and to avoid wasting work.

My attempt failed because my knowledge is limited in context of Netty and RePlay.

Is there a fundamental incompatibility?

I don't know yet, but something looks wrong. The Netty3 -> Netty4 server framework change should be transparent to users of RePlay (e.g. asolntsev / criminals), IMHO!

It would be nice if there were end-to-end tests here or in the criminals, to test all the features of RePlay, like "serving static resources".

The work required to complete the transition to Netty4 ...

... is unknown to me.
One should discover all the things in (Re)Play-Netty3 integration and should improve it with this Netty3 -> Netty4 migration in the mind.

If the Netty3 integration is cleaned up, then this Netty3 -> Netty4 upgrade would be easy.

Did I answer your question?

cies · 2022-03-28T12:07:29Z

@xabolcs wrote:

Did I answer your question?

Yes, completely. Thanks.

xabolcs · 2022-04-24T12:57:17Z

and it stays indefinitely in "loading" state. :(

@asolntsev
Updated to latest (4.1.76) Netty and it looks good now.

xabolcs · 2022-04-26T08:06:51Z

~~I could split up the commits to help git detect the changes.~~ I can't. sweat_smile

I'm going to split up the changes to multiple commits:

move current (Netty3 related) play.server packages to play.server.netty3.*
copy them to play.server.netty4
switch play.server.netty4.* packages to Netty 4, make it compilable (but don't care about deprecation)
fix deprecations
switch play.server.Server to use the Netty 4 backend

There are a lot of duplicated code, because of package only visibility. Should I change them to public, like ResettableFileInputStream or I have to keep them as package only e.g. like ByteRange?

Netty 5.0.0.Alpha1 released

To make it easier to experiment with netty 5 while still use 4.1 at the same time we decided to take the effort and put it into a different package. This allows to have both version co-exist at the same time. ...

If my upgrade attempt is stable enough, I would like add Netty 5 too on another branch, but on top of this. ... To make this Netty3 -> Netty 4 upgrade more compact and robust.
Toughts?

asolntsev · 2023-01-10T22:40:16Z

@xabolcs @cies I think I found how to fix it.
Now how it would be better to merge this PR with my fixes?

Probably I will merge this PR and then create next PR with my fixes.

xabolcs · 2023-01-11T06:44:43Z

Now how it would be better to merge this PR with my fixes?

Thanks @asolntsev for finding that haystack!

That checkbox is already checked, so I hope you could add more commit to my branch:

xabolcs · 2023-01-11T06:58:30Z

Probably I will merge this PR and then create next PR with my fixes.

How about splitting commit "Netty: add Netty 4 as a new Netty backend and make it default" to just add Netty 4, but only making default after your fixes?

Just checkout my branch, rebase, add your changes and force push to my branch, or just open a new PR with those rewritten commits.

asolntsev · 2023-01-11T07:25:20Z

checkout my branch, rebase, add your changes and force push to my branch, or just open a new PR with those rewritten commits.

Sounds good. I will try to do that.

xabolcs · 2023-01-11T15:39:11Z

Now I see that the right place to log this error is FileService.operationComplete

Thank you for the hint!
I'll check it tonight!

xabolcs · 2023-01-11T17:17:40Z

I think I found how to fix it.

    18:11:04,361 INFO  ~ served /re-play1/replay-tests/helloworld/app/public/hello_world.txt in 13.790 ms with future DefaultChannelProgressivePromise@69445ae5(failure: io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: PooledUnsafeHeapByteBuf, state: 0)
    18:11:04,362 INFO  ~ future is: cancelled? false, done? true, success? false

So there are two problem at least! 😀

no error handler in FileService 🙈 (see future.isSuccess())
something wrong with the Netty setup (in PlayHandler or in the HttpServerInitializer): Netty throws that io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: PooledUnsafeHeapByteBuf

xabolcs · 2023-01-11T18:56:32Z

something wrong with the Netty setup

No relevant code in the trace 😕

    19:55:23,710 INFO  ~ future is: cancelled? false, done? true, success? false
    19:55:23,710 WARN  ~ An exception was thrown by play.server.netty4.FileService$1.operationComplete()
    play.exceptions.UnexpectedException: io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: PooledUnsafeHeapByteBuf, state: 0
        at play.server.netty4.FileService$1.operationComplete(FileService.java:121)
        at play.server.netty4.FileService$1.operationComplete(FileService.java:105)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
        at io.netty.handler.stream.ChunkedWriteHandler$PendingWrite.fail(ChunkedWriteHandler.java:366)
        at io.netty.handler.stream.ChunkedWriteHandler.handleEndOfInputFuture(ChunkedWriteHandler.java:321)
        at io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:272)
        at io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:131)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:923)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:941)
        at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1247)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
    Caused by: io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: PooledUnsafeHeapByteBuf, state: 0
        at io.netty.handler.codec.http.HttpObjectEncoder.write(HttpObjectEncoder.java:108)
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:940)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:966)
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:934)
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:984)
        at io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:269)
        ... 12 more
    Caused by: java.lang.IllegalStateException: unexpected message type: PooledUnsafeHeapByteBuf, state: 0
        at io.netty.handler.codec.http.HttpObjectEncoder.throwUnexpectedMessageTypeEx(HttpObjectEncoder.java:348)
        at io.netty.handler.codec.http.HttpObjectEncoder.encodeNotHttpMessageContentTypes(HttpObjectEncoder.java:267)
        at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:181)
        at io.netty.handler.codec.http.HttpObjectEncoder.write(HttpObjectEncoder.java:97)
        ... 19 more

xabolcs · 2023-01-11T19:00:08Z

io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: PooledUnsafeHeapByteBuf

🤔

Some related SO:

Netty 4 thows "unexpected message type: io.netty.handler.codec.http.DefaultHttpContent" after chunked file load

xabolcs · 2023-01-11T21:00:34Z

Now I see that the right place to log this error is FileService.operationComplete:

Yup, but before migrating to Netty 4!
Here, in FileService.java#L105:

        if (writeFuture != null) {
            writeFuture.addListener(future -> logger.trace("served {} in {} ms", file, formatNanos(nanoTime() - startedAt)));

... and do the closeSafely dance if future.isSuccess() is false.

asolntsev · 2023-01-11T21:12:03Z

@xabolcs Yes, it seems the problem with ignore future.isSuccess() existed before. :)

xabolcs · 2023-01-11T21:42:40Z

I think I found how to fix it.

@asolntsev
I'm redoing this PR: adding that future.isSuccess() + closeSafely stuff, and splitting my commit.
But please feel free to force-push my branch, as I still don't have any clue why it's failing!

asolntsev

There is still a problem with service static resources, but I will fix it in a new PR.

@xabolcs Sorry, I didn't see your last comment. Already merged the PR. :)

This is an important test as it's failing with Netty 4. See pull request #25.

* users need to add dependency either `replay-netty3` or `replay-netty4` * each module has its own `play.server.Server` class * replay own integration tests run on Netty4 implementation details: * class `NettyInvocation` was not really needed anymore; now we just extend `Invocation`. * I had to find a replacement for `Server.httpPort` which was only used in PDF module. I hope `request.getBase()` is as good as `"http://localhost:" + Server.httpPort`. :)

asolntsev · 2023-04-14T09:24:29Z

@xabolcs @tomparle @cies Hi.
Status update from me:

We still have the problem with rendering static resources in Netty4 implementation. I could not find the reason, but can stably reproduce the problem - see reproduce problem with resources: create a html page with several images #150
I created another implementation (similar to netty3 and netty4) - called javanet. It doesn't use Netty at all. Instead, it uses Java built-in http server (class com.sun.net.httpserver.HttpServer). It passes all the tests, including static resources. I will submit a pull request soon.

UPD Here is the pull request forjavanet implementation: #152

xabolcs · 2023-04-15T21:05:22Z

We still have the problem with rendering static resources in Netty4 implementation. I could not find the reason, but can stably reproduce the problem - see reproduce problem with resources: create a html page with several images #150

You already opened a PR for that, back in January - #115 🙂

asolntsev · 2023-04-15T21:34:44Z

@xabolcs Ah yes, that's right. :) Thank you. :)

asolntsev · 2023-04-18T21:10:18Z

@cies @xabolcs @tomparle Hi.
I have released RePlay 2.0.0.
Now users should include one of 3 engines:

replay-netty3 (this is what was used in RePlay 1.x)
replay-netty4 (uses Netty 4; has problems with static resources)
javanet (uses Java built-in http server instead of Netty, seems to work stably)

Feel free to share you feedback.

tomparle · 2023-04-25T17:50:33Z

Nice work !
Any insight about javanet regarding performances, stability and roadmap comparing to Netty ?
Should we rely more on one implementation between the two ?

asolntsev · 2023-04-25T18:03:46Z

@tomparle Thank you!
No, I didn't do any serious performance tests.

But I am pretty sure that in real projects (like internet-banks etc.), the asynchrony of Netty was not really used.
Anyway, Play/Replay application opened a database transaction for (almost) every request. Most of database drivers nowadays are still synchronous.

So the only case where asynchrony of Netty could have some benefit is serving static resources. But nowadays static resources are usually served by a separate http server (e.g. Ngynx, Amazon or other cloud service) and cached by browser.

So I don't think there should be some remarkable difference in performance.

asolntsev self-assigned this Jan 12, 2021

asolntsev self-requested a review January 12, 2021 19:55

xabolcs force-pushed the branch-netty-4-1-x branch from abc76e2 to e858dd8 Compare January 13, 2021 00:12

xabolcs mentioned this pull request Feb 21, 2022

(WIP) Migration to Netty 4 playframework/play1#1284

Open

xabolcs force-pushed the branch-netty-4-1-x branch from e858dd8 to f8464cd Compare April 23, 2022 12:59

xabolcs mentioned this pull request Apr 23, 2022

b instanceof FileChannelBuffer: never happened in my projects 🤷 #48

Closed

xabolcs force-pushed the branch-netty-4-1-x branch 4 times, most recently from 907fab9 to f613acf Compare April 24, 2022 00:22

xabolcs marked this pull request as ready for review April 24, 2022 12:37

xabolcs changed the title ~~[RFC] Netty: add Netty 4 as a new Netty backend and make it default~~ Netty: add Netty 4 as a new Netty backend and make it default Apr 24, 2022

xabolcs changed the title ~~Netty: add Netty 4 as a new Netty backend and make it default~~ [RFT] Netty: add Netty 4 as a new Netty backend and make it default Apr 24, 2022

xabolcs force-pushed the branch-netty-4-1-x branch from f613acf to 87c0de2 Compare April 24, 2022 20:07

This comment was marked as off-topic.

Sign in to view

xabolcs force-pushed the branch-netty-4-1-x branch from 87c0de2 to 479f2a1 Compare April 25, 2022 07:50

xabolcs force-pushed the branch-netty-4-1-x branch from 9fc4d94 to bb06d20 Compare January 11, 2023 16:54

asolntsev added this to the 1.12.0 milestone Jan 11, 2023

asolntsev marked this pull request as ready for review January 11, 2023 21:50

asolntsev approved these changes Jan 11, 2023

View reviewed changes

asolntsev merged commit f2e617d into replay-framework:master Jan 11, 2023

asolntsev pushed a commit that referenced this pull request Jan 11, 2023

Netty: add static file download test

464928b

This is an important test as it's failing with Netty 4. See pull request #25.

xabolcs deleted the branch-netty-4-1-x branch January 11, 2023 21:52

asolntsev modified the milestones: 1.12.0, 2.0.0 Jan 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC] Netty: add Netty 4 as a new Netty backend and make it default #25

[RFC] Netty: add Netty 4 as a new Netty backend and make it default #25

xabolcs commented Jan 10, 2021 •

edited

Loading

ghost commented Jan 10, 2021 •

edited by ghost

Loading

asolntsev commented Jan 11, 2021

xabolcs commented Jan 11, 2021 •

edited

Loading

asolntsev commented Jan 16, 2021

xabolcs commented Jan 16, 2021 •

edited

Loading

xabolcs commented Jan 16, 2021 •

edited

Loading

xabolcs commented Jan 17, 2021 •

edited

Loading

xabolcs commented Jan 25, 2021

cies commented Mar 15, 2022 •

edited

Loading

tomparle commented Mar 15, 2022 •

edited

Loading

xabolcs commented Mar 21, 2022

cies commented Mar 28, 2022 •

edited

Loading

xabolcs commented Apr 24, 2022

This comment was marked as off-topic.

xabolcs commented Apr 26, 2022 •

edited

Loading

asolntsev commented Jan 10, 2023

xabolcs commented Jan 11, 2023 •

edited

Loading

xabolcs commented Jan 11, 2023

asolntsev commented Jan 11, 2023

xabolcs commented Jan 11, 2023

xabolcs commented Jan 11, 2023

xabolcs commented Jan 11, 2023

xabolcs commented Jan 11, 2023

xabolcs commented Jan 11, 2023

asolntsev commented Jan 11, 2023

xabolcs commented Jan 11, 2023

asolntsev left a comment •

edited

Loading

asolntsev commented Apr 14, 2023 •

edited

Loading

xabolcs commented Apr 15, 2023

asolntsev commented Apr 15, 2023

asolntsev commented Apr 18, 2023

tomparle commented Apr 25, 2023

asolntsev commented Apr 25, 2023 •

edited

Loading

[RFC] Netty: add Netty 4 as a new Netty backend and make it default #25

[RFC] Netty: add Netty 4 as a new Netty backend and make it default #25

Conversation

xabolcs commented Jan 10, 2021 • edited Loading

I didn't tested this in production

ghost commented Jan 10, 2021 • edited by ghost Loading

DeepCode's analysis on #e858dd found:

Top issues

👉 View analysis in DeepCode’s Dashboard | Configure the bot

asolntsev commented Jan 11, 2021

xabolcs commented Jan 11, 2021 • edited Loading

asolntsev commented Jan 16, 2021

xabolcs commented Jan 16, 2021 • edited Loading

xabolcs commented Jan 16, 2021 • edited Loading

xabolcs commented Jan 17, 2021 • edited Loading

xabolcs commented Jan 25, 2021

cies commented Mar 15, 2022 • edited Loading

tomparle commented Mar 15, 2022 • edited Loading

xabolcs commented Mar 21, 2022

cies commented Mar 28, 2022 • edited Loading

xabolcs commented Apr 24, 2022

This comment was marked as off-topic.

xabolcs commented Apr 26, 2022 • edited Loading

asolntsev commented Jan 10, 2023

xabolcs commented Jan 11, 2023 • edited Loading

xabolcs commented Jan 11, 2023

asolntsev commented Jan 11, 2023

xabolcs commented Jan 11, 2023

xabolcs commented Jan 11, 2023

xabolcs commented Jan 11, 2023

xabolcs commented Jan 11, 2023

xabolcs commented Jan 11, 2023

asolntsev commented Jan 11, 2023

xabolcs commented Jan 11, 2023

asolntsev left a comment • edited Loading

Choose a reason for hiding this comment

asolntsev commented Apr 14, 2023 • edited Loading

xabolcs commented Apr 15, 2023

asolntsev commented Apr 15, 2023

asolntsev commented Apr 18, 2023

tomparle commented Apr 25, 2023

asolntsev commented Apr 25, 2023 • edited Loading

xabolcs commented Jan 10, 2021 •

edited

Loading

ghost commented Jan 10, 2021 •

edited by ghost

Loading

xabolcs commented Jan 11, 2021 •

edited

Loading

xabolcs commented Jan 16, 2021 •

edited

Loading

xabolcs commented Jan 16, 2021 •

edited

Loading

xabolcs commented Jan 17, 2021 •

edited

Loading

cies commented Mar 15, 2022 •

edited

Loading

tomparle commented Mar 15, 2022 •

edited

Loading

cies commented Mar 28, 2022 •

edited

Loading

xabolcs commented Apr 26, 2022 •

edited

Loading

xabolcs commented Jan 11, 2023 •

edited

Loading

asolntsev left a comment •

edited

Loading

asolntsev commented Apr 14, 2023 •

edited

Loading

asolntsev commented Apr 25, 2023 •

edited

Loading