[discussion] Java Async

[andreaTP]

I think it's worth having this conversation even just to keep track of the reasoning behind the decisions.
When I looked at the codebase, at first, I was pretty impressed by the fact that we are using async calls everywhere for Java.

This is a subject pretty much full of nuances in Java land as far as I know it.

Let's try to summarize a few key points:

• CompletableFuture API is not great and doesn't play well with most of the rest of the standard library, also, using it makes the code much much harder to understand and follow (e.g. we miss an async/await construct)
• Reactive programming has become quite popular in Java, and exposing a CompletableFuture interface is going to play nicely with all the frameworks adopting this paradigm (Akka, Vert-X etc. etc.)
• Most of the libraries around are exposing still a blocking API(e.g. OkHttp), and, wrapping every call in CompletableFutures might not be the ideal solution given the number of options that comes (scheduling on which Thread Pool? Keep separated IO blocking operations from in-process computing etc. etc.)
• "Idiomatic" Java 8 code (as per Android compatibility) is not using the async API and it doesn't play nicely with imperative and blocking code
• Project Loom is coming with the promise to enable blocking idioms to easily work in contexts with concurrency and parallelism (since here we are keeping compatibility with Android I'm not sure about timelines here)

[baywet]

thanks for bringing this up. A lot of the thinking comes from the Microsoft Graph SDK work (pre-kiota) and was kind of recaped here and there.

Long story short; completable future seemed to be the best compromise at the time, only raising API level to 24 (we rose to 26 because of the DateTime and other things we needed). It's not ideal, but its the one with the broader compatibility and broader support.
API level 26 is now supported by 93% of devices (compared with 91% in late 2020), and 24 96.4%. Over time the gains of investing into a broader compatibility solution will naturally reduce.
As per investing into a better developer experience solution, we should demonstrate a significant improvement before doing so. I wouldn't be surprised if Fibers come up with helper functions to wrap completable future to ease up the migration path.

Lastly, we're currently wrapping the call using enqueue which OkHttp will defer to another thread or not depending on whether other requests are being executed, and depending on the dispatcher. That gives full control to the caller (they can implement/swap their dispatcher) rather than implementing our own custom logic.

[andreaTP]

Thanks a lot for the context @baywet , I think that your answer perfectly recaps the reasoning behind the current decisions/implementation 👍

[baywet]

do you think further discussion is needed or should we close this issue at this time?

[andreaTP]

Perfectly fine closing this, thanks for taking the time to share the background.

[andreaTP]

We briefly touched on this point during the community meeting today.

Reopening to gather additional feedback from @papegaaij and @mikrethor 🙏

[baywet]

For additional context we have one precedent of a language not being async: go. There is no primitive for that in go because usually defer the routines at the highest level.
I'm guessing that if we took away the async API surface, people could always do it similarly to go (in this case wrapping the call in a future or other construct themselves).
Do we have any visibility over a potential async/await set of keywords equivalent in Java being worked on? or the reason why this might have been declined? (hopefully we're not talking about compatibility again)

[andreaTP]

Do we have any visibility over a potential async/await set of keywords equivalent in Java being worked on? or the reason why this might have been declined? (hopefully we're not talking about compatibility again)

In Java land, there is an exciting turnaround with Project Loom and part of it is going to land in Java 21 (going to be released GA in September).
To play well with Loom execution model we should only have blocking calls 🙂

It would be interesting to do a quick experiment to see how much we can generalize some utility methods in an external "extension" to wrap the most relevant calls with async.

[papegaaij]

I don't think you can actually wrap a synchronous call in an async utility. For true async, you need to be async all the way down. I don't know how good the async implementation for OkHttp actually is, but in theory a good http client using Http 2 could multiplex multiple http requests in parallel over a limited set of connections using only a few threads. There is no way to get this if the SDK performs the calls sync.

That being said, very few people actually use async in Java and using it "the right way" is very hard. Many SDKs in Java provide both options. Many Java developers really prefer a sync API. A suggestion would be something like get and getAsync. This is very common in many Java SDKs and should be possible in the current request builders.

[baywet]

The current Microsoft Graph SDK offers both API surfaces for the last couple of years. However it's built in a different way (using our older generator) and we don't have telemetry over which API surface people are using. Our current plan for the Microsoft Graph Java SDK is to release a preview based of kiota and see what kind of feedback we get.

As per okHttp, we're calling enqueue which in turns calls enqueue on the dispatcher.
As far as I understand kotlin the dispatcher then uses a combination of thread pools and queues to defer the call to a different thread.
This is not "truly" async but between that and the fact we wrap the callback to a Future, it's close enough.

[papegaaij]

These pools are one of the major issues with using async APIs. We've had several major production outages to some of our other applications due to pools getting exhausted and the whole thing deadlocking.

We use the current Graph SDK in Topicus KeyHub and we use the sync API as do other applications in our company AFAIK.

[andreaTP]

Maybe someone like @brunoborges (on Microsoft side) and @Sanne (on RH) are interested in weighting in on this decision before we go GA 🙏

[JonathanGiles]

Hi all - I'm over on the Azure SDK side of the Microsoft house. We've been building client libraries since 2018, and have a fairly large amount of experience in this area. We did a lot of user testing and investigations back when we started, and we ended up using Project Reactor for our async story, but we ship sync and async clients side-by-side (in the same library). We have full sync and full async stacks. This allows developers to choose to use sync or async, as their needs dictate. Whilst I have zero evidence of this, my gut feeling is that most developers choose to use our sync stack implementation, for its simplicity and debuggability. I imagine only our largest customers bother with the complexities of async, to eek out the final few percentage points worth of performance.

Obviously when we started the concept of virtual threads was still a long way off, and even then, it feels too low-level of an abstraction to give to developers, as it would be asking them to manage threads (and, realistically, block on them). Similarly, CompletableFuture is good in concept for many use cases, but in some places (in particular, streaming cases), it falls down. This is why we ended up going with Reactor for our async story - it gave a complete story for developers to understand, for all async cases.

Looking ahead to the future, I've done some early thinking about how I would do things today, and it would probably be a custom async API that wasn't based on a third party dependency. This is because Reactor does break more often than we would like, both in terms of implementation and public-facing API. If I had more time back in the 2018 time frame, I would probably have dived more into this, but as it stands I've only got high level thoughts (and a lot of investigations I would like to do).

I should also add, it sounds like you are working on a number of problems that we have already worked through in our azure-core library. I do wonder if there is any benefit to you making use of our libraries, or finding better ways to collaborate, rather than duplicate a lot of effort?

[andreaTP]

@JonathanGiles thanks a lot for chipping into this!
I'm not familiar with the graph API, but, if you are shipping artifacts through Sonatype/Maven Central you should have easy access to the download statistics.
We should not trust the absolute numbers, but the delta in between the two distributions might give us an "objective enough" data point if available.

Re: Virtual Threads, they are not such a low level abstraction in the Loom incarnation ...

[baywet]

Interestingly enough, project loom seems to be aligning on futures as well with the executor pattern. The main difference with the current solution being the future is instantiated at the highest level over a sync API surface (where we instantiate the Future as close as possible to the actual HTTP request). This pattern looks a little like go co-routines.
While the example doesn't show it, I wonder whether this is designed to also accept Futures.

As to providing both APIs (sync and async) this is going to be a challenge in terms of binary size for large APIs (like Microsoft Graph), hence the necessity to align on a single API surface. The current Java SDK (previous generator) provides both but it has serious limitations in terms of which endpoints/operations are supported, reducing the size, which is already a concern. And no, we don't have telemetry over who's using the sync vs async API today.

[papegaaij]

Do you really think this will make that much of a difference in jar size? Looking at the code, it should be enough to duplicate the method pairs like 'get' and 'post'. A quick test gave an increase in size of about 600 bytes per operation in the class file, which would be less after compression in the jar. I doubt it would be more than 1 or 2% in the total size for an API like the graph API.

[baywet]

Microsoft graph v1 has about 20k operations last time I counted and beta around 35k. At that scale those seemingly "tiny" changes make a substantial difference.

[baywet]

I've asked @maisarissi, our Java SDK PM, to engage with existing customers so we understand better what usage of the API surface they are doing today. But the feedback here helps a lot! Thanks everyone!

[papegaaij]

I checked some numbers, and the graph API indeed has about 20k operations. All class files together are about 100MB, the resulting jar is about 42MB. I've then taken one of the generated request builders and modified the source in 2 steps:

• The get() method now calls get(null) rather than duplicate the body. This saves about 180 bytes per operation in class files.
• Both get(...) methods are duplicated to getAsync, with getAsync() calling getAsync(null). The get(...) were made sync by calling async(...).get() and the try-catch was modified to catch and ignore all exceptions, That's not very nice, but this is just a test. The real version should probably use a RuntimeException. This added about 740 bytes.

For serviceprincipals/delta/DeltaRequestBuilder.class this resulted in a class that went from 6312 to 6871 bytes, a difference of 559 bytes for one operation. This will probably result in about 11MB more class files and a jar that's about 4.3MB larger. It is however likely that it is possible to save some bytes by moving the exception handling to the abstraction layer.

Some observations I made: moving common code to a new method often increases the size of the class file. Exception handling (and throwing) takes a lot of bytes. toGetRequestInformation(...) declares to throw URISyntaxException, but I can't find any place where it actually does. Removing that exception saved over 400 bytes.

In my opinion, if 46MB is too big (and this can probably be brought back to about 43-44MB), 42MB also is. I think the solution would be to split the jar, not to try to save space on not providing both sync and async.

[baywet]

Thanks for leading that investigation! We should probably fork off to another thread to focus on size optimization.

The baseline for the previous generation of SDK is 11MB (5.66), but it supports far fewer endpoints (about 3 times fewer).

Mapping overloads with fewer parameters to their "complete" equivalents with default values instead: very good idea. I'm not sure why we did it for request generator methods and not for request executor methods. Let me create an issue after I post that reply.

UriSyntaxException. It all comes from here, I didn't realize exceptions were so costly in Java.

kiota-java/components/abstractions/src/main/java/com/microsoft/kiota/RequestInformation.java

Line 52 in 1a1fc24

public URI getUri() throws URISyntaxException,IllegalStateException{

We should probably review that method to wrap these specific exceptions in runtime exception (bad practice I know) and simplify a great deal the generated code. (let me create another issue for that)

Splitting the SDK: in the case of Microsoft Graph there's no good split we could assume for the customers, because it's a Graph and we can't assume in advance what endpoints/models/properties people are going to need. The functional boundaries are inexistent as well. Best case scenario we'd end up with a lot of duplication and mapping code on the application side. Worst case we'd end up with gaps in the API surface area. This is actually one of the requirements that got kiota started! :-) The long term play would be to have people choose between an SDK (easy and ready to go) or generating their own client for Microsoft Graph (tailored to their needs, lower footprint). But on the Microsoft Graph we still have infrastructure and tooling to allow that new experience to be seamless.

[andreaTP]

Looking again at this discussion with fresh eyes and I completely understand that this decision is completely dominated by the fact that we are excluding "a-priori" supporting both sync and async.
I would like to challenge this requirement:

• in the microsoft/kiota project generating code with and without CompletableFuture wrapper should be trivial, and, it should be possible to keep the logic for the switch pretty much isolated.
• in this repo, most of the logic can be shared again, and we only need a thin layer to expose the 2 separate APIs

All in all, this seems to be:

• low effort
• easy to swap in/out in case a different decision has been taken

A possible approach here might be:

• add the possibility to generate both sync/async and mark it as "experimental"
• (not mandatory) delay a little going to GA with Java
• start to check and track the usage of the variants based on the "Unique IPs" downloads of the core libraries
• re-iterate

I can see that, having to decide upfront to limit the flags in Kiota, seems to generate lengthy conversations and opinion-sharing other than actual "data-based" directions.
Thoughts?

[baywet]

Thanks for the additional input here.

I'm all for data driven decisions. The challenge we've faced with questions that are so specific is that answering to them based only on data from our own telemetry/experiments is lengthy at best, often times inaccurate:

• we could add telemetry for that specific point in the current SDK which offers both API surfaces today. But we'd need to allow some time (typically between 30 to 90 days) for people to pick up the update and run with that in production so we can collect enough sample data. Even with that the data would be biased towards sync since it was the first API surface offered, it's the API surface present in the docs, and most people just grab the update without changing their code.
• we could experiment by putting a preview version out there with both API surfaces and adding telemetry. The data would probably be more accurate (only new projects) but getting a meaningful sample would take much longer (less than 1% of aps try preview builds if a stable release is available, and because most of them are net new apps, it takes a while for them to reach production)
• experimenting by putting two libs out there, a sync and an async is pretty much going to yield the same results as the previous one. We'll probably incur a drop from the setup overhead though.

This is why for questions that are this specific we usually rely on readily industry data (yes that includes experts like yourselves opinions), interview with customers, experience from other teams at Microsoft...
The challenge with Java is there doesn't seem to be a clear line in the sand. Not only there's no clear consensus between async (like Task in dotnet) or sync (like Go), there are also multiple flavors of async.

The fact that server frameworks like spring, ORMs like hibernate and reactive libs like project reactor all integrate with Futures strongly leads me to think the best way forward will be future based. This will cover scenarios like "inbound request, query to an API, query to a database" non-blocking all the way.

People who really need to block will be able to do so over the completablefuture. and one of the added benefit is to be able to cancel requests, which a sync API makes much more difficult.

Right now we'll probably focus on the size improvements before we come back to this topic, which should give time to @maisarissi to interview with customers and give us some additional feedback.

[Sanne]

Hello all again,

apologies I'm definitely not an expert about kiota, but since I was asked to shed some light on our experience I feel the need to clarify some points here. I hope this migh e useful for the greater good but feel free to ignore me of course :)

I'm all for data driven decisions. The challenge we've faced with questions that are so specific is that answering to them based only on data from our own telemetry/experiments is lengthy at best, often times inaccurate:

It's really great that you all are attempting to make data driven decision, but please allow me to warn about the metrics that I'm seeing being called out here (several posts earlier too): they are fluctuating over time, and most old data shouldn't be taken into account. For example - the fact that the vast majority of artifacts in Maven Central is using "javax." prefix rather than "jakarta." prefix is totally expected as "javax." has been a pillar of integration among libraries for 20+ years. But most such libraries are out of date and abandoned; I'm fully confident that it won't be in the future, so please try to sample data by taking trends into account, and also that while many libraries in the Java ecosystem are fully committed into migrating to jakarta, several haven't done so yet as it's a slow ongoing process which will take years: the existing API is very well ingrained, although also totally not future proof for legal reasons (I hope you're aware: the package name is trademarked so it can't be evolved further).

Also to bear in mind: some popular libraries, for example the Java security APIs, are not part of the "enterprise libraries" but part of the base platform, so in those cases they will retain the javax. prefix as they part of the Java API. That's also expected, so you won't see any transition to a new naming strategy in this important area, and needless to say many such packages need to integrate with those APIs (on top, or exclusively).

The fact that server frameworks like spring, ORMs like hibernate and reactive libs like project reactor all integrate with Futures strongly leads me to think the best way forward will be future based. This will cover scenarios like "inbound request, query to an API, query to a database" non-blocking all the way.

I can't speak for the other libraries, but I'm the lead architect of Hibernate and designed the API you're liking to here; might be useful to bear in mind that we offer two "reactive" APIs: one based on the CompletionStage and one based on the more modern Mutiny library as described in this section.

The rationale at the time (4+ years ago) was that we wished to make it easy for users to integrate with various other reactive libraries, and the CompletionStage system, since it's being offered by the Java standard libraries, felt like an essential integration to have, while Mutiny was looking promising but was a brand new bet.

Nowadays, I regret integrating with CompletionStage as it has been extemely tricky to handle correctly and we regularly received feedback from users shooting themselves in the foot as they don't always understand all nuances of this very complex API.
Mutiny on the other hand has proven itself as a reliable choice and offers robust integrations with several other reactive types, so it alone would satisfy our requirement of easy integration with all such popular systems; of course I would have preferred there was a single type system so that people wouldn't even need to invoke a tiny type conversion layer, however this seems like the best solution available today.

However, basing one's integration on CompletableFuture is very tricky as the futures might (or might not) be executed right away, implying it's very hard to control several important semantics of asynchronous operations, I would suggest to stay clear of it unless your needs are extremely basic cc/ @jponge

HTH

[baywet]

Thank you for joining the discussion. This kind of library maintainer feedback is gold!

Yes, the caller does not have control over the scheduling of the work. This is because at the end of the day we end up calling enqueue from OkHttp, and OkHttp itself does the scheduling on a thread pool.

kiota-java/components/http/okHttp/src/main/java/com/microsoft/kiota/http/OkHttpRequestAdapter.java

Line 660 in 0886948

this.client.newCall(getRequestFromRequestInformation(requestInfo, span, spanForAttributes)).enqueue(wrapper);

We might be able to control the execution by implementing our own dispatcher and/or executor service here and make it so the request execution only begins when the chain of Futures is being started and/or asked for a result. But I don't think scheduling here was the primary goal, offering non-blocking APIs (for the main thread) was.

As for Munity it seems to be mixing two concerns: async programing and dataset streaming, like a lot of the afore-mentioned options. This later capability is not something kiota clients will need as in most cases API results are either objects or small collections. Additionally for large collections, most Java JSON serialization libraries lack the support of data streaming deserialization as far as I understand, so streaming the items coming from a big memory block would not help performance wise.

All this feedback, and those options, make a great base for discussion with customers for @maisarissi while we make good progress on the size optimization aspects. Keep it coming.

[papegaaij]

One of the arguments I haven't seen in this discussion so far is the reliability and scalability of an async API when used in a synchronous way (i.e. calling get() right away). As I've mentioned before, in our applications we almost exclusively use APIs in a synchronous way (we block the thread that's performing the external call). Our applications are designed to deal with this in a reliable way, with a high number of task threads, circuit breakers and rather aggressive timeouts.

We do however, occasionally need to use an SDK that only provides an async API (like what's kiota offering at the moment). These have turned out to be really problematic because they often require careful management of a separate thread pool. Let's say the application has 500 task threads. Every task might need to use the async API at some point. Often, these APIs by default use really small thread pools (like 10), which will then be a major bottleneck (this has even resulted in outage). However, you cannot make the thread pool too large either, as threads are expensive to create and keep around. Also, in many cases it's not a good idea to share thread pool between different API. In the end, you are wasting a large amount of resources on threads, spread over many different pools, that are idle most of the time and performance is still suffering when a large amount of concurrent requests needs to be processed because that particular pool is not large enough.

[baywet]

Update: the code reduction improvement yielded marginal gains, we could still implement a third one, but we're not expecting much bigger gains either. So unless we can identify more size reduction opportunities, the questions we should answer are most likely sync OR async API? (not both) (in addition to selecting the right async "framework").

@maisarissi is out for personal reasons for the next couple of weeks, maybe talking to existing Java customers is something @sebastienlevert could take over to understand whether they are writing async code, why or why not, and which framework do they employ?

[papegaaij]

What about a totally different solution: providing both sync and async behavior via the same API? It should not be that difficult to change the implementation to return completed CompletableFutures. This will certainly be a compromise, mostly on the sync part, because the caller still needs to 'get()' the result, but at least it prevents the problems associated with having to deal with a separate thread pool.

[baywet]

are you suggesting that all layer are sync, and we only have 2 executor methods, one sync (default) and one async that calls the sync one wrapping things in a CompletableFuture?

[Sanne]

When having both blocking (sync) and asynchronous capabilities in the same stack, we try really hard to avoid the two needing to interact: every time an asynchronous operation needs to invoke a blocking one it's a headache of integration and design issues, and every time a blocking method needs to invoke an asynchronous one it needs to, at very least, switch the physical carrier thread which comes at a significant cost of performance.

I wouldn't recommend that.

The one reason in other frameworks we exposed "natively asynch" APIs is precisly for these reasons; if you think you'd better be off with a very thin adaptation layer, I'd recommend rather expose only one of them and let the users deal with it explicitly - it would be a better experience as they would be aware and responsible for how exactly the translation layer works.

[papegaaij]

@baywet I'm suggesting that you can flip the http abstraction layer in sync-mode, so it will do the http call in the caller thread and simply return CompletableFuture.completedFuture(value). This will make the CompletableFuture nothing more than a wrapper for the actual value, without all the async behavior attached.

The main problem I'm having with an async API in an application that's designed to use sync behavior is the need for a separate thread pool to handle the async calls. If too small, you will get significant congestion on these calls (even up to the point of deadlocks), if too big, they will waste resources (even up to the point of OutOfMemoryError). We've seen both in our applications.

[baywet]

@Sanne yep I was aware of the kind of issues sync over async creates. And async over sync doesn't provide much value when compared to only sync.

@papegaaij I don't think returning a completed future over a sync call would provide much value. And we've already ruled out providing both APIs due to a size constraint/duplicating the pipelines.

The decision is sync XOR async API. And if we maintain async (with completable futures or something else) we should make sure it plays nicely with the HTTP client and its thread pool scheduler (if it uses any).

Another aspect we discussed when looking at this issue if we switch to a sync API surface, we'd probably have to declare the throws of the custom exceptions generated from the description.

[EricWittmann]

If the actual async behavior is being handled in the OK layer (using a thread pool maintained by OK), then does that mean users/community can solve the problem @papegaaij has by contributing their own HTTP client layer? I haven't dug into the library the way that @andreaTP has, but I know that was discussed. We were interested because we wanted to e.g. provide a Quarkus specific HTTP client implementation.

But if we can do that, then someone could presumably implement the approach @papegaaij suggested, by providing an HTTP client that actually does the call synchronously and then just wraps the result in an already-completed future.

Personally, as a consumer of the generated SDKs, I would prefer a synchronous API because that's just easier to use and debug. And for the use-cases I care about, client scalability isn't really much of an issue. But I understand why async is being used now - it's probably only possible to practically adapt async->sync, not the other way around: as mentioned, adapting a sync API to async doesn't really accomplish the point of async for scalability - it needs to be actually non-blocking/async all the way down.

[andreaTP]

We briefly touched on the subject another couple of times, I think that my mind is settled now, on 2 opposite directions 🙂 :

• from the "theoretical" point of view(threads control etc.) the async implementation is by far preferable
• from the pragmatic point of view, using async code in a sync codebase(which is the majority according to any data I've ever collected), is extremely unpractical and doesn't play well with most of the constructs (exceptions are always wrapped, controlling execution in lambdas is challenging, the "Functional interface" APIs are mostly unusable) and it's likely to force the introduction of breaking changes in public APIs

[papegaaij]

And I think it is safe to assume that with the introduction of virtual threads in Java 21, async will become more of niche. Virtual threads bring most of the benefits of async without the downsides of unreadable and impossible to debug code.

Go has adopted virtual threads from the start and uses blocking sync APIs everywhere (at least that's what I see in the code I work with). Threads (or goroutines) are cheap and you can spawn as many as you like. There's no problem in having lots of them blocking in a sync call. I wouldn't be surprised if Java is heading the same way when more application servers and frameworks start supporting virtual threads.

[maisarissi]

Hello everyone! Thanks for all the inputs and discussion here! Things like this is what help us to make the best product and better decisions.

After some considerations like the introduction of virtual thread in Java 21 and conversations with @JonathanGiles @brunoborges and internally on Graph DevX team, we believe that sync only is the best way to go. With that, we can improve debugging capabilities as well as decrease complexity on development.
We are building on top of @JonathanGiles work that have been working with Java community and got feedback like this https://twitter.com/JonathanGiles/status/1711583034007564484 where most developers believe sync only is also the way to go. This also gives people the possibility to wrap calls in their own preferred async way and gives us the possibility to provide our own wrappers in the future, if we want to.

[baywet]

Thank you @maisarissi for going the extra mile here and getting both quantitative data, and experts input in addition to the amazing feedback we got here.

@ramsessanchez please go ahead and clean up completable futures from the API surface here, also remove any async suffix we might have + make the relevant generation change (kiota/snippets/raptor).