Expose RustBuffer.create outside its kotlin package

[mattyg]

I'm running into an issue with external types that are output to separate kotlin packages.

I have 2 uniffi crates: one has just types, and the other has business logic and uses the types.

The types crate and bindings are output to a kotlin library.
The business logic crate and bindings are output to a separate kotlin project, which has the kotlin library as a dependency.

My issue is that the business logic crate bindings give me the following error:

e: file:///.../business_logic_crate.kt:1740:42 Cannot access 'fun create(capacity: ULong, len: ULong, data: Pointer?): RustBuffer.ByValue': it is internal in 'org/types_crate_library/RustBuffer.Companion'.

I'm not sure if this PR is the best approach, but it does resolve the error.

[bendk]

The code looks good, tests would be even better. Could you add a test case for this in the ext-libs crate or just an entirely new fixture?

The code in that crate is pretty messy and it's okay if yours is too. I'm currently working on refactoring the tests and creating a new fixture.

[bendk]

This looks like the right approach to me. It's unfortunate that we have to expose these internal methods, but we haven't found any better way yet.

[mattyg]

Thanks @bendk -- added a simple test to import RustBuffer and call create

[mhammond]

What we'd like is a change to one of our "external types" fixtures, or a new "external types" fixture which demonstrates the problem - we have extensive tests for sharing types between crates, so would like to better understand what you are trying to do which isn't already covered.

To be clear, when you say "crate" above, you really do just mean a crate but and both crates are in the same .so library, right?

[mattyg]

What we'd like is a change to one of our "external types" fixtures, or a new "external types" fixture which demonstrates the problem - we have extensive tests for sharing types between crates, so would like to better understand what you are trying to do which isn't already covered.

Ah okay will do.

To be clear, when you say "crate" above, you really do just mean a crate but and both crates are in the same .so library, right?

Not sure I'm following, but I don't think so. I'm building the types crate and business logic crate as 2 separate .so libraries, and then generating bindings for each. I'll make the example fixture to clarify

[bendk]

Not sure I'm following, but I don't think so. I'm building the types crate and business logic crate as 2 separate .so libraries, and then generating bindings for each. I'll make the example fixture to clarify.

The typical build method is to create a single .so with both crates. I wonder if creating the separate .so libraries and using external types between them is part of the issue.

[mattyg]

Not sure I'm following, but I don't think so. I'm building the types crate and business logic crate as 2 separate .so libraries, and then generating bindings for each. I'll make the example fixture to clarify.

The typical build method is to create a single .so with both crates. I wonder if creating the separate .so libraries and using external types between them is part of the issue.

Yes I suspect so. Is there a different recommended approach for generating types to a separate kotlin library as business logic that uses it?

[bendk]

Yes I suspect so. Is there a different recommended approach for generating types to a separate kotlin library as business logic that uses it?

There's no recommended approach for this, can you help me understand your use-case better? Why do you need 2 separate libraries? I can definitely understand 2 different Rust crates, but then we typically bundle both crates into a single library.

[mattyg]

Yes I suspect so. Is there a different recommended approach for generating types to a separate kotlin library as business logic that uses it?

There's no recommended approach for this, can you help me understand your use-case better? Why do you need 2 separate libraries? I can definitely understand 2 different Rust crates, but then we typically bundle both crates into a single library.

I have a client-server architecture. One rust crate contains types, which are used by both the client & server. The other crate contains the server, which has some functions that use those types for inputs and responses.

The client needs to be published as a standalone kotlin package. It needs the types, but not the generated kotlin code with the server bindings, nor the actual compiled server library (which is quite large).

The server needs the generated kotlin code with the server bindings, the compiled server library, AND the types imported from the standalone kotlin client package.

I hope that is somewhat clearer!

[mhammond]

I think here you'd just have your shared crate be consumed by both the client and server code. ie, the client and the server would have their own shared libraries, which each include a copy of the "types" crate.

The core problem is that Rust doesn't support ABI compatibility - even as I described, the Rust compiler is free to lay out some of the types differently in each library.

[mattyg]

Sorry accidentally closed. I realized I left out some of the complexity of my setup. The client and server are communicating using android's binder IPC which I'm calling via android's Kotlin sdk. So the client and server are actually Kotlin classes that use the generated libraries and bindings under-the-hood.

I think here you'd just have your shared crate be consumed by both the client and server code. ie, the client and the server would have their own shared libraries, which each include a copy of the "types" crate.

This does seem more straightforward in general. Unfortunately for my use case it means needing rust bindings to android's binder IPC library, I'm not sure I want to take on that additional complexity for this project if I can avoid it.

I recognize that this is a strange use case. Would you still be open to merging if I add the text fixture to demonstrate this setup?

[mhammond]

Unfortunately for my use case it means needing rust bindings to android's binder IPC library, I'm not sure I want to take on that additional complexity for this project if I can avoid it.

I don't quite understand this - IIUC, your original idea would result in 3 libs (types, client and server) whereas this would result in 2 - how does that change whether you need the IPC library or not?

Would you still be open to merging if I add the text fixture to demonstrate this setup?

Best I can tell, Rust very explicitly says this (ie, a Rust ABI between dynamic libs) is not supported. Here at Mozilla we'd have a number of problems go away if we could do this in a supported way. So I'd personally be reluctant without more information.

[mattyg]

I don't quite understand this - IIUC, your original idea would result in 3 libs (types, client and server) whereas this would result in 2 - how does that change whether you need the IPC library or not?

Ah sorry I'm using the term "library" vaguely.

There are 2 compiled libraries:

• types
• business logic used by the sever

Then there are 2 kotlin packages:

• client (depends on types library)
• server (depends on client package & business logic library)

The kotlin packages communicate over androids binder IPC.

[bendk]

Let's focus on just the libraries. Could you have a client lib and a server lib that each self-contained and don't require linking to any other Rust libs?

Internally, these libraries could contain shared Rust crates and there are many possibilities here. But in the end you would build a single library for the client and a single library for the server.

[mattyg]

Internally, these libraries could contain shared Rust crates and there are many possibilities here. But in the end you would build a single library for the client and a single library for the server.

Ah, okay I see what you're saying. Yes that is definitely possible. I'll refactor to take that approach. Thanks so much for the guidance! Really appreciate it.