Provide unified Decode trait

[duesee]

Due to @FelixPodint's suggestion, I wanted to write down some thoughts on how to improve imap-codec's API.

I do like the idea of having Encode (already implemented), and Decode (to be done).

How should the Decode trait look like?

@FelixPodint's suggested something like ...

pub trait Decode where Self: Sized {
    fn decode(input: &[u8]) -> IResult<&[u8], Self>;
}

... that can be implemented like this ...

impl Decode for Envelope {
    fn decode(input: &[u8]) -> IResult<&[u8], Self> {
        crate::parse::envelope::envelope(input)
    }
}

... and be used like this ...

let rez = match Envelope::decode(input) {
    Ok((remaining, envelope)) => envelope,
    Err(...) => panic!(),
};

Questions:

• 1. What should be the return type?

The above solution would expose nom's IResult to the public API. This can make sense, but I am not sure if it does here. Are there any features of IResult that we need?

I think a better solution would be to use an Outcome, (or ParseResult, Result, ... -- name up to debate), e.g., like this ...

pub struct Outcome<Object> {
    // Parser succeeded.
    Success(&[u8], Object), // remainder, parsed Object

    // Parser needs more data.
    Incomplete(...), // TODO

    // Parser needs more data, and an action is required, e.g., a literal acknowledgement must be send from the server.
    ActionRequired(...), // TODO

    // Parser failed.
    Failed(...), // TODO
}

These are the four cases I am aware of an IMAP server will experience when reading from network. For a more detailled explanation, see the command parsing example.

• 2. What types should implement Decode?

Encode is pretty much implemented by any type in imap-codec, because it is used similar to how Display is used. An implementation of Encode will likely only serialize some required syntax here and there, but mostly arrange how to call further Encode functions.

However, should we really implement Decode for all parsers, too? This would be another layer requiring maintenance. What would be the benefit?

---

I would propose to implement Decode only for the "happy path", that is, for all parsers directly required to produce a stream of IMAP commands or responses, i.e., Greeting, Command, Response, and a small count of other types.

For everything that is not directly relevant to processing a stream of IMAP objects, we would reference to the internal module. Sadely, @FelixPodint, this would mean that Decode would not be implemented for Envelope and you would still need to refer to the internal module.

Other solutions

• imap-proto uses a method Response::from_bytes that returns a ParseResult. ParseResult exposes nom's internals.
• tokio-util has the Decoder trait A real-world implementation should likely use this trait. In fact, imap-codec should provide a Decoder implementation in the future (feature-gated with tokio-codec or so.).

[duesee]

More thoughts on why having Encode and Decode is nice. The documentation could focus on IMAP types, i.e., Command, Response, etc.. When importing Command, the user could explore the API by creating new (possibly complex) commands. In order to serialize them, they would need to import the Encode trait. When encountering Encode, it should be pretty obvious, that there is also a Decode trait.

However, when following the reasoning from above, Decode would not be implemented for all types the user may stumble upon ... This could be confusing.

On the other hand, the API still exposes a lot of types that might not be easy to use, e.g., AString and so on.

[FelixPodint]

I see your point. imap-codec should focus on:

• taking in raw data -> converting to Command || helping with building up a Response -> outputting raw data/encoding.

I started all this because, as I said, I'm trying to build an IMAP server (similar problems would apply to IMAP client). As a server I need to store some data on disk, permanent flags, envelope, body, mailbox and its info, etc... For that I use SQLite database. To build a Response I need to take some data from database and parse it to an imap-codec type, after some searching I noticed that you already did the "parsing to imap-codec type" part. This is why I asked for parse exports.

The problem that I try to solve is "How and what data to store inside my database".

Lets take envelope as an example.

• Currently I'm storing whole envelope in a single column. (I start to think this was a bad idea. This was also the reason why I asked to export parser's.)
• If I store every envelope field in a separate columns, when I would only need to create NString's from returned data, but also Vec<Address> :/
• How do you store Vec<Address> inside SQL database? Separate table with mail_message_id column? (Need to google it).
• Now I need to store all Address fields in separate columns. (Even more columns, this database starts to look complex.)
• Maybe I need to store all mail data in separate columns down to most basic of imap-codec types (AString, IString, NString, etc.), so I can build response up from this data, it would make search Command a little easier, but also make SQL code more complex (maybe not a big problem)?
• I do not even dare to think about message body and all commands/responses associated with it and how much it would expand this database structure :D...

Sorry for rambling. These are my thoughts.
I probably need to rethink my database structure. It's not as easy as to "store mail message inside column and retrieve it when needed".
I start to think that this whole "export parser's" thing was unnecessary.
Or maybe not. Maybe I do take whole envelope from database -> parse it to Envelope type -> create appropriate Response's for fetch/search/etc. commands -> done/send response, discard the rest. Do the same for body. Only problem with body is that it's much bigger than envelope, so storing/retrieving data in pieces would make more sense.
If you understood anything, maybe you can give me some pointers to set me on the right way? If I guess correctly, you have way more experience with e-mails than me.
Thanks.

[duesee]

Thanks for the explanation!

First thought: do you know that all imap-codec types derive serde's Serialize and Deserialize traits? When you use the "serdex" feature in imap-codec, you should be able to serialize and deserialize any type, including Envelope, to JSON, YAML, XML, CBOR, ... :-) I think this is a better idea than to put the IMAP syntax of Envelope into the database and parse it with the envelope function. Maybe you can find an even better solution to use serde and some database crate to do more fancy stuff. (Btw, should serialization of some type not work, I will definitely want to fix it!)

Still, I do like the idea of exposing the "smaller" parsers. Although, I also start to think that we should at least postpone that until there is a really good usecase for it. What I mean with really good usecase:

When I started imap-codec, I basically implemented IMAP's Formal Syntax with nom. While doing so, I noticed that certain ABNF rules are missing, e.g., ALPHA, SP, ... These rules are ABNF's Core Rules and are assumed to be available in all ABNF definitions. Thus, I implemented them in a separate crate, abnf-core.

I see abnf-core as a "building block" for nom parsers that do anything with ABNF. Thus, the API of abnf-core exposes all nom internals, such that the parsers can directly be used by the nom ecosystem, e.g., many1(SP). My idea now was to do the same with the rfc3501 module. This module would expose the "raw" nom parsers in case someone wants to use them. But, thinking about it, this might never happen. At least, I am not aware of other protocols than IMAP, that would use the IMAP ABNF rules. I know that SMTP uses definitions from MIME, but I am not aware of any instance in IMAP currrently.

[FelixPodint]

First thought: do you know that all imap-codec types derive serde's Serialize and Deserialize traits?

HOLY Biscuits!!! Saw the feature but my brain, for some reason, didn't add two and two together. This might lead me to something. Will try next week. Thanks!

[FelixPodint]

I think a better solution would be to use an Outcome, (or ParseResult, Result, ... -- name up to debate)...

100% agree. I used IResult just for a quick example (prof of concept for myself).