lnrpc+lncli: add send support for custom data records

[joostjager]

With the introduction of custom record sending and receiving, it became possible to attach arbitrary data to a payment. One obvious use case is attaching a human-readable message to a payment. Especially in the case of spontaneous key send payments, this can give the receiver some context on the payment.

For example: tipping. Usually people sending a tip would want to include some information on who they are or what the tip is for. For Lightning this may be even more desired than for other payment methods, because payments are anonymous by default.

Attaching and retrieving the message was already possible before, but this PR makes it easier to do so by introducing an additional lncli flag --data to attach one or more custom records:

lncli sendpayment -d 0274e7fb33eafd74fe1acb6db7680bb4aa78e9c839a6e954e38abfad680f645ef7 -a 100 --key_send --data 323442=00,3234556=ffff080812

To specify a string value, the standard command line tool xxd can be used (the example record id here is the 3-byte ascii string 'tip' converted to an integer):

--data 7629168=$(echo -n "Thank you!" | xxd -pu -c 10000)

(The -c parameter is to prevent xxd from inserting line breaks)

Note: The available onion blob space of 1300 bytes is used for routing info and custom records. The bigger the size of the custom records, the fewer bytes remain for routing info and the shorter the maximum route length will be.

[Roasbeef]

I thought this was intended to allow users to set arbitrary kv tlv pairs on the command line rather than carve out a new record for messages.

[cfromknecht]

What is probably the most important accomplishment of this PR however is that it makes an attempt to standardize an unofficial field for payment messages.

I'm not convinced this should happen within lnd until there is some actual standardization, especially a field that knowingly collides with whatsat.

[joostjager]

Message record id updated to a different random number.

My thought was that if we don't pick a field and provide a few hooks to make it easy to use (string flag on lncli and string type invoice htlc field), payment messages are less likely to be used. The payment message functionality can serve as a demonstrator of everything else that is possible using custom records. it seemed nice to me to get people thinking about that.

But as @Roasbeef suggests, we can also generalize this to just a new lncli flag. The question though is what a good format is. Any thoughts on that?

lncli sendpayment --data=34378273:"hello world!",124343:0feab32827cba23

It gets complicated with the spaces and special chars. Also not sure what different shells do there. In my bash, at least the data argument comes through as one block, but the quotes have disappeared.

[joostjager]

Removed hard-coded record and replaced by generic --data flag.

[cfromknecht]

LGTM 🏈

[Roasbeef]

LGTM 🎄

Very simple yet powerful PR!

[curly60e]

With the introduction of custom record sending and receiving, it became possible to attach arbitrary data to a payment. One obvious use case is attaching a human-readable message to a payment. Especially in the case of spontaneous key send payments, this can give the receiver some context on the payment.

For example: tipping. Usually people sending a tip would want to include some information on who they are or what the tip is for. For Lightning this may be even more desired than for other payment methods, because payments are anonymous by default.

Attaching and retrieving the message was already possible before, but this PR makes it easier to do so by introducing an additional lncli flag --data to attach one or more custom records:

lncli sendpayment -d 0274e7fb33eafd74fe1acb6db7680bb4aa78e9c839a6e954e38abfad680f645ef7 -a 100 --key_send --data 323442=00,3234556=ffff080812

To specify a string value, the standard command line tool xxd can be used (the example record id here is the 3-byte ascii string 'tip' converted to an integer):

--data 7629168=$(echo -n "Thank you!" | xxd -pu -c 10000)

(The -c parameter is to prevent xxd from inserting line breaks)

Note: The available onion blob space of 1300 bytes is used for routing info and custom records. The bigger the size of the custom records, the fewer bytes remain for routing info and the shorter the maximum route length will...

This is the only method to send a message over lightning?

There is a short command method like a standard?

[cfromknecht]

Currently yes, there is no standardized way of sending messages apart from these custom data records. Note that this require sender and receiver to be aware of the record type and format. There isn't any BOLT standard on how to do so, though some folks have implemented their own protocols on top of this primitive. Personally I wouldn't expect to see a messaging standard integrated into LND any time soon given now nascent these higher-level specifications are.

[drmartinberger]

Dear lnd developers @Roasbeef @cfromknecht,

I am investigating custom data records attached to a Lightning Payment as should be available in lnd with this PR #3900. Thank you for that!

@joostjager was mentioning:

Note: The available onion blob space of 1300 bytes is used for routing info and custom records. The bigger the size of the custom records, the fewer bytes remain for routing info and the shorter the maximum route length will be.

Do you have an idea or experience, how many bytes can be consumed by the custom_records field and still getting a routeable onion payment? Do you have an estimate x if one wants to route a payment with custom data record via y hops, there usually remain 1300/y*x bytes for the data?

Can you point me to any sources to learn more on that? Any other way to transport encrypted custom data with a lightning payment, maybe compatible with other lightning implementations?

Thank you!

Cheers, Martin

[bitromortac]

Perhaps you could constrain your route length to about the diameter of the network, lncli getnetworkinfo shows me a value of 12, or a bit less, which should be plenty. Here some comment about the maximal number of hops in an onion.

[drmartinberger]

Perhaps you could constrain your route length to about the diameter of the network, lncli getnetworkinfo shows me a value of 12, or a bit less, which should be plenty. Here some comment about the maximal number of hops in an onion.

Thanks @bitromortac for your hint. If I understand your mentioned comment correctly, in a route with x non-exist hops there should remain at most 1300-37-x*65 bytes for a custom data record (assuming larger legacy hop payloads), right?

[bitromortac]

Right, but you can assume a base data size of 47 bytes as legacy onions will be phased out soon. If no other data is transported in there this should be the limit.

[ZoltanAB]

If I make a payment using lncli payinvoice ... --data 65536=$myuniquehex, is there any way to (later) easily find all payments that matches that hex?
Listing all payments (lncli listpayments) and manually comparing the hex is not the ideal solution when you might have hundreds of thousands of payments but only 5-10 has $myuniquehex. Any advise?