Allow numeric tokens containing 'e' that aren't exponents be passed to proc macros

[ogoffart]

I tried this code:

// This could also be a complex proc macro
macro_rules! print_token {
    ($x:tt) => { println!("{}", stringify!($x)) }
}

fn main() {
    print_token!(123aefg123);  // ok, unknown suffix
    print_token!(123e123); // ok: float literal
    
    print_token!(123ef3); // error:  expected at least one digit in exponent
}

I expected to see this happen: It should compile in print the token verbatim. If unkown/invalid suffix are OK, so should invalid float, because the macro DSL my use it for something else

Instead, this happened: Compilation error :

error: expected at least one digit in exponent
  --> src/main.rs:11:18
   |
11 |     print_token!(123ef3); // error:  expected at least one digit in exponent
   |                  ^^^^^^

(tested with Rust 1.69 as well as current nightly, this always has been a problem)

Context

This was discussed with @nnethercote and others last week and I wanted to make sure there is a track of this.

The usecase is so that Slint's color literal work in the slint! maco DSL.
Currently one can do color: #0e9 but not color:#0ea in the slint! macro. And Makepad DSL has to workaround the same issue.

[makepaddev]

Chiming in that this is also absolutely a problem for our project (github.com/makepad/makepad) and would really appreciate a fix!. All proc-macro embedded UI systems for Rust need this.

[chenyukang]

This error comes out at next_token, which is too early.
So it's a general issue, here is another case I read from code:

macro_rules! print_token {
    ($x:tt) => { println!("{}", stringify!($x)) }
}

fn main() {
    print_token!("hello"_);
}

Output:

error: underscore literal suffix is not allowed
  --> ./p/mac.rs:11:25
   |
11 |     print_token!("hello"_);
   |                         ^

Seems not easy to fix, we need to change to report this kind of error after macro expanding.

[nnethercote]

I have a partial implementation that works for most literals involving an 'e', including CSS colours of the form #xxyyzz. I need to extend it a bit more to properly distinguish cases like 1e+_______3 (a valid float literal) and 1e+_______a (not a valid float literal, and currently rejected by the lexer).

From an implementation point of view it shouldn't be hard, but the change will need sign-off from the lang team. The relevant section of the Reference will need changing; in particular the Reserved forms similar to literals section as well as the SUFFIX_NO_E nonterminal in the grammar. Working through all this will take some time.

@chenyukang "hello"_ is a different issue to the one facing @ogoffart and @makepaddev. _ is not accepted as a suffix because a suffix must be a valid identifier or keyword, and _ is not. Let's keep this issue about numeric tokens with 'e' in them.

[nnethercote]

#111628 has an draft implementation. Zulip discussion is here.

[joshtriplett]

I'm going to re-nominate this for lang.

To the best of my knowledge, the primary tradeoff here is that it would prevent us from ever defining any integer suffix that starts with e or E and contains only 0-9a-fA-F, if such an integer suffix cannot itself accept arbitrary numeric tokens to the left of it. For instance, we couldn't add a suffix ef for some hypothetical future "Extra-special Float" type, unless that suffix accepted any series of digits to its left. (As long as it does accept any series of digits, I don't think that'd be an issue, because the lexer will parse it as an literal with that suffix but the proc macro can apply whatever interpretation it wants.)

That seems like a potentially reasonable restriction, and I don't think it will arise in practice.

[dtolnay]

it would prevent us from ever defining any integer suffix that starts with e or E and contains only 0-9a-fA-F, if such an integer suffix cannot itself accept arbitrary numeric tokens to the left of it.

This does not follow from the proposal. There would be no issue with defining such a suffix. Any restrictions to it just need to be applied after macro expansion.

For example, exactly the way that fn main() { let _ = stringify!(1e1000); } is a valid program but fn main() { let _ = 1e1000; } fails to compile. This feature request is asking for fn main() { let _ = stringify!(1ef); } to be a valid program. fn main() { let _ = 1ef; } can continue to fail to compile or not, and is orthogonal to this feature request.

[joshtriplett]

Deferring the restrictions until after macro expansion would work, that's true. And since we already do that in some cases, it's reasonable to expect that we'll do it in future cases as well.

[elliotwaite]

it would prevent us from ever defining any integer suffix that starts with e or E and contains only 0-9a-fA-F ...

Although hex strings were the focus of the original comment, a more general solution would be ideal, allowing cases like this to be supported as well:
letter-spacing: .025em

[richard-uk1]

#131656 should close this issue as fixed.