Implement datum->syntax for explicitely renamed identifiers

[mnieper]

This also fixed the implementation of SRFI-99, see issue #494.

Some unhygienic macros (like define-record-type of SRFI 99 when the constructor or predicate name is not given explicitly) need to create identifiers that behave as if they were present in the input form. The following example is from Chicken's description of er-macro-transformer:

(define-syntax loop
  (er-macro-transformer
    (lambda (x r c)
      (let ((body (cdr x)))
        `(,(r 'call-with-current-continuation)
          (,(r 'lambda) (exit)
           (,(r 'let) ,(r 'f) () ,@body (,(r 'f)))))))))

Here, exit is not renamed as it shall be eq? to the identifier with the same name where the loop macro is called. However, by issue #494, this macro behaves incorrectly in some sense. When loop is called in the instance of another (hygienic) macro, the identifier exit of the instance may be renamed, while the identifier exit introduced by loop refers to the uncolored identifier of the name exit.

The conclusion is that one cannot write robust unhygienic macros with er-macro-transformer alone. What is missing is something like the (datum->syntax id datum) expression of the R6RS, which takes an identifier id and a datum datum representing an expression and turns datum into a syntax object (an expression that contains syntactic closures (of identifiers)) that behaves as if introduced where id was introduced. The corrected loop macro would be:

(define-syntax loop
  (er-macro-transformer
    (lambda (x r c)
      (let ((body (cdr x)))
        `(,(r 'call-with-current-continuation)
          (,(r 'lambda) ,(datum->syntax (car x) 'exit)
           (,(r 'let) ,(r 'f) () ,@body (,(r 'f)))))))))

This patch does the following to implement datum->syntax:

1. Syntactic closures get a fourth field, called rename, which is #f initially.
2. Whenever er-macro-transformer creates identifiers (as syntactic closures), it stores its rename procedure in the rename field of the created identifier.
3. A procedure (symbol->identifier id sym) is defined, where id is an identifier and sym is a symbol. If id is a syntactic closure, sym is being renamed using the rename procedure stored in id; otherwise sym is returned unchanged.
4. The procedure datum->syntax just recursively applies symbol->identifier to each symbol occurring in the expression datum.

[ashinn]

Thanks very much for the investigation and patch!

This is perhaps the most obvious approach, and I believe was proposed at one time by Taylor Campbell and possibly implemented in his riaxpander.

Another approach involves a hybrid of (or switch to) counters.

I want to think a little about this before merging since it's a fairly heavy-handed approach.

[mnieper]

I have to apologize if this approach had been proposed before and I didn't acknowledge it because I didn't know of prior discussions of this problem. When I discovered the problem for myself, I was surprised that all descriptions/advertisements of er-macro-transformer and sc-macro-transformer I found seemed to be ignorant of it.

Thinking about the patch before merging sound very reasonable. For example, one can ask what the meaning of the rename field in a syntactic closure that is not an identifier should be.

What do you mean by counters? That a colored identifier is just a symbol together with a counter and that the counter is increased for any instantiation of any macro? This sounds like a cut-down version of the marks and substitutions algorithm by Dybvig and Hieb, right?

Whatever the short-time fix for the issue is, it would be nice if the underlying macro system of Chibi would become powerful enough to implement even syntax-case or an analogous system. People have asked for ir-macro-transformer, which syntax-case (apart from the forest of #'s) basically is, but with linear expansion time in the size of the input.

Maybe one can take a look at the implementation of SRFI 72, which I haven't yet, though. I think this is also the implementation used by Larceny.

[Hamayama]

Hello.

I tried to add datum->syntax to Gauche.
shirok/Gauche#399
(sorry Japanese ...)

But I was pointed out that
this solution doesn't work if another macro layer is introduced.

(define-syntax divide
  (lambda (stx)
    (syntax-case stx ()
      ((k x y)
       (with-syntax ((failure (datum->syntax #'k 'failure)))
         #'(if (zero? y)
               (failure)
               (/ x y)))))))

(define-syntax divide2
  (syntax-rules ()
    ((divide2 x y)
     (divide x y))))

(define-syntax wrapper
  (syntax-rules ()
    ((wrapper)
     (let ((failure (lambda ()
                      (display "division by zero")
                      (newline))))
       (divide2 1 0)))))

(let ()
  (wrapper))

This solution needs an identifier in macro which uses 'failure'.

I have no idea to workaround for this case.

Though, I report this for information sharing.

[mnieper]

Hamayama, the reason why your example doesn't work is that (datum->syntax #'k 'failure) creates an identifier that behaves as if introduced in the context of #'k, which is the context of the instantiation of divide2. The identifier failure in your example, however, is bound in the context of the instantiation of wrapper.

Could you briefly explain what your post has to do with my proposed addition to Chibi? Note that Chibi does not have syntax-case (yet?) but you can easily rewrite your divide macro as an er-macro, which Chibi supports.

[Hamayama]

Umm, I know 'divide' can be written by er-macro and datum->syntax.
This is not point...

if divide2 doesn't exist, wrapper works well.
But if divide2 exists, wrapper doesn't work.

From end-user view, divide and divide2 are the same interface.
So datum->syntax solution only works under limited conditions.

[mnieper]

Assume that divide2 introduces another identifier named failure. Without forwarding more contextual information to divide, divide has principally no way to decide which of the two failure identifiers (the one introduced by wrapper and the hypothetical introduced by divide) the user wants it to reference.

So it is not a problem of datum->syntax; it is a principal problem (or, rather, an ill-posed problem, I think).

If you want to create a true wrapper for divide, you can use identifier-syntax if your macro system supports it (one of the better uses for identifier syntax):

(define-syntax divide2 (identifier-syntax divide))

[mnieper]

P.S.: The divide example stems apparently from a post of mine in the Scheme newsgroup. There, it was not meant as an example of practical programming but just an example to illustrate the power of macro systems with (or without) certain primitives like datum->syntax. From a practical point of view, a much better example is given by SRFI 99's needs, see #494.

So, your example applied to SRFI 99 would be something like the following:

(let-syntax
    ((wrapper
     (syntax-rules ()
       ((wrapper1 record)
        (define-record-type record #t #t))
       ((wrapper2)
        (wrapper1 record)
        (record? (make-record)))))
  (wrapper1))

This example (legatimately) fails because the identifiers that (define-record-type) introduces (make-record and record) are renamed when wrapper1 is instantiated. So (wrapper1 record) is (rightly!) not equivalent to (define-record-type record #t #t) (for a good reason because the latter has to be equivalent to (define-record-type record make-RECORD RECORD?) by SRFI 99, where RECORD is the underlying name of the identifier record (which is record in this case), and make-RECORD and RECORD? have to underlie hygiene as well).

[Hamayama]

Uhm, I understood what you say ...

But datum->syntax used in library (such as SRFI 99) introduces
which identifier (e.g. define-record-type or record) are used to make other identifiers (e.g. make-record, record?) implicitly.

To avoid confusion, We should pass arguments to syntax-rules explicitly, shoudn't we?

(But if specification of library (such as SRFI 99) is well documented, datum->syntax might be useful ...
But datum->syntax needs compiler level implemantaion ...
Is it worth to do so? )

[mnieper]

Convincing use cases for unhygienic macros are few in my opinion. For example, to achieve what the divide example from above wants to achieve it would be much better to make use of syntax parameters (see SRFI 139, also implemented in Chibi), which allow the macro to become hygienic:

(define-syntax-parameter failure (syntax-rules ()))

(define-syntax divide
  (syntax-rules ()
    ((divide x y)
     (if (zero? y) (failure) (/ x y)))))

(define-syntax divide2
  (syntax-rules ()
    ((divide2 x y)
     (divide x y))))

(define-syntax wrapper
  (syntax-rules ()
    ((wrapper)
     (let ((f (lambda () (display "division by zero") (newline))))
       (syntax-parameterize ((failure (identifier-syntax f))) ;Or use syntax rules.
         (divide2 1 0))))))

(let ()
  (wrapper))

There is, however, one legitimate use case of unhygienic macros, namely the one of SRFI 99. It mainly saves the user from writing down the derived names make-record and record explicitely.

And if one wanted to implement the module system of the R7RS as a macro itself, one would also need this kind of unhygiene to implement the prefix import modifier.

[Hamayama]

Thank you for detailed explanation.
I became to be able to imagine use cases.
If I recognize where the identifier is inserted,
I would be able to write highly independently macros...
Sorry for many comments here. Thank you!

[mnieper]

@ashinn Before you merge this branch, please wait for my next pull request. I managed to implement the syntax-case system for Chibi. The corresponding pull request includes datum->syntax. Thanks to the power of Chibi's underlying low-level macro system and due to its clearly written source code, it wasn't too difficult in the end.