Merge pull request #1169 from petrochenkov/procmacrotokens

Document compatibility between declarative and procedural macro tokens

Author: ehuss

src/procedural-macros.md

@@ -271,12 +271,73 @@ fn invoke4() {}
 // out: item: "fn invoke4() {}"
 ```
 
+### Declarative macro tokens and procedural macro tokens
+
+Declarative `macro_rules` macros and procedural macros use similar, but
+different definitions for tokens (or rather [`TokenTree`s].)
+
+Token trees in `macro_rules` (corresponding to `tt` matchers) are defined as
+- Delimited groups (`(...)`, `{...}`, etc)
+- All operators supported by the language, both single-character and
+  multi-character ones (`+`, `+=`).
+    - Note that this set doesn't include the single quote `'`.
+- Literals (`"string"`, `1`, etc)
+    - Note that negation (e.g. `-1`) is never a part of such literal tokens,
+      but a separate operator token.
+- Identifiers, including keywords (`ident`, `r#ident`, `fn`)
+- Lifetimes (`'ident`)
+- Metavariable substitutions in `macro_rules` (e.g. `$my_expr` in
+  `macro_rules! mac { ($my_expr: expr) => { $my_expr } }` after the `mac`'s
+  expansion, which will be considered a single token tree regardless of the
+  passed expression)
+
+Token trees in procedural macros are defined as
+- Delimited groups (`(...)`, `{...}`, etc)
+- All punctuation characters used in operators supported by the language (`+`,
+  but not `+=`), and also the single quote `'` character (typically used in
+  lifetimes, see below for lifetime splitting and joining behavior)
+- Literals (`"string"`, `1`, etc)
+    - Negation (e.g. `-1`) is supported as a part of integer
+      and floating point literals.
+- Identifiers, including keywords (`ident`, `r#ident`, `fn`)
+
+Mismatches between these two definitions are accounted for when token streams
+are passed to and from procedural macros. \
+Note that the conversions below may happen lazily, so they might not happen if
+the tokens are not actually inspected.
+
+When passed to a proc-macro
+- All multi-character operators are broken into single characters.
+- Lifetimes are broken into a `'` character and an identifier.
+- All metavariable substitutions are represented as their underlying token
+  streams.
+    - Such token streams may be wrapped into delimited groups ([`Group`]) with
+      implicit delimiters ([`Delimiter::None`]) when it's necessary for
+      preserving parsing priorities.
+    - `tt` and `ident` substitutions are never wrapped into such groups and
+      always represented as their underlying token trees.
+
+When emitted from a proc macro
+- Punctuation characters are glued into multi-character operators
+  when applicable.
+- Single quotes `'` joined with identifiers are glued into lifetimes.
+- Negative literals are converted into two tokens (the `-` and the literal)
+  possibly wrapped into a delimited group ([`Group`]) with implicit delimiters
+  ([`Delimiter::None`]) when it's necessary for preserving parsing priorities.
+
+Note that neither declarative nor procedural macros support doc comment tokens
+(e.g. `/// Doc`), so they are always converted to token streams representing
+their equivalent `#[doc = r"str"]` attributes when passed to macros.
+
 [Attribute macros]: #attribute-macros
 [Cargo's build scripts]: ../cargo/reference/build-scripts.html
 [Derive macros]: #derive-macros
 [Function-like macros]: #function-like-procedural-macros
+[`Delimiter::None`]: ../proc_macro/enum.Delimiter.html#variant.None
+[`Group`]: ../proc_macro/struct.Group.html
 [`TokenStream`]: ../proc_macro/struct.TokenStream.html
 [`TokenStream`s]: ../proc_macro/struct.TokenStream.html
+[`TokenTree`s]: ../proc_macro/enum.TokenTree.html
 [`compile_error`]: ../std/macro.compile_error.html
 [`derive` attribute]: attributes/derive.md
 [`extern` blocks]: items/external-blocks.md