lexical structure: move the description of shebang-removal

This takes place after CRLF normalization.

It's better not to list the shebang in a Lexer block, as it isn't a token that
can be fed to a macro.

Author: mattheww

src/crates-and-source-files.md

@@ -2,14 +2,9 @@
 
 > **<sup>Syntax</sup>**\
 > _Crate_ :\
-> &nbsp;&nbsp; SHEBANG<sup>?</sup>\
 > &nbsp;&nbsp; [_InnerAttribute_]<sup>\*</sup>\
 > &nbsp;&nbsp; [_Item_]<sup>\*</sup>
 
-> **<sup>Lexer</sup>**\
-> SHEBANG : `#!` \~`\n`<sup>\+</sup>[†](#shebang)
-
-
 > Note: Although Rust, like any other language, can be implemented by an
 > interpreter as well as a compiler, the only existing implementation is a
 > compiler, and the language has always been designed to be compiled. For these
@@ -51,6 +46,8 @@ that apply to the containing module, most of which influence the behavior of
 the compiler. The anonymous crate module can have additional attributes that
 apply to the crate as a whole.
 
+> **Note**: The file's contents may be preceded by a [shebang].
+
 ```rust
 // Specify the crate name.
 #![crate_name = "projx"]
@@ -63,28 +60,6 @@ apply to the crate as a whole.
 #![warn(non_camel_case_types)]
 ```
 
-## Shebang
-
-A source file can have a [_shebang_] (SHEBANG production), which indicates
-to the operating system what program to use to execute this file. It serves
-essentially to treat the source file as an executable script. The shebang
-can only occur at the beginning of the file.
-It is ignored by the compiler. For example:
-
-<!-- ignore: tests don't like shebang -->
-```rust,ignore
-#!/usr/bin/env rustx
-
-fn main() {
-    println!("Hello!");
-}
-```
-
-A restriction is imposed on the shebang syntax to avoid confusion with an
-[attribute]. The `#!` characters must not be followed by a `[` token, ignoring
-intervening [comments] or [whitespace]. If this restriction fails, then it is
-not treated as a shebang, but instead as the start of an attribute.
-
 ## Preludes and `no_std`
 
 This section has been moved to the [Preludes chapter](names/preludes.md).
@@ -153,19 +128,17 @@ or `_` (U+005F) characters.
 [_InnerAttribute_]: attributes.md
 [_Item_]: items.md
 [_MetaNameValueStr_]: attributes.md#meta-item-attribute-syntax
-[_shebang_]: https://en.wikipedia.org/wiki/Shebang_(Unix)
 [`ExitCode`]: ../std/process/struct.ExitCode.html
 [`Infallible`]: ../std/convert/enum.Infallible.html
 [`Termination`]: ../std/process/trait.Termination.html
 [attribute]: attributes.md
 [attributes]: attributes.md
-[comments]: comments.md
 [function]: items/functions.md
 [module]: items/modules.md
 [module path]: paths.md
+[shebang]: input-format.md#shebang-removal
 [trait or lifetime bounds]: trait-bounds.md
 [where clauses]: items/generics.md#where-clauses
-[whitespace]: whitespace.md
 
 <script>
 (function() {

src/input-format.md

@@ -19,9 +19,31 @@ Each pair of characters `U+000D` (CR) immediately followed by `U+000A` (LF) is r
 
 Other occurrences of the character `U+000D` (CR) are left in place (they are treated as [whitespace]).
 
+## Shebang removal
+
+If the remaining sequence begins with the characters `!#`, the characters up to and including the first `U+000A` (LF) are removed from the sequence.
+
+For example, the first line of the following file would be ignored:
+
+<!-- ignore: tests don't like shebang -->
+```rust,ignore
+#!/usr/bin/env rustx
+
+fn main() {
+    println!("Hello!");
+}
+```
+
+As an exception, if the `#!` characters are followed (ignoring intervening [comments] or [whitespace]) by a `[` token, nothing is removed.
+This prevents an [inner attribute] at the start of a source file being removed.
+
 ## Tokenization
 
 The resulting sequence of characters is then converted into tokens as described in the remainder of this chapter.
 
+[inner attribute]: attributes.md
 [BYTE ORDER MARK]: https://en.wikipedia.org/wiki/Byte_order_mark#UTF-8
+[comments]: comments.md
 [Crates and source files]: crates-and-source-files.md
+[_shebang_]: https://en.wikipedia.org/wiki/Shebang_(Unix)
+[whitespace]: whitespace.md