Merge #26 feat!: listitem (line_li)

Author: justinmk

README.md

@@ -1,32 +1,33 @@
 tree-sitter-vimdoc
 ==================
 
-This grammar intentionally support a subset of the vimdoc "spec"; predictable
-results are the primary goal, so that _output_ formats (e.g. HTML) are
-well-formed; the _input_ (vimdoc) is secondary. The first step should always be
-to try to fix the input (within reason) rather than insist on a grammar that
-handles vimdoc's endless quirks.
+This grammar intentionally support a subset of the vimdoc "spec"
+([ref1](https://neovim.io/doc/user/helphelp.html#help-writing),
+[ref2](https://github.com/nanotee/vimdoc-notes));
+predictable results are the primary goal, so that _output_ formats (e.g. HTML)
+are well-formed; the _input_ (vimdoc) is secondary. The first step should
+always be to try to fix the input rather than insist on a grammar that handles
+vimdoc's endless quirks.
 
 Overview
 --------
 
-- vimdoc format "spec":
-    - [:help help-writing](https://neovim.io/doc/user/helphelp.html#help-writing)
-    - https://github.com/nanotee/vimdoc-notes
-- whitespace is intentionally captured in all atoms, because it is often used
-  for "layout" and ascii art in legacy help files.
-- `block` is the main top-level node which contains `line` nodes.
-  - ends at blank line(s) or a line starting with `<`.
+- `block` is the main top-level node which contains `line` and `line_li` nodes.
+  - delimited by blank line(s) or any line starting with `<` (codeblock terminator).
 - `line`:
   - contains atoms (words, tags, taglinks, …)
-  - contains `codeblock` because `>` can start a codeblock at the end of a line.
   - contains headings (`h1`, `h2`, `h3`) because `codeblock` terminated by
     "implicit stop" (no terminating `<`) consumes blank lines, so `block` has
     no way to end.
   - contains `column_heading` because `<` (the `codeblock` terminating char)
     can appear at the start of `column_heading`.
+- `line_li` ("list item")
+  - consumes lines until blank line, codeblock, or next listitem.
+  - nesting is ignored: indented listitems are parsed as siblings.
 - `codeblock`:
-  - contains `line` nodes which do not contain `word` nodes, it's just the full
+  - contained by `line` or `line_li`. Because ">" can start
+    a codeblock at the end of any line.
+  - contains `line` nodes without `word` nodes, it's just the full
     raw text line including whitespace. This is somewhat dictated by its
     "preformatted" nature; parsing the contents would require loading a "child"
     language (injection). See [#2](https://github.com/neovim/tree-sitter-vimdoc/issues/2).
@@ -38,16 +39,20 @@ Overview
 Known issues
 ------------
 
-- `line_li` ("list item") is experimental. It doesn't support nesting yet.
+- Input must end with newline/EOL (`\n`). Grammar does not support files without EOL.
+- Input must end with a blank line. Though this doesn't seem to matter in practice.
 - Spec requires that `codeblock` delimiter ">" must be preceded by a space
   (" >"), not a tab. But currently the grammar doesn't enforce this. Example:
   `:help lcs-tab`.
 - `url` doesn't handle _surrounding_ parens. E.g. `(https://example.com/#yay)` yields `word`
 - `url` doesn't handle _nested_ parens. E.g. `(https://example.com/(foo)#yay)`
+- `column_heading` currently only recognizes tilde "~" preceded by space (i.e.
+  "foo ~" not "foo~"). This covers 99% of :help files, but the grammar should
+  probably support "foo~" also.
 
 TODO
 ----
 
-- `line_noeol` is a special-case to support documents that don't end in EOL.
-  Grammar could be simpler if we require EOL at end of document.
 - `line_modeline` ?
+- `tag_heading` : line(s) containing only tags, typically implies a "heading"
+  before a block.

corpus/arguments.txt

@@ -2,6 +2,8 @@
 simple argument
 ================================================================================
 This in an argument: {arg}
+
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -19,6 +21,7 @@ multiple arguments on the same line
 ================================================================================
 {foo} {bar} {baz}
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -31,11 +34,28 @@ multiple arguments on the same line
       (argument
         (word)))))
 
+================================================================================
+argument in parentheses
+================================================================================
+({aaa})
+
+
+--------------------------------------------------------------------------------
+
+(help_file
+  (block
+    (line
+      (word)
+      (argument
+        (word))
+      (word))))
+
 ================================================================================
 NOT an argument
 ================================================================================
 {foo "{bar}" `{baz}` |{baz| } {}
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -45,9 +65,6 @@ NOT an argument
         (word)
         (MISSING "}"))
       (word)
-      (argument
-        (word))
-      (word)
       (codespan
         (word))
       (taglink

corpus/codeblock.txt

@@ -16,6 +16,8 @@ block3:
 <
 
   text
+
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -62,6 +64,7 @@ From luaref.txt:
 
 text
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -79,6 +82,7 @@ text
       (word)))
   (block
     (line
+      (word)
       (word)
       (taglink
         (word))
@@ -116,6 +120,7 @@ h2-headline *foo*
 
 H3 HEADLINE *foo*
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -162,8 +167,8 @@ codeblock with empty lines
 
 Example: >
 
-    let buf = nvim_create_buf(v:false, v:true)
-    call nvim_buf_set_lines(buf, 0, -1, v:true, ["test", "text"])
+    - let buf = nvim_create_buf(v:false, v:true)
+    - call nvim_buf_set_lines(buf, 0, -1, v:true, ["test", "text"])
 
     let opts = {'relative': 'cursor', 'width': 10, 'height': 2, 'col': 0,
         \ 'row': 1, 'anchor': 'NW', 'style': 'minimal'}
@@ -172,6 +177,7 @@ Example: >
     call nvim_win_set_option(win, 'winhl', 'Normal:MyHighlight')
 x
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -211,6 +217,7 @@ tricky codeblock
 
 tricky
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -237,7 +244,6 @@ tricky
 ================================================================================
 strange codeblock
 ================================================================================
-
 Note that in the command: >
 	:if "foo"
 	:" NOT executed
@@ -295,6 +301,7 @@ To test for a non-empty string, use empty(): >
       (word)
       (word)
       (word)
+      (word)
       (codeblock
         (line)))))
 

corpus/codespan.txt

@@ -4,6 +4,7 @@ simple codespan
 
 a `foobar` b `:echo`
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -23,6 +24,7 @@ codespan in text
 Hello `world`, I am `markup language`. But `this is
 an error`.
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -52,6 +54,7 @@ NOT codespan
 				*g'* *g'a* *g`* *g`a*
 g'{mark}  g`{mark}
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -76,8 +79,4 @@ g'{mark}  g`{mark}
         (word)))
     (line
       (word)
-      (argument
-        (word))
-      (word)
-      (argument
-        (word)))))
+        (word))))

corpus/heading1_2.txt

@@ -12,6 +12,7 @@ Heading two                                                *tag2.1*   *tag2.2*
 
 Text2
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -51,6 +52,7 @@ Text
 
 Text
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -104,6 +106,7 @@ Text
 Not-a-heading-4                                                     *heading4*
 Text
 
+
 --------------------------------------------------------------------------------
 
 (help_file