Fetch readme and generate Markdown for CLI flags at build time

[katrinafyi]

It looks like this:

This implementation works but there are fun details to think about:

• Is the formatting good? atm, it's a funny mix of rich text markdown (for the headings) and monospace text (for the help). I think this is a good compromise between having nice formatting and enabling automatic generation
• also on formatting, astro will combine two hyphens (--) into a unicode em dash (—) and this affects the option flags in markdown headings. I've prevented this using a zero width space but it's a bit :/ is this okay? do we care about keeping the original hyphens?
• The generation is kind of hacky. It doesn't hook into any astro APIs. instead, it just runs JavaScript code in the content definition file. This means that, for instance, astro needs to be manually restarted after changes to the generator or the _cli.md template file.
• I tried custom astro content collections as a more "official" way to do this generation. however, it looked like headings would not be propagated if you render a fragment within another page (as we would want to embed the generated options within the template page). this negates most of the benefit

Todo:

• assertion for body text

[mre]

The rich text headings + monospace blocks is actually quite good. It makes the docs scannable while preserving the exact CLI output format. I'd keep this.

About the em dash issue, a few alternatives come to mind:

• Wrap option names in backticks: ### `--dump` (keeps hyphens, looks cleaner)
• Use a custom Astro component for option headings
• Configure Astro's typography settings to disable smart quotes/dashes

Have you tried to use Astro's integration system instead?
I'm not an Astro expert by any means. Just found it in their docs: https://docs.astro.build/en/guides/integrations-guide/

Create a proper Astro integration that runs during the build:

// astro.config.mjs
import { generateCliDocs } from './src/integrations/cli-docs.ts';

export default defineConfig({
  integrations: [
    generateCliDocs(),
  ]
});

// src/integrations/cli-docs.ts
import type { AstroIntegration } from 'astro';
import { generateCliOptionsMarkdown } from '../fetchReadme';
import { writeFileSync, readFileSync, rmSync } from 'node:fs';

export function generateCliDocs(): AstroIntegration {
  return {
    name: 'generate-cli-docs',
    hooks: {
      'astro:config:setup': async () => {
        const docTemplateFile = "src/content/docs/guides/_cli.md";
        const docOutputFile = docTemplateFile.replace('_', '');
        
        rmSync(docOutputFile, { force: true });
        const usageText = await generateCliOptionsMarkdown();
        
        const docTemplateText = readFileSync(docTemplateFile, "utf-8");
        const docOutput = docTemplateText.replace(
          'README-OPTIONS-PLACEHOLDER', 
          usageText
        );
        writeFileSync(docOutputFile, docOutput);
      }
    }
  };
}

The main benefit would be that the dev server would regenerate the docs on restart, and you wouldn't need to manually run a script to update the docs.
But I haven't tested it. @jacobdalamb knows way more about Astro than me.

[jacobdalamb]

@mre good point. I hadn't considered using Astro's integration API to add this functionality. Because you mentioned that, I started going through community made integrations and came across this https://github.com/natemoo-re/astro-remote#README which enables "render remote HTML or Markdown content in Astro with full control over the output." what do you think?
@katrinafyi

[katrinafyi]

astro-readme looks promising! I tried it but it has the same problem as the content collections approach, it doesn't add headers from the remote content to the TOC.

But maybe we can cook up our own integration to get reloading.

[katrinafyi]

I've made changes following @mre's suggestions. We disable smart dashes across the entire site and we use a custom integration to generate the file. The integration code is basically the same as mre's suggestion, but we have to add addWatchFile to reload when the md file is changed.

[mre]

Neat! I can see how this is taking shape. 😊

One thing I wanted to mention was that I like the colorful examples in the current version (these little "terminal" boxes). Can we get them back somehow? I don't know how hard that would be, so that's completely optional, of course.

Before

After

[thomas-zahner]

@katrinafyi Cool, I really like that. Rebasing on top of master should fix the link check errors.

[katrinafyi]

I've added some colourful boxes. However, it's pretty naive and they're not quite valid examples - they still use placeholder variables and, in the case where a flag requires another, it does not show the flags being used together.

It looks like this

The more I work with this, the more I think that it would be nice if the docs site shows the CLI help as markdown formatted text. Initially, I was reluctant because it means the CLI help text would have to be written with a particular formatting which is only useful to the website. But maybe it's not so bad because it's already most of the way there and the hard wrapping of lines is already a requirement with the code block approach. I'll investigate.

The CI pipeline that builds astro is failing at the astro build step. Idk why because the package should exist as a transitive dependency, and it works on my machine with an npm ci. Also, I feel like the README for this repo used to show npm commands to use but those have disappeared - why is pnpm being used in CI? I will appreciate any guidance.

[katrinafyi]

Update to my last comment, I've gone ahead and made it markdown text rather than code blocks. See commit for commentary de6c2a8

This generally works fine and looks nice, aside from --files-from:

Examples:
  lychee --files-from list.txt
  find . -name '*.md' | lychee --files-from -
  echo 'README.md' | lychee --files-from -

File Format:
  Each line should contain one input (file path, URL, or glob pattern).
  Lines starting with '#' are treated as comments and ignored.
  Empty lines are also ignored.

If this is something we want, I'll PR to tweak the files-from help text.

[jacobdalamb]

Check if remark-smartypants exists within node_modules if not, delete it and pnpm-lock.ymal and reinstall packages.

[katrinafyi]

It exists in my node modules after I use npm ci. Should I be using npm or pnpm? Is there a difference? Idk pnpm, so it would be good to have some commands in the readme.

[jacobdalamb]

Use PNPM. I've added back the commands in the readme.

[mre]

I really like the new formatting!

[thomas-zahner]

Greatly appreciated PR, this really helps to keep our documentation up to date!