Implement anchors for CLI flags, and automatically fetch --help

[katrinafyi]

This PR is two changes in one:

• Changes the CLI options page to fetch from https://raw.githubusercontent.com/lycheeverse/lychee/master/README.md at build-time. (The previous CLI options were somewhat out of date and would've needed updating anyway).
• Adds anchor links to each option in the help text output. This is done using astro-expressive-code's line markers feature which allows annotating certain lines with custom text. We use a small custom plugin to convert these text markers into anchored hyperlinks.

The result looks like this. Each line beginning with # is a hyperlink to that line. The anchor links work as expected when clicking on the same page and when navigating to the page from elsewhere.

One thing to be aware of with the automatic fetching of README.md is that this will always get the latest version from master. This might be undesirable because re-builds of the same docs could fetch different copies of README.md. Also, it may show options which are not yet available in an official version.

I've added a LYCHEE_VERSION variable for specifying the version which it fetches from, but I don't know where this variable should be defined - in CI maybe?

Also, it would be easy to split this PR into two if it would be easier to discuss the two changes separately.

Related to #95

[jacobdalamb]

Thank you for this PR!

I've implemented an alternative approach that addresses your suggestions:

1. CLI Documentation Organization

Instead of fetching from the README at build-time, I've manually organized all CLI options into logical sections with individual headings, descriptions, and examples. This gives us better UX and control over documentation quality, with no risk of build inconsistencies.

Good news: Starlight/Astro already auto-generates anchor links for all headings, so each option like --exclude and --timeout is automatically linkable!

See the updated documentation.

2. Keeping Docs In Sync

I've created a weekly workflow that checks if our documented version matches the latest release. When there's a mismatch, it creates an issue to remind us to update. This gives us review control and ties documentation to released versions (not master).

See the workflow: .github/workflows/check-cli-version.yml

Your concern about fetching from master showing unreleased options is spot-on. This approach ensures users only see options actually available to them, while the workflow keeps us current without build-time complexity.

Thanks for identifying this issue and proposing a solution! 🚀

[katrinafyi]

Right.......... 😕

[mre]

Hm, to be honest I liked @katrinafyi's format better. Any chance we can adopt it?

I'd also prefer generating the docs at build time and not via a workflow that creates an issue. The reason is that it's one less manual step to worry about. It's also relatively quick to fetch that file at build time. The one downside is that it can't start if GitHub is down, but that's an acceptable risk in my opinion and it rarely happens.

Any chance we can harvest some of Katarina's work?

[jacobdalamb]

@mre Sure! I think we can. I originally dismissed the build-time fetch + anchors idea (though it's a smart one) mainly because I'm less familiar with that setup and wanted to prioritize a manual structure for more control over the UX. That said, I apologize for not engaging more enthusiastically earlier @katrinafyi, and thanks for the solid implementation. Would you like to reopen or create a new pull request to iterate on it?

[katrinafyi]

For those curious, jacobdelamb's PR is #103.

I should make it clear that I'm dismayed at the way this PR was handled and closed.

I send PRs with the hope that they will be reviewed, and I write the code and description with the hope that it makes reviewing easier. Review means that comments are posted to the PR, maybe my questions are answered, and we can iterate and converge on the best solution. Taking the notes and suggestions from this PR and "addressing" them in an unrelated PR is.. nonsensical.

Not to mention, although the other PR superficially solves the same problem, it is an entirely different design that makes different design decisions and has a vastly bigger scope. I think the other pr would've benefitted from much more review! Though, that was made to be impossible given its fearsome size. In general, I'm also extremely suspicious of using AI to write large amounts of technical documentation, especially given the nuance and corner cases I've found in Lychee.

Anyway, I can make a new PR when I have time. You (anyone) are also welcome to take the changes here and build on them you want - I'll answer any questions to help you become familiar.

[katrinafyi]

This is where we're at. I could port the old changes as-is, but I've become attached to the list of options on the side in the table of contents. I think it's quite handy. However, technically, this seems mutually exclusive with having the automatically-fetched README. The TOC rendering seems to happen using the Markdown only, and this happens before any execution of build-time Javascript.

The most reasonable solution I have right now is to run Javascript code very early in the build to generate a md file in src/content/docs which will then go through the usual markdown parsing as if it was a static file.

This would look something like this, in content.config.ts:

import { readFileSync, writeFileSync } from "node:fs";

const readme = fetch("https://raw.githubusercontent.com/lycheeverse/lychee/refs/heads/master/LICENSE-MIT")

const docTemplate = readFileSync("src/content/docs/usage/_cli.md", "utf-8");

const usageText = addHeadings(await (await readme).text());

const x = docTemplate.replace('OPTIONSGOHERE', usageText);

writeFileSync("src/content/docs/usage/cli.md", x);

Which is obviously goofy. But I do think it's important to have some automated fetching, otherwise maintenance will become a huge burden.

What're your thoughts?

[mre]

Currently busy, but I'll reopen this PR, just that we don't forget to follow up on that. @jacobdalamb, do you have any input?

[katrinafyi]

Doing the idea from #102 (comment), it currently looks like this. The markdown is derived from the readme text by some javascript code and writes a md file to disk.

I had the thought that the old approach of annotating lines in a single code block could still be useful where the code block is one logical file, like if we want to add anchor links to the lychee.toml example.

[jacobdalamb]

Changes look solid. I think the URL fetching should have error handling though.

[katrinafyi]

Yeah sure, I've opened a PR at #113. This was just a rough POC to see if we liked the idea.

[katrinafyi]

Superseded by

• Fetch readme and generate Markdown for CLI flags at build time #113