docs: Generate SQL documentation reference

[paleolimbot]

Our existing SQL documentation works but is tricky to keep up-to-date with changes to sedona-functions!

sedona-db/docs/reference/sql.md

Lines 20 to 26 in 11e2993

	# SQL API Reference
	The following SQL functions are available for SedonaDB.
	You can query data directly from files and URLs by treating them like database tables. This feature supports formats like **Parquet**, **CSV**, and **JSON**.
	To query a file, place its path or URL in single quotes within the `FROM` clause.

This file was initially generated from the documentation we've been keeping inline in the functions themselves. For example:

sedona-db/rust/sedona-functions/src/st_dimension.rs

Lines 41 to 50 in 11e2993

	fn st_dimension_doc() -> Documentation {
	Documentation::builder(
	DOC_SECTION_OTHER,
	"Return the dimension of the geometry",
	"ST_Dimension (A: Geometry)",
	)
	.with_argument("geom", "geometry: Input geometry")
	.with_sql_example("SELECT ST_Dimension(ST_GeomFromWKT('POLYGON EMPTY'))")
	.build()
	}

We can recreate the script we used to generate the initial reference section to ensure we keep this all up to date (and potentially generate reference pages for R and Python when we have function wrappers there). Alternatively we could go in the other direction (have extremely carefully formatted markdown reference that we parse into Rust, R, and Python documentation).

It would also be great to have one page per function to allow these pages room to grow as we accumulate experience and examples with these functions!

[paleolimbot]

There are a few potentially useful things we can do here. First, we could formalize the script we use to generate the one file (this would probably be a good time to also split up everything so we have one markdown file per function. I just found this out but apparently it's available from the information schema:

import sedona.db

sd = sedona.db.connect()

sd.sql("""
SELECT DISTINCT routine_name 
FROM information_schema.routines 
WHERE routine_type = 'FUNCTION' AND routine_name LIKE 'st%'
ORDER BY routine_name
""").show()
#> ┌────────────────┐
#> │  routine_name  │
#> │      utf8      │
#> ╞════════════════╡
#> │ st_analyze_agg │
#> ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
#> │ st_area        │
#> ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
#> │ st_asbinary    │
#> ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤

I think after we do that generation we might want to have the individual markdown files be the canonical documentation for that function. We can make them .qmd files and use Quarto to execute them (among other things, this would let us include plot output in the results so we could visualize some of these functions!).

[paleolimbot]

(Credit to @petern48 for the SQL to make this happen!)

[yutannihilation]

Nice usage of Quarto! Regarding metadata such as changes between versions (e.g. #368 (comment)), the description stores all the information? Or, should it be stored in some other file?

[paleolimbot]

Currently the only thing that is actually rendered to the docs site is the Markdown file that's edited in that PR. I think initially it was rendered using the descriptions in rust/sedona-functions but may have evolved a little. Probably it would make it easier for contributors to go in the other direction (derive the DataFusion Documentation from the markdown file or maybe even just provide a permalink) but really anything that improves on the current situation would be great.