Creating Sitemap for Search Engine

[billy1624]

I'm wondering why crates.io does not have sitemap created for search engine? This makes Rust developer hard to search for corresponding published crate using search engine.

Most of the time I only found the GitHub repo and lib.rs, instead of crates.io.

REF: For example lib.rs provide sitemap here.

[Turbo87]

that file has 28204 records (one per crate). crates.io has 70990 crates. that means the sitemap there is incomplete. we could add all the crates to that file, but then what happens when crates.io at somepoint will have 1000000 crates. that file would would become unmaintainably large.

[billy1624]

We can split it into multiple sitemap. Just like what https://github.com/rust-lang/crates.io-index does.

REF: https://developers.google.com/search/docs/advanced/sitemaps/large-sitemaps

[Turbo87]

Just like what https://github.com/rust-lang/crates.io-index does.

where is the sitemap for that repository? 🤔

REF: https://developers.google.com/search/docs/advanced/sitemaps/large-sitemaps

yeah, I guess that might be an option.

I'm not sure whether the missing sitemap is actually the problem here though. crates on crates.io show up in the Google results, so they generally do find our pages. it's more a question on why other results are higher up in the results.