Adding Data Skipping Examples Guide, Crosslinking pages related to skipping indexes

Author: dhtclk

docs/best-practices/using_data_skipping_indices.md

@@ -13,7 +13,7 @@ import Image from '@theme/IdealImage';
 import building_skipping_indices from '@site/static/images/bestpractices/building_skipping_indices.gif';
 import using_skipping_indices from '@site/static/images/bestpractices/using_skipping_indices.gif';
 
-Data skipping indices should be considered when previous best practices have been followed i.e. types are optimized, a good primary key has been selected and materialized views have been exploited.
+Data skipping indices should be considered when previous best practices have been followed i.e. types are optimized, a good primary key has been selected and materialized views have been exploited. If you're new to skipping indices, [this guide](/optimize/skipping-indexes) is a good place to start.
 
 These types of indices can be used to accelerate query performance if used carefully with an understanding of how they work.
 
@@ -251,3 +251,9 @@ WHERE (CreationDate > '2009-01-01') AND (ViewCount > 10000000)
 We also show an animation how the minmax skipping index prunes all row blocks that cannot possibly contain matches for the `ViewCount` > 10,000,000 predicate in our example query:
 
 <Image img={using_skipping_indices} size="lg" alt="Using skipping indices"/>
+
+## Related docs {#related-docs}
+- [Data skipping indices guide](/optimize/skipping-indexes)
+- [Data skipping index examples](/optimize/skipping-indexes/examples)
+- [Manipulating data skipping indices](/sql-reference/statements/alter/skipping-index)
+- [System table information](/operations/system-tables/data_skipping_indices)

docs/guides/best-practices/skipping-indexes-examples.md

@@ -0,0 +1,212 @@
+---
+slug: /optimize/skipping-indexes/examples
+sidebar_label: 'Data Skipping Indexes - Examples'
+sidebar_position: 2
+description: 'Consolidated Skip Index Examples'
+title: 'Data Skipping Index Examples'
+doc_type: 'guide'
+---
+
+# Data Skipping Index Examples {#data-skipping-index-examples}
+
+This page consolidates ClickHouse data skipping index examples, showing how to declare each type, when to use them, and how to verify they're applied. All features work with MergeTree-family tables.
+
+**Index syntax:** `INDEX name expr TYPE type(...) [GRANULARITY N]`
+
+ClickHouse supports five skip index types:
+
+* **minmax** \- Tracks minimum and maximum values in each granule  
+* **set(N)** \- Stores up to N distinct values per granule  
+* **bloom\_filter(\[false\_positive\_rate\])** \- Probabilistic filter for existence checks  
+* **ngrambf\_v1** \- N-gram bloom filter for substring searches  
+* **tokenbf\_v1** \- Token-based bloom filter for full-text searches
+
+Each section provides examples with sample data and demonstrates how to verify index usage in query execution.
+
+## MinMax index {#minmax-index}
+
+Best for range predicates on loosely sorted data or columns correlated with ORDER BY.
+
+```sql
+-- Define in CREATE TABLE
+CREATE TABLE events
+(
+  ts DateTime,
+  user_id UInt64,
+  value UInt32,
+  INDEX ts_minmax ts TYPE minmax GRANULARITY 1
+)
+ENGINE=MergeTree
+ORDER BY ts;
+
+-- Or add later and materialize
+ALTER TABLE events ADD INDEX ts_minmax ts TYPE minmax GRANULARITY 1;
+ALTER TABLE events MATERIALIZE INDEX ts_minmax;
+
+-- Query that benefits
+SELECT count() FROM events WHERE ts >= now() - 3600;
+
+-- Verify usage
+EXPLAIN indexes = 1
+SELECT count() FROM events WHERE ts >= now() - 3600;
+```
+
+See a [worked example](/best-practices/use-data-skipping-indices-where-appropriate#example) with EXPLAIN and pruning.
+
+## Set index {#set-index}
+
+Use when local (per-block) cardinality is low; not helpful if each block has many distinct values.
+
+```sql
+ALTER TABLE events ADD INDEX user_set user_id TYPE set(100) GRANULARITY 1;
+ALTER TABLE events MATERIALIZE INDEX user_set;
+
+SELECT * FROM events WHERE user_id IN (101, 202);
+
+EXPLAIN indexes = 1
+SELECT * FROM events WHERE user_id IN (101, 202);
+```
+
+Creation/materialization workflow and before/after effect are shown in the [basic operation guide](/optimize/skipping-indexes#basic-operation).
+
+## Generic Bloom filter (scalar) {#generic-bloom-filter-scalar}
+
+Good for "needle in a haystack" equality/IN membership. Optional parameter is the false-positive rate (default 0.025). 
+
+```sql
+ALTER TABLE events ADD INDEX value_bf value TYPE bloom_filter(0.01) GRANULARITY 3;
+ALTER TABLE events MATERIALIZE INDEX value_bf;
+
+SELECT * FROM events WHERE value IN (7, 42, 99);
+
+EXPLAIN indexes = 1
+SELECT * FROM events WHERE value IN (7, 42, 99);
+```
+
+## N-gram Bloom filter (ngrambf\_v1) for substring search {#n-gram-bloom-filter-ngrambf-v1-for-substring-search}
+
+Splits strings into n-grams; works well for LIKE '%...%'. Supports String/FixedString/Map (via mapKeys/mapValues). Tunable size, hash count, seed. See documentation for [N-gram bloom filter](/engines/table-engines/mergetree-family/mergetree#n-gram-bloom-filter).
+
+```sql
+-- Create index for substring search
+ALTER TABLE logs ADD INDEX msg_ngram msg TYPE ngrambf_v1(3, 10000, 3, 7) GRANULARITY 1;
+ALTER TABLE logs MATERIALIZE INDEX msg_ngram;
+
+-- Substring search
+SELECT count() FROM logs WHERE msg LIKE '%timeout%';
+
+EXPLAIN indexes = 1
+SELECT count() FROM logs WHERE msg LIKE '%timeout%';
+```
+
+[This guide](/use-cases/observability/schema-design#bloom-filters-for-text-search) shows practical examples and when to use token vs ngram.
+
+**Parameter optimization helpers:**
+
+The four ngrambf\_v1 parameters (n-gram size, bitmap size, hash functions, seed) significantly impact performance and memory usage. Use these functions to calculate optimal bitmap size and hash function count based on your expected n-gram volume and desired false positive rate:
+
+```sql
+CREATE FUNCTION bfEstimateFunctions AS
+(total_grams, bits) -> round((bits / total_grams) * log(2));
+
+CREATE FUNCTION bfEstimateBmSize AS
+(total_grams, p_false) -> ceil((total_grams * log(p_false)) / log(1 / pow(2, log(2))));
+
+-- Example sizing for 4300 ngrams, p_false = 0.0001
+SELECT bfEstimateBmSize(4300, 0.0001) / 8 AS size_bytes;  -- ~10304
+SELECT bfEstimateFunctions(4300, bfEstimateBmSize(4300, 0.0001)) AS k; -- ~13
+```
+
+See [parameter docs](/engines/table-engines/mergetree-family/mergetree#n-gram-bloom-filter) for complete tuning guidance.  
+
+## Token Bloom filter (tokenbf\_v1) for word-based search {#token-bloom-filter-tokenbf-v1-for-word-based-search}
+
+Indexes tokens separated by non-alphanumeric characters; use with hasToken, LIKE word patterns, equals/IN. Supports String/FixedString/Map. See: [Token bloom filter](/engines/table-engines/mergetree-family/mergetree#token-bloom-filter) and [Bloom filter types](/optimize/skipping-indexes#skip-index-types).
+
+```sql
+ALTER TABLE logs ADD INDEX msg_token lower(msg) TYPE tokenbf_v1(10000, 7, 7) GRANULARITY 1;
+ALTER TABLE logs MATERIALIZE INDEX msg_token;
+
+-- Word search (case-insensitive via lower)
+SELECT count() FROM logs WHERE hasToken(lower(msg), 'exception');
+
+EXPLAIN indexes = 1
+SELECT count() FROM logs WHERE hasToken(lower(msg), 'exception');
+```
+
+See observability examples and guidance on token vs ngram [here](/use-cases/observability/schema-design#bloom-filters-for-text-search).
+
+## Add indexes during CREATE TABLE (multiple examples) {#add-indexes-during-create-table-multiple-examples}
+
+Also supports composite expressions and Map/Tuple/Nested.
+
+```sql
+CREATE TABLE t
+(
+  u64 UInt64,
+  s String,
+  m Map(String, String),
+
+  INDEX idx_bf u64 TYPE bloom_filter(0.01) GRANULARITY 3,
+  INDEX idx_minmax u64 TYPE minmax GRANULARITY 1,
+  INDEX idx_set u64 * length(s) TYPE set(1000) GRANULARITY 4,
+  INDEX idx_ngram s TYPE ngrambf_v1(3, 10000, 3, 7) GRANULARITY 1,
+  INDEX idx_token mapKeys(m) TYPE tokenbf_v1(10000, 7, 7) GRANULARITY 1
+)
+ENGINE = MergeTree
+ORDER BY u64;
+```
+
+## Materializing on existing data and verifying {#materializing-on-existing-data-and-verifying}
+
+Add an index to existing data parts using MATERIALIZE, and inspect pruning with EXPLAIN or trace logs.
+
+```sql
+ALTER TABLE t MATERIALIZE INDEX idx_bf;
+
+EXPLAIN indexes = 1
+SELECT count() FROM t WHERE u64 IN (123, 456);
+
+-- Optional: detailed pruning info
+SET send_logs_level = 'trace';
+```
+
+A [worked minmax example](/best-practices/use-data-skipping-indices-where-appropriate#example) demonstrates EXPLAIN output structure and pruning counts.
+
+## When use and when to avoid {#when-use-and-when-to-avoid}
+
+**Use skip indexes when:**
+
+* Filter values are sparse within data blocks  
+* Strong correlation exists with ORDER BY columns or data ingestion patterns group similar values together  
+* Performing text searches on large log datasets (ngrambf\_v1/tokenbf\_v1 types)
+
+**Avoid skip indexes when:**
+
+* Most blocks likely contain at least one matching value (blocks will be read regardless)  
+* Filtering on high-cardinality columns with no correlation to data ordering
+
+**Important considerations:** If a value appears even once in a data block, ClickHouse must read the entire block. Test indexes with realistic datasets and adjust granularity and type-specific parameters based on actual performance measurements.
+
+## Temporarily ignore or force indexes {#temporarily-ignore-or-force-indexes}
+
+Disable specific indexes by name for individual queries during testing and troubleshooting. Settings also exist to force index usage when needed. See [ignore\_data\_skipping\_indices](/operations/settings/settings#ignore_data_skipping_indices).
+
+```sql
+-- Ignore an index by name
+SELECT * FROM logs
+WHERE hasToken(lower(msg), 'exception')
+SETTINGS ignore_data_skipping_indices = 'msg_token';
+```
+
+## Notes and caveats {#notes-and-caveats}
+
+* Only supported on MergeTree-family tables; pruning happens at the granule/block level.  
+* Bloom-filter-based indexes are probabilistic (false positives cause extra reads but won't skip valid data).  
+* Bloom filters and other skip indexes should be validated with EXPLAIN and tracing; adjust granularity to balance pruning vs. index size.
+
+## Related docs {#related-docs}
+- [Data skipping index guide](/optimize/skipping-indexes)
+- [Best practices guide](/best-practices/use-data-skipping-indices-where-appropriate)
+- [Manipulating data skipping indices](/sql-reference/statements/alter/skipping-index)
+- [System table information](/operations/system-tables/data_skipping_indices)

docs/guides/best-practices/skipping-indexes.md

@@ -218,3 +218,9 @@ data skipping index behavior is not easily predictable. Adding them to a table i
 that for any number of reasons don't benefit from the index. They should always be tested on real world type of data, and testing should
 include variations of the type, granularity size and other parameters. Testing will often reveal patterns and pitfalls that aren't obvious from
 thought experiments alone.
+
+## Related docs {#related-docs}
+- [Best practices guide](/best-practices/use-data-skipping-indices-where-appropriate)
+- [Data skipping index examples](/optimize/skipping-indexes/examples)
+- [Manipulating data skipping indices](/sql-reference/statements/alter/skipping-index)
+- [System table information](/operations/system-tables/data_skipping_indices)

sidebars.js

@@ -1108,7 +1108,16 @@ const sidebars = {
         "guides/best-practices/sparse-primary-indexes",
         "guides/best-practices/query-parallelism",
         "guides/best-practices/partitioningkey",
-        "guides/best-practices/skipping-indexes",
+        {
+          type: "category",
+          label: "Data Skipping Indexes",
+          collapsed: true,
+          collapsible: true,
+          link: { type: "doc", id: "guides/best-practices/skipping-indexes" },
+          items: [
+            "guides/best-practices/skipping-indexes-examples"
+          ],
+        },
         "guides/best-practices/prewhere",
         "guides/best-practices/bulkinserts",
         "guides/best-practices/asyncinserts",