Merge branch 'main' of https://github.com/ClickHouse/clickhouse-docs into dhtclk-patch3

Author: Blargian

.github/CODEOWNERS

@@ -1,3 +1,3 @@
 *                      @ClickHouse/docs
-/docs/integrations/data-ingestion/clickpipes/ @ClickHouse/clickpipes @ClickHouse/docs
 /docs/integrations/ @ClickHouse/integrations-ecosystem @ClickHouse/docs
+/docs/integrations/data-ingestion/clickpipes/ @ClickHouse/clickpipes @ClickHouse/docs

README.md

@@ -134,9 +134,9 @@ Please assign any pull request (PR) against an issue; this helps the docs team t
 
 Check out the GitHub docs for a refresher on [how to create a pull request](https://docs.github.com/en/desktop/working-with-your-remote-repository-on-github-or-github-enterprise/creating-an-issue-or-pull-request-from-github-desktop).
 
-### Style guidelines
+### Style and contribution guidelines
 
-For documentation style guidelines, see ["Style guide"](/contribute/style-guide.md). 
+For documentation style guidelines, see ["Style guide"](/contribute/style-guide.md).
 
 To check spelling and markdown is correct locally run:
 

code_snippets/ClickStack/config-unstructured-logs-with-processor.yaml

@@ -0,0 +1,43 @@
+receivers:
+  filelog:
+    include:
+      - /opt/data/logs/access-unstructured.log
+    start_at: beginning
+    operators:
+      - type: regex_parser
+        regex: '^(?P<ip>[\d.]+)\s+-\s+-\s+\[(?P<timestamp>[^\]]+)\]\s+"(?P<method>[A-Z]+)\s+(?P<url>[^\s]+)\s+HTTP/[^\s]+"\s+(?P<status>\d+)\s+(?P<size>\d+)\s+"(?P<referrer>[^"]*)"\s+"(?P<user_agent>[^"]*)"'
+        timestamp:
+          parse_from: attributes.timestamp
+          layout: '%d/%b/%Y:%H:%M:%S %z'
+          #22/Jan/2019:03:56:14 +0330
+processors:
+  batch:
+    timeout: 1s
+    send_batch_size: 100
+  memory_limiter:
+    check_interval: 1s
+    limit_mib: 2048
+    spike_limit_mib: 256
+exporters:
+  # HTTP setup
+  otlphttp/hdx:
+    endpoint: 'http://localhost:4318'
+    headers:
+      authorization: <YOUR_INGESTION_API_KEY>
+    compression: gzip
+
+  # gRPC setup (alternative)
+  otlp/hdx:
+    endpoint: 'localhost:4317'
+    headers:
+      authorization: <YOUR_API_INGESTION_KEY>
+    compression: gzip
+service:
+  telemetry:
+    metrics:
+      address: 0.0.0.0:9888 # Modified as 2 collectors running on same host
+  pipelines:
+    logs:
+      receivers: [filelog]
+      processors: [batch]
+      exporters: [otlphttp/hdx]

contribute/style-guide.md

@@ -112,12 +112,73 @@ SELECT * FROM system.contributors;
 \```
 ```
 
+Note: in the snippet above `\` is used only for formatting purposes in this guide.
+You should not include it when you write markdown.
+
 Code blocks:
 - Should always have a language defined immediately next to the opening 3
   backticks, without any space.
 - Have a title (optional) such as 'Query' or 'Response'
 - Use language `response` if it is for the result of a query.
 
+#### Importing code from files or URLs
+
+There are a few additional parameters you can include on a code block if you want
+to import code.
+
+To import from a file use `file=`:
+
+```text
+\```python file=code_snippets/integrations/example.py
+Code will be inserted here
+\```
+```
+
+When `yarn build` is run, the code from the file will be inserted as text into
+the code block.
+
+To import from a url use `url=`:
+
+```text
+\```python url=https://raw.githubusercontent.com/ClickHouse/clickhouse-connect/refs/heads/main/examples/pandas_examples.py
+Code will be inserted here
+\```
+```
+
+You should commit the code inserted to the snippet as we want people (or LLMs) 
+reading the markdown to be able to see the code. The advantage of importing code
+to snippets this way is that you can test your snippets externally or store them
+wherever you want.
+
+If you want to only import a section from a file, surround the section with `docs-start`
+and `docs-end` comments, for example:
+
+```python
+a = 200
+b = 33
+#docs-start
+if b > a:
+  print("b is greater than a")
+elif a == b:
+  print("a and b are equal")
+else:
+  print("a is greater than b")
+#docs-end
+```
+
+Only the code between those comments will be pulled.
+
+If you want to make multiple code snippets from one file then you can use the `snippet` parameter:
+
+```markdown
+
+\```python url=https://raw.githubusercontent.com/ClickHouse/clickhouse-connect/refs/heads/main/examples/pandas_examples.py snippet=1
+Code will be inserted here
+\```
+```
+
+You will then use `docs-start-1`, `docs-end-1` comments for the first snippet, `docs-start-2`, `docs-end-2` for the second snippet and so on.
+
 ### Highlighting
 
 You can highlight lines in a code block using the following keywords:

docs/_snippets/_users-and-roles-common.md

@@ -42,64 +42,73 @@ Create these tables and users to be used in the examples.
 
 #### Creating a sample database, table, and rows {#creating-a-sample-database-table-and-rows}
 
-1. Create a test database
+<VerticalStepper headerLevel="h5">
 
-   ```sql
-   CREATE DATABASE db1;
-   ```
+##### Create a test database {#create-a-test-database}
 
-2. Create a table
+```sql
+CREATE DATABASE db1;
+```
 
-   ```sql
-   CREATE TABLE db1.table1 (
-       id UInt64,
-       column1 String,
-       column2 String
-   )
-   ENGINE MergeTree
-   ORDER BY id;
-   ```
+##### Create a table {#create-a-table}
 
-3. Populate the table with sample rows
+```sql
+CREATE TABLE db1.table1 (
+   id UInt64,
+   column1 String,
+   column2 String
+)
+ENGINE MergeTree
+ORDER BY id;
+```
 
-   ```sql
-   INSERT INTO db1.table1
-       (id, column1, column2)
-   VALUES
-       (1, 'A', 'abc'),
-       (2, 'A', 'def'),
-       (3, 'B', 'abc'),
-       (4, 'B', 'def');
-   ```
+##### Populate the table with sample rows {#populate}
 
-4. Verify the table:
+```sql
+INSERT INTO db1.table1
+   (id, column1, column2)
+VALUES
+   (1, 'A', 'abc'),
+   (2, 'A', 'def'),
+   (3, 'B', 'abc'),
+   (4, 'B', 'def');
+```
 
-   ```sql
-   SELECT *
-   FROM db1.table1
-   ```
+##### Verify the table {#verify}
 
-   ```response
-   Query id: 475015cc-6f51-4b20-bda2-3c9c41404e49
+```sql title="Query"
+SELECT *
+FROM db1.table1
+```
 
-   ┌─id─┬─column1─┬─column2─┐
-   │  1 │ A       │ abc     │
-   │  2 │ A       │ def     │
-   │  3 │ B       │ abc     │
-   │  4 │ B       │ def     │
-   └────┴─────────┴─────────┘
-   ```
+```response title="Response"
+Query id: 475015cc-6f51-4b20-bda2-3c9c41404e49
 
-5. Create a regular user that will be used to demonstrate restrict access to certain columns:
+┌─id─┬─column1─┬─column2─┐
+│  1 │ A       │ abc     │
+│  2 │ A       │ def     │
+│  3 │ B       │ abc     │
+│  4 │ B       │ def     │
+└────┴─────────┴─────────┘
+```
 
-   ```sql
-   CREATE USER column_user IDENTIFIED BY 'password';
-   ```
+##### Create `column_user` {#create-a-user-with-restricted-access-to-columns}
 
-6. Create a regular user that will be used to demonstrate restricting access to rows with certain values:
-   ```sql
-   CREATE USER row_user IDENTIFIED BY 'password';
-   ```
+Create a regular user that will be used to demonstrate restrict access to certain columns:
+
+```sql
+CREATE USER column_user IDENTIFIED BY 'password';
+```
+
+##### Create `row_user` {#create-a-user-with-restricted-access-to-rows-with-certain-values}
+
+Create a regular user that will be used to demonstrate restricting access to rows with certain values:
+   
+```sql
+CREATE USER row_user IDENTIFIED BY 'password';
+```
+   
+</VerticalStepper>
 
 #### Creating roles {#creating-roles}
 

docs/best-practices/_snippets/_table_of_contents.md

@@ -0,0 +1,12 @@
+| Page                                                                                 | Description                                                                                             |
+|--------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
+| [Choosing a Primary Key](/best-practices/choosing-a-primary-key)                     | How to select primary keys that maximize query performance and minimize storage overhead.               |
+| [Select Data Types](/best-practices/select-data-types)                               | Choose optimal data types to reduce memory usage, improve compression, and accelerate queries.          |
+| [Use Materialized Views](/best-practices/use-materialized-views)                     | Leverage materialized views to pre-aggregate data and dramatically speed up analytical queries.         |
+| [Minimize and Optimize JOINs](/best-practices/minimize-optimize-joins)               | Best practices for using ClickHouse's `JOIN` capabilities efficiently.                                  |
+| [Choosing a Partitioning Key](/best-practices/choosing-a-partitioning-key)           | Select partitioning strategies that enable efficient data pruning and faster query execution.           |
+| [Selecting an Insert Strategy](/best-practices/selecting-an-insert-strategy)         | Optimize data ingestion throughput and reduce resource consumption with proper insert patterns.         |
+| [Data Skipping Indices](/best-practices/use-data-skipping-indices-where-appropriate) | Apply secondary indices strategically to skip irrelevant data blocks and accelerate filtered queries.   |
+| [Avoid Mutations](/best-practices/avoid-mutations)                                   | Design schemas and workflows that eliminate costly `UPDATE`/`DELETE` operations for better performance. |
+| [Avoid OPTIMIZE FINAL](/best-practices/avoid-optimize-final)                         | Prevent performance bottlenecks by understanding when `OPTIMIZE FINAL` hurts more than it helps.        |
+| [Use JSON where appropriate](/best-practices/use-json-where-appropriate)             | Balance flexibility and performance when working with semi-structured JSON data in ClickHouse.          |

docs/best-practices/index.md

@@ -6,19 +6,10 @@ hide_title: true
 description: 'Landing page for Best Practices section in ClickHouse'
 ---
 
+import TableOfContents from '@site/docs/best-practices/_snippets/_table_of_contents.md';
+
 # Best Practices in ClickHouse {#best-practices-in-clickhouse}
 
 This section provides the best practices you will want to follow to get the most out of ClickHouse.
 
-| Page                                                                 | Description                                                              |
-|----------------------------------------------------------------------|--------------------------------------------------------------------------|
-| [Choosing a Primary Key](/best-practices/choosing-a-primary-key)     | Guidance on selecting an effective Primary Key in ClickHouse.            |
-| [Select Data Types](/best-practices/select-data-types)               | Recommendations for choosing appropriate data types.                     |
-| [Use Materialized Views](/best-practices/use-materialized-views)     | When and how to benefit from materialized views.                         |
-| [Minimize and Optimize JOINs](/best-practices/minimize-optimize-joins)| Best practices for minimizing and optimizing JOIN operations.            |
-| [Choosing a Partitioning Key](/best-practices/choosing-a-partitioning-key) | How to choose and apply partitioning keys effectively.              |
-| [Selecting an Insert Strategy](/best-practices/selecting-an-insert-strategy) | Strategies for efficient data insertion in ClickHouse.             |
-| [Data Skipping Indices](/best-practices/use-data-skipping-indices-where-appropriate) | When to apply data skipping indices for performance gains.    |
-| [Avoid Mutations](/best-practices/avoid-mutations)                   | Reasons to avoid mutations and how to design without them.               |
-| [Avoid OPTIMIZE FINAL](/best-practices/avoid-optimize-final)         | Why `OPTIMIZE FINAL` can be costly and how to work around it.           |
-| [Use JSON where appropriate](/best-practices/use-json-where-appropriate) | Considerations for using JSON columns in ClickHouse.               |
+<TableOfContents/>