diff --git a/docs/user-guide/concepts/architecture.md b/docs/user-guide/concepts/architecture.md index 823f80865d..c73e77c118 100644 --- a/docs/user-guide/concepts/architecture.md +++ b/docs/user-guide/concepts/architecture.md @@ -14,7 +14,7 @@ level, there are three main components in GreptimeDB architecture: Datanode, Frontend and Metasrv. - [**Metasrv**](/contributor-guide/metasrv/overview.md) is the central command of - GreptimeDB cluster. In a typical cluster deployment, at least three nodes is required to + GreptimeDB cluster. In a typical cluster deployment, at least two nodes is required to setup a reliable Metasrv mini-cluster. Metasrv manages database and table information, including how data spread across the cluster and where to route requests to. It also keeps monitoring availability and performance of Datanodes, diff --git a/docs/user-guide/deployments-administration/configuration.md b/docs/user-guide/deployments-administration/configuration.md index 1020656798..3b7509e9bd 100644 --- a/docs/user-guide/deployments-administration/configuration.md +++ b/docs/user-guide/deployments-administration/configuration.md @@ -124,90 +124,79 @@ specifying protocol server addresses and other protocol-related options. The HTTP protocol configuration is available for all GreptimeDB components: `frontend`, `datanode`, `flownode`, and `metasrv`. ::: -Below is an example configuration with default values. +Below is an example configuration with the most commonly used options for all protocols that require attention. You can change the values or disable certain protocols in your configuration file. For example, to disable OpenTSDB protocol support, set the `enable` parameter to `false`. Note that HTTP and gRPC protocols cannot be disabled for the database to function correctly. ```toml [http] +# HTTP server binding address. Defaults to "127.0.0.1:4000" and therefore listens only on the loopback interface, so external hosts cannot reach it. +# For production, adjust this to an accessible IP address or domain based on your network configuration. addr = "127.0.0.1:4000" timeout = "30s" body_limit = "64MB" +# Maximum total memory (bytes) for all concurrent HTTP request bodies, rejects requests immediately when exceeded, defaults to "0" (unlimited) +# Limits memory usage for HTTP writes on a single node to prevent OOM from fast writes, only tracks HTTP body size, excludes decompression and internal processing memory +# Recommended to adjust after evaluating with `greptime_servers_http_memory_usage_bytes` metric, this option provides auxiliary protection only +max_total_body_memory = "0" +enable_cors = true +prom_validation_mode = "strict" [grpc] +# gRPC service binding address, defaults to "127.0.0.1:4001". +# The default configuration only listens on the loopback address and is inaccessible from external hosts. For production, adjust to an accessible IP address or domain based on your network configuration. bind_addr = "127.0.0.1:4001" +# Communication address registered with metasrv for other nodes to discover and connect, can be IP address or domain +# This value is only for external registration and does not affect the local listening `grpc.bind_addr`; it defaults to 127.0.0.1, so other nodes cannot connect via the metasrv-returned address. +# If left empty or unset, the server automatically uses the first network interface's IP address with the same port as `grpc.bind_addr` +server_addr = "127.0.0.1:4001" runtime_size = 8 +# Maximum total memory (bytes) for all concurrent gRPC request messages, rejects requests immediately when exceeded, defaults to "0" (unlimited) +# Limits memory usage for gRPC writes on a single node to prevent OOM from fast writes, only tracks gRPC message size, excludes decompression and internal processing memory +# Recommended to adjust after evaluating with `greptime_servers_grpc_memory_usage_bytes` metric, this option provides auxiliary protection only +max_total_message_memory = "0" +# To enable TLS for gRPC traffic, add a `[grpc.tls]` section; field details are listed in the TLS table below. [mysql] +# Whether to enable the MySQL protocol, defaults to true. enable = true +# MySQL server address. Defaults to "127.0.0.1:4002" and therefore listens only on the loopback interface, so external hosts cannot reach it. +# For production, adjust this to an accessible IP address or domain based on your network configuration. addr = "127.0.0.1:4002" -runtime_size = 2 - -[mysql.tls] -mode = "disable" -cert_path = "" -key_path = "" -watch = false +# To enable TLS for MySQL, add a `[mysql.tls]` section; field details are listed in the TLS table below. [postgres] +# Whether to enable the PostgreSQL protocol, defaults to true. enable = true +# PostgreSQL server address. Defaults to "127.0.0.1:4003" and therefore listens only on the loopback interface, so external hosts cannot reach it. +# For production, adjust this to an accessible IP address or domain based on your network configuration. addr = "127.0.0.1:4003" -runtime_size = 2 - -[postgres.tls] -mode = "disable" -cert_path = "" -key_path = "" -watch = false +# To enable TLS for PostgreSQL, add a `[postgres.tls]` section; field details are listed in the TLS table below. [opentsdb] +# Whether to enable OpenTSDB protocol support over the HTTP API, defaults to true. enable = true [influxdb] +# Whether to enable InfluxDB line protocol support over the HTTP API, defaults to true. +enable = true + +[jaeger] +# Whether to enable Jaeger trace ingestion over the HTTP API, defaults to true. enable = true [prom_store] +# Whether to enable Prometheus remote read/write endpoints, defaults to true. enable = true +# Whether to use Metric Engine when handling Prometheus remote write requests, defaults to true. with_metric_engine = true ``` - -The following table describes the options in detail: - -| Option | Key | Type | Description | -| ---------- | -------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| http | | | HTTP server options | -| | addr | String | Server address, "127.0.0.1:4000" by default | -| | timeout | String | HTTP request timeout, "30s" by default | -| | body_limit | String | HTTP max body size, "64MB" by default | -| | prom_validation_mode | String | Whether to check if strings are valid UTF-8 strings in Prometheus remote write requests. Available options: `strict`(reject any request with invalid UTF-8 strings), `lossy`(replace invalid characters with [UTF-8 REPLACEMENT CHARACTER U+FFFD, which looks like �](https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-23/#G24272)), `unchecked`(do not validate strings). | -| grpc | | | gRPC server options | -| | bind_addr | String | The address to bind the gRPC server, "127.0.0.1:4001" by default | -| | runtime_size | Integer | The number of server worker threads, 8 by default | -| | max_connection_age | String | Maximum lifetime of a gRPC connection that the server keeps it. Refer to ["MAX_CONNECTION_AGE"](https://grpc.io/docs/guides/keepalive/) for details. Defaults to not set. Example: "1h" for 1 hour, "30m" for 30 minutes | -| | flight_compression | String | Compression mode for frontend side Arrow IPC service. Available options: `none`: disable all compression, `transport`: only enable gRPC transport compression (zstd), `arrow_ipc`: only enable Arrow IPC compression (lz4), `all`: enable all compression. Default value is `none`.| -| mysql | | | MySQL server options | -| | enable | Boolean | Whether to enable MySQL protocol, true by default | -| | addr | String | Server address, "127.0.0.1:4002" by default | -| | runtime_size | Integer | The number of server worker threads, 2 by default | -| influxdb | | | InfluxDB Protocol options | -| | enable | Boolean | Whether to enable InfluxDB protocol in HTTP API, true by default | -| opentsdb | | | OpenTSDB Protocol options | -| | enable | Boolean | Whether to enable OpenTSDB protocol in HTTP API, true by default | -| prom_store | | | Prometheus remote storage options | -| | enable | Boolean | Whether to enable Prometheus Remote Write and read in HTTP API, true by default | -| | with_metric_engine | Boolean | Whether to use the metric engine on Prometheus Remote Write, true by default | -| postgres | | | PostgresSQL server options | -| | enable | Boolean | Whether to enable PostgresSQL protocol, true by default | -| | addr | String | Server address, "127.0.0.1:4003" by default | -| | runtime_size | Integer | The number of server worker threads, 2 by default | - -For MySQL, Postgres and gRPC interface, TLS can be configured to enable transport -layer security. +You can enable TLS for MySQL, Postgres and gRPC by adding `[mysql.tls]`, `[postgres.tls]` or `[grpc.tls]` sections to your configuration. The following table lists the shared fields for these TLS sections: | Option | Key | Type | Description | | ------------------------------------------| ----------- | ------- | ------------------------------------------------------------- | -| `mysql.tls`, `postgres.tls` or `grpc.tls` | | | TLS configuration for MySQL and Postgres | +| `mysql.tls`, `postgres.tls` or `grpc.tls` | | | TLS configuration for MySQL / Postgres / gRPC | | | `mode` | String | TLS mode, options are `disable`, `prefer` and `require` | | | `cert_path` | String | File path for TLS certificate | | | `key_path` | String | File path for TLS private key | @@ -236,13 +225,16 @@ The `storage` options are valid in datanode and standalone mode, which specify t GreptimeDB supports storing data in local file system, AWS S3 and compatible services (including MinIO, digitalocean space, Tencent Cloud Object Storage(COS), Baidu Object Storage(BOS) and so on), Azure Blob Storage and Aliyun OSS. -| Option | Key | Type | Description | -| ------- | ------------------------- | ------- | -------------------------------------------------------------------------------- | -| storage | | | Storage options | -| | type | String | Storage type, supports "File", "S3" and "Oss" etc. | -| File | | | Local file storage options, valid when type="File" | -| | data_home | String | Database storage root directory, "./greptimedb_data" by default | -| S3 | | | AWS S3 storage options, valid when type="S3" | +| Option | Key | Type | Description | +| ------- | ------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| storage | | | Storage options | +| | type | String | Storage type, supports "File", "S3", "Gcs", "Azblob", and "Oss" | +| | enable_read_cache | Boolean | Whether to enable read cache. Enabled by default when using object storage. Recommended for better performance with object storage | +| | cache_path | String | Read cache directory path for object storage. Defaults to `{data_home}/cache`. Set to empty string to disable cache | +| | cache_capacity | String | Maximum cache capacity. Recommended to set larger if disk space is sufficient. Examples: "10GB", "512MB" | +| File | | | Local file storage options, valid when type="File" | +| | data_home | String | Database storage root directory, "./greptimedb_data" by default | +| S3 | | | AWS S3 storage options, valid when type="S3" | | | name | String | The storage provider name, default is `S3` | | | bucket | String | The S3 bucket name | | | root | String | The root path in S3 bucket | @@ -291,6 +283,11 @@ bucket = "test_greptimedb" root = "/greptimedb" access_key_id = "" secret_access_key = "" + +# Read cache configuration +enable_read_cache = true +#+ cache_path = "" +cache_capacity = "10GB" ``` ### Storage http client @@ -299,12 +296,13 @@ secret_access_key = "" Only applied for storage types "S3", "Oss", "Azblob" and "Gcs". -| Key | Type | Default | Description | -| ------------------------ | ------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- | -| `pool_max_idle_per_host` | Integer | 1024 | The maximum idle connection per host allowed in the pool. | -| `connect_timeout` | String | "30s" (30 seconds) | The timeout for only the connect phase of a http client. | -| `timeout` | String | "30s" (30 seconds) | The total request timeout, applied from when the request starts connecting until the response body has finished. Also considered a total deadline. | -| `pool_idle_timeout` | String | "90s" (90 seconds) | The timeout for idle sockets being kept-alive. | +| Key | Type | Default | Description | +| ------------------------ | ------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `pool_max_idle_per_host` | Integer | 1024 | The maximum idle connection per host allowed in the pool. | +| `connect_timeout` | String | "30s" (30 seconds) | The timeout for only the connect phase of a http client. | +| `timeout` | String | "30s" (30 seconds) | The total request timeout, applied from when the request starts connecting until the response body has finished. Also considered a total deadline. | +| `pool_idle_timeout` | String | "90s" (90 seconds) | The timeout for idle sockets being kept-alive. | +| `skip_ssl_validation` | Boolean | `false` | Whether to skip SSL certificate verification. **Security Notice**: Setting `skip_ssl_validation = true` disables certificate verification, making connections vulnerable to man-in-the-middle attacks. Only use this in development or trusted private networks. Default: false | ### Storage engine provider @@ -348,19 +346,33 @@ root = "/greptimedb" access_key_id = "" secret_access_key = "" cache_capacity = "10GiB" -# cache_path = "/path/to/cache/home" +#+ cache_path = "/path/to/cache/home" ``` The `cache_path` specifies the home directory for storing cache files, while `cache_capacity` determines the maximum total file size allowed in the cache directory in bytes. You can disable the read cache by setting `cache_path` to an empty string. The default cache path is under the `{data_home}`. We recommend that you don't set the `cache_path` because the database can choose it automatically. -The write cache is no more experimental since `v0.12`. You can configure the cache size in the mito config if you don't want to use the default value. +The write cache is no longer experimental since `v0.12`. When using object storage, write cache is automatically enabled by default. You can configure it in the mito config: + ```toml [[region_engine]] [region_engine.mito] +# Write cache is automatically enabled when using object storage +enable_write_cache = true +#+ write_cache_path = "" write_cache_size = "10GiB" +#+ write_cache_ttl = "" ``` +Available write cache options: + +| Key | Type | Default | Description | +| -------------------- | ------- | ------- | ----------------------------------------------------------------------------------------------------------- | +| `enable_write_cache` | Boolean | `false` | Whether to enable write cache. Automatically enabled when using object storage. Recommended for performance | +| `write_cache_path` | String | `""` | File system path for write cache. Defaults to `{data_home}`. Leave empty for automatic path selection | +| `write_cache_size` | String | `5GiB` | Write cache capacity. Recommended to set larger if disk space is sufficient | +| `write_cache_ttl` | String | Unset | TTL (time-to-live) for write cache entries. Example: "7d" for 7 days | + Read [Performance Tuning Tips](/user-guide/deployments-administration/performance-tuning/performance-tuning-tips.md) for more detailed info. ### WAL options @@ -376,18 +388,30 @@ GreptimeDB supports two WAL storage options—Local WAL and Remote WAL. See the dir = "./greptimedb_data/logs" level = "info" enable_otlp_tracing = false -otlp_endpoint = "localhost:4317" +otlp_endpoint = "http://localhost:4318/v1/traces" +otlp_export_protocol = "http" append_stdout = true +log_format = "text" +max_log_files = 720 + [logging.tracing_sample_ratio] default_ratio = 1.0 ``` -- `dir`: log output directory. -- `level`: output log level, available log level are `info`, `debug`, `error`, `warn`, the default level is `info`. -- `enable_otlp_tracing`: whether to turn on tracing, not turned on by default. -- `otlp_endpoint`: Export the target endpoint of tracing using gRPC-based OTLP protocol, the default value is `localhost:4317`. -- `append_stdout`: Whether to append logs to stdout. Defaults to `true`. -- `tracing_sample_ratio`: This field can configure the sampling rate of tracing. How to use `tracing_sample_ratio`, please refer to [How to configure tracing sampling rate](/user-guide/deployments-administration/monitoring/tracing.md#guide-how-to-configure-tracing-sampling-rate). +Available logging options: + +| Key | Type | Default | Description | +| ----------------------- | ------- | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | +| `dir` | String | `./greptimedb_data/logs` | Log output directory. Set to empty string to disable file logging | +| `level` | String | `info` | Log level. Available: `info`, `debug`, `warn`, `error` | +| `log_format` | String | `text` | Log format. Available: `text`, `json` | +| `max_log_files` | Integer | `720` | Maximum number of log files to keep | +| `append_stdout` | Boolean | `true` | Whether to append logs to stdout | +| `enable_otlp_tracing` | Bool | `false` | Whether to enable OTLP tracing | +| `otlp_endpoint` | String | `http://localhost:4318/v1/traces` | OTLP tracing endpoint URL | +| `otlp_export_protocol` | String | `http` | OTLP export protocol. Available: `http`, `grpc` | +| `tracing_sample_ratio` | -- | -- | Tracing sampling configuration. See [tracing sampling rate](/user-guide/deployments-administration/monitoring/tracing.md#guide-how-to-configure-tracing-sampling-rate) | +| `tracing_sample_ratio.default_ratio` | Float | `1.0` | Default sampling ratio. Valid range: [0, 1]. 1 means all traces sampled, 0 means none | How to use distributed tracing, please reference [Tracing](/user-guide/deployments-administration/monitoring/tracing.md#tutorial-use-jaeger-to-trace-greptimedb) @@ -401,30 +425,54 @@ Frequently used options: [[region_engine]] [region_engine.mito] num_workers = 8 +worker_channel_size = 128 +worker_request_batch_size = 64 manifest_checkpoint_distance = 10 +#+ compress_manifest = false max_background_jobs = 4 +#+ max_background_flushes = 4 +#+ max_background_compactions = 2 +#+ max_background_purges = 8 auto_flush_interval = "1h" global_write_buffer_size = "1GB" global_write_buffer_reject_size = "2GB" sst_meta_cache_size = "128MB" vector_cache_size = "512MB" page_cache_size = "512MB" +selector_result_cache_size = "512MB" sst_write_buffer_size = "8MB" +parallel_scan_channel_size = 32 +max_concurrent_scan_files = 384 scan_parallelism = 0 +#+ min_compaction_interval = "0m" +#+ allow_stale_entries = false [region_engine.mito.index] aux_path = "" staging_size = "2GB" +staging_ttl = "7d" metadata_cache_size = "64MiB" content_cache_size = "128MiB" content_cache_page_size = "64KiB" +result_cache_size = "128MiB" [region_engine.mito.inverted_index] create_on_flush = "auto" create_on_compaction = "auto" apply_on_query = "auto" -mem_threshold_on_create = "64M" -intermediate_path = "" +mem_threshold_on_create = "auto" + +[region_engine.mito.fulltext_index] +create_on_flush = "auto" +create_on_compaction = "auto" +apply_on_query = "auto" +mem_threshold_on_create = "auto" + +[region_engine.mito.bloom_filter_index] +create_on_flush = "auto" +create_on_compaction = "auto" +apply_on_query = "auto" +mem_threshold_on_create = "auto" [region_engine.mito.memtable] type = "time_series" @@ -445,8 +493,14 @@ Available options: | Key | Type | Default | Descriptions | | ---------------------------------------- | ------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `num_workers` | Integer | `8` | Number of region workers. | +| `worker_channel_size` | Integer | `128` | Request channel size of each worker. | +| `worker_request_batch_size` | Integer | `64` | Max batch size for a worker to handle requests. | | `manifest_checkpoint_distance` | Integer | `10` | Number of meta action updated to trigger a new checkpoint for the manifest. | +| `compress_manifest` | Boolean | `false` | Whether to compress manifest and checkpoint file by gzip. | | `max_background_jobs` | Integer | `4` | Max number of running background jobs | +| `max_background_flushes` | Integer | Auto | Max number of running background flush jobs. Defaults to 1/2 of cpu cores. | +| `max_background_compactions` | Integer | Auto | Max number of running background compaction jobs. Defaults to 1/4 of cpu cores. | +| `max_background_purges` | Integer | Auto | Max number of running background purge jobs. Defaults to number of cpu cores. | | `auto_flush_interval` | String | `1h` | Interval to auto flush a region if it has not flushed yet. | | `global_write_buffer_size` | String | `1GB` | Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB. | | `global_write_buffer_reject_size` | String | `2GB` | Global write buffer size threshold to reject write requests. If not set, it's default to 2 times of `global_write_buffer_size` | @@ -455,19 +509,34 @@ Available options: | `page_cache_size` | String | `512MB` | Cache size for pages of SST row groups. Setting it to 0 to disable the cache.
If not set, it's default to 1/8 of OS memory. | | `selector_result_cache_size` | String | `512MB` | Cache size for time series selector (e.g. `last_value()`). Setting it to 0 to disable the cache.
If not set, it's default to 1/8 of OS memory. | | `sst_write_buffer_size` | String | `8MB` | Buffer size for SST writing. | +| `parallel_scan_channel_size` | Integer | `32` | Capacity of the channel to send data from parallel scan tasks to the main task. | +| `max_concurrent_scan_files` | Integer | `384` | Maximum number of SST files to scan concurrently. | | `scan_parallelism` | Integer | `0` | Parallelism to scan a region (default: 1/4 of cpu cores).
- `0`: using the default value (1/4 of cpu cores).
- `1`: scan in current thread.
- `n`: scan in parallelism n. | +| `min_compaction_interval` | String | `0m` | Minimum time interval between two compactions. Default is 0 (no restrictions). | +| `allow_stale_entries` | Boolean | `false` | Whether to allow stale WAL entries read during replay. | | `index` | -- | -- | The options for index in Mito engine. | | `index.aux_path` | String | `""` | Auxiliary directory path for the index in the filesystem. This path is used to store intermediate files for creating the index and staging files for searching the index. It defaults to `{data_home}/index_intermediate`. The default name for this directory is `index_intermediate` for backward compatibility. This path contains two subdirectories: `__intm` for storing intermediate files used during index creation, and `staging` for storing staging files used during index searching. | | `index.staging_size` | String | `2GB` | The maximum capacity of the staging directory. | +| `index.staging_ttl` | String | `7d` | TTL of the staging directory. Defaults to 7 days. Set to "0s" to disable TTL. | | `index.metadata_cache_size` | String | `64MiB` | Cache size for index metadata. | | `index.content_cache_size` | String | `128MiB` | Cache size for index content. | | `index.content_cache_page_size` | String | `64KiB` | Page size for index content cache. | +| `index.result_cache_size` | String | `128MiB` | Cache size for index result. | | `inverted_index` | -- | -- | The options for inverted index in Mito engine. | | `inverted_index.create_on_flush` | String | `auto` | Whether to create the index on flush.
- `auto`: automatically
- `disable`: never | | `inverted_index.create_on_compaction` | String | `auto` | Whether to create the index on compaction.
- `auto`: automatically
- `disable`: never | | `inverted_index.apply_on_query` | String | `auto` | Whether to apply the index on query
- `auto`: automatically
- `disable`: never | -| `inverted_index.mem_threshold_on_create` | String | `64M` | Memory threshold for performing an external sort during index creation.
Setting to empty will disable external sorting, forcing all sorting operations to happen in memory. | -| `inverted_index.intermediate_path` | String | `""` | File system path to store intermediate files for external sorting (default `{data_home}/index_intermediate`). | +| `inverted_index.mem_threshold_on_create` | String | `auto` | Memory threshold for index creation.
- `auto`: automatically determine based on system memory (default)
- `unlimited`: no memory limit
- `[size]` e.g. `64MB`: fixed memory threshold | +| `fulltext_index` | -- | -- | The options for full-text index in Mito engine. | +| `fulltext_index.create_on_flush` | String | `auto` | Whether to create the index on flush.
- `auto`: automatically (default)
- `disable`: never | +| `fulltext_index.create_on_compaction` | String | `auto` | Whether to create the index on compaction.
- `auto`: automatically (default)
- `disable`: never | +| `fulltext_index.apply_on_query` | String | `auto` | Whether to apply the index on query.
- `auto`: automatically (default)
- `disable`: never | +| `fulltext_index.mem_threshold_on_create` | String | `auto` | Memory threshold for index creation.
- `auto`: automatically determine based on system memory (default)
- `unlimited`: no memory limit
- `[size]` e.g. `64MB`: fixed memory threshold | +| `bloom_filter_index` | -- | -- | The options for bloom filter index in Mito engine. | +| `bloom_filter_index.create_on_flush` | String | `auto` | Whether to create the bloom filter on flush.
- `auto`: automatically (default)
- `disable`: never | +| `bloom_filter_index.create_on_compaction`| String | `auto` | Whether to create the bloom filter on compaction.
- `auto`: automatically (default)
- `disable`: never | +| `bloom_filter_index.apply_on_query` | String | `auto` | Whether to apply the bloom filter on query.
- `auto`: automatically (default)
- `disable`: never | +| `bloom_filter_index.mem_threshold_on_create` | String | `auto` | Memory threshold for bloom filter creation.
- `auto`: automatically determine based on system memory (default)
- `unlimited`: no memory limit
- `[size]` e.g. `64MB`: fixed memory threshold | | `memtable.type` | String | `time_series` | Memtable type.
- `time_series`: time-series memtable
- `partition_tree`: partition tree memtable (experimental) | | `memtable.index_max_keys_per_shard` | Integer | `8192` | The max number of keys in one shard.
Only available for `partition_tree` memtable. | | `memtable.data_freeze_threshold` | Integer | `32768` | The max rows of data inside the actively writing buffer in one shard.
Only available for `partition_tree` memtable. | @@ -478,20 +547,33 @@ Available options: The `meta_client` options are valid in `datanode` and `frontend` mode, which specify the Metasrv client information. ```toml +[meta_client] metasrv_addrs = ["127.0.0.1:3002"] timeout = "3s" +heartbeat_timeout = "500ms" connect_timeout = "1s" ddl_timeout = "10s" tcp_nodelay = true -``` -The `meta_client` configures the Metasrv client, including: +# Metadata cache configuration +metadata_cache_max_capacity = 100000 +metadata_cache_ttl = "10m" +metadata_cache_tti = "5m" +``` -- `metasrv_addrs`: The Metasrv address list. -- `timeout`: operation timeout, `3s` by default. -- `connect_timeout`, connect server timeout, `1s` by default. -- `ddl_timeout`, DDL execution timeout, `10s` by default. -- `tcp_nodelay`, `TCP_NODELAY` option for accepted connections, true by default. +Available meta client options: + +| Key | Type | Default | Description | +| ----------------------------- | ------- | --------- | --------------------------------------------------------- | +| `metasrv_addrs` | Array | -- | The Metasrv address list | +| `timeout` | String | `3s` | Operation timeout | +| `heartbeat_timeout` | String | `500ms` | Heartbeat timeout | +| `connect_timeout` | String | `1s` | Connect server timeout | +| `ddl_timeout` | String | `10s` | DDL execution timeout | +| `tcp_nodelay` | Boolean | `true` | `TCP_NODELAY` option for accepted connections | +| `metadata_cache_max_capacity` | Integer | `100000` | Maximum capacity of the metadata cache | +| `metadata_cache_ttl` | String | `10m` | TTL (time-to-live) of the metadata cache | +| `metadata_cache_tti` | String | `5m` | TTI (time-to-idle) of the metadata cache | ### Monitor metrics options diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/concepts/architecture.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/concepts/architecture.md index d3df2a731a..68d8d81059 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/concepts/architecture.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/concepts/architecture.md @@ -11,7 +11,7 @@ description: 介绍 GreptimeDB 的基础架构，包括 Metasrv、Frontend 和 D 为了形成一个强大的数据库集群，并控制其复杂性，GreptimeDB 架构中有三个主要组成部分：Metasrv，Frontend 和 Datanode。 -- [**Metasrv**（元数据服务器）](/contributor-guide/metasrv/overview.md) 控制着 GreptimeDB 集群的核心命令。在典型的部署结构中，至少需要三个节点才能建立一个可靠的 Metasrv 小集群。Metasrv 管理着数据库和表的信息，包括数据如何在集群中传递、请求的转发地址等。它还负责监测 `Datanode` 的可用性和性能，以确保路由表的最新状态和有效性。 +- [**Metasrv**（元数据服务器）](/contributor-guide/metasrv/overview.md) 控制着 GreptimeDB 集群的核心命令。在典型的部署结构中，至少需要两个节点才能建立一个可靠的 Metasrv 小集群。Metasrv 管理着数据库和表的信息，包括数据如何在集群中传递、请求的转发地址等。它还负责监测 `Datanode` 的可用性和性能，以确保路由表的最新状态和有效性。 - [**Frontend**（前端服务）](/contributor-guide/frontend/overview.md) 作为无状态的组件，可以根据需求进行伸缩扩容。它负责接收请求并鉴权，将多种协议转化为 GreptimeDB 集群的内部 gRPC 协议，并根据 Metasrv 中的表的分片路由信息将请求转发到相应的 Datanode。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/concepts/data-model.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/concepts/data-model.md index c4eabd7a14..78592f0072 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/concepts/data-model.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/concepts/data-model.md @@ -7,7 +7,7 @@ description: 介绍 GreptimeDB 的数据模型，包括表的结构、列类型 ## 模型 -GreptimeDB 使用时序表来进行数据的组织、压缩和过期管理。数据模型主要基于关系型数据库中的表模型，同时考虑到了指标（metrics）、日志（logs）及链路追踪（traces）数据的特点。 +GreptimeDB 使用时序表来进行数据的组织、压缩和生命周期管理。数据模型主要基于关系型数据库中的表模型，同时考虑到了指标（metrics）、日志（logs）及链路追踪（traces）数据的特点。 GreptimeDB 中的所有数据都被组织成具有名称的表，每个表中的数据项由三种语义类型的列组成：`Tag`、`Timestamp` 和 `Field`。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/concepts/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/concepts/overview.md index a79689f752..9c43088aa2 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/concepts/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/concepts/overview.md @@ -7,7 +7,7 @@ description: 概述 GreptimeDB 的特点和优势，并提供相关文档链接 - [Why GreptimeDB](./why-greptimedb.md)：介绍了 GreptimeDB 的特点和优势，包括其对指标、日志和链路追踪数据的统一处理，云原生和灵活架构允许在各种环境中部署，从嵌入式到云平台等。GreptimeDB 还具有成本优势、高性能和开发者友好等特点。 - [数据模型](./data-model.md)：介绍了 GreptimeDB 的数据模型，包括表的模式、索引列等。 -- [基础架构](./architecture.md)：获取 GreptimeDB 的云原生架构。 +- [基础架构](./architecture.md)：了解 GreptimeDB 的云原生架构。 - [存储位置](./storage-location.md)：介绍了 GreptimeDB 的存储位置，包括本地磁盘、HDFS、AWS S3 和阿里云 OSS 等云对象存储。 - [核心概念](./key-concepts.md)：介绍了 GreptimeDB 的核心概念，包括表、时间索引约束、表 Region 和数据类型等。 - [关键特性](./features-that-you-concern.md): 介绍了 TSDB 用户较为关心的指标（metrics）、日志（logs）和事件（events）数据库的特性。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/concepts/why-greptimedb.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/concepts/why-greptimedb.md index e2896e7b06..eb4dffdbfe 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/concepts/why-greptimedb.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/concepts/why-greptimedb.md @@ -21,13 +21,13 @@ GreptimeDB 通过以下方式统一处理指标、日志和链路追踪： ## 基于对象存储的成本优势 -GreptimeDB 采用云对象存储（如 AWS S3、阿里云 OSS 和 Azure Blob Storage 等）作为存储层，与传统存储方案相比显著降低了成本。通过优化的列式存储和先进的压缩算法，实现了高达 50 倍的成本效率，而按需付费模式的 [GreptimeCloud](https://greptime.com/product/cloud) 确保您只需为实际使用的资源付费。 +GreptimeDB 采用云对象存储（如 AWS S3、阿里云 OSS 和 Azure Blob Storage 等）作为存储层，与传统存储方案相比显著降低了成本。通过优化的列式存储和先进的压缩算法，实现了高达 50 倍的成本效率，而按需付费模式的 [GreptimeCloud](https://greptime.com/product/cloud) 确保你只需为实际使用的资源付费。 ## 高性能 在性能优化方面，GreptimeDB 运用了多种技术，如 LSM Tree、数据分片和灵活的 WAL 选项（本地磁盘或 Kafka 等分布式服务），以处理大规模可观测数据的写入。 -GreptimeDB 使用纯 Rust 编写，具有卓越的性能和可靠性。强大而快速的查询引擎由向量化执行和分布式并行处理（感谢 [Apache DataFusion](https://datafusion.apache.org/)）驱动，并结合了丰富的[索引选项](/user-guide/manage-data/data-index.md)，例如倒排索引、跳数索引和全文索引等。GreptimeDB将智能索引和大规模并行处理 (MPP) 结合在一起，以提升查询过程中数据剪枝和过滤的效率。 +GreptimeDB 使用纯 Rust 编写，具有卓越的性能和可靠性。强大而快速的查询引擎由向量化执行和分布式并行处理（感谢 [Apache DataFusion](https://datafusion.apache.org/)）驱动，并结合了丰富的[索引选项](/user-guide/manage-data/data-index.md)，例如倒排索引、跳数索引和全文索引等。GreptimeDB 将智能索引和大规模并行处理 (MPP) 结合在一起，以提升查询过程中数据剪枝和过滤的效率。 GreptimeDB 在[ClickHouse 的 JSONBench 测试中 Cold Run 斩获第一！](https://greptime.cn/blogs/2025-03-18-json-benchmark-greptimedb)，更多报告请参阅[性能测试报告](https://greptime.cn/blogs/2024-09-09-report-summary)。 @@ -77,7 +77,7 @@ GreptimeDB 支持多种数据摄入协议，从而实现与现有可观测性技 - **SQL**：用于实时查询、复杂分析和数据库管理 - **PromQL**：原生支持实时指标查询和 Grafana 集成 -GreptimeDB 与您的现有可观测性技术栈无缝集成，同时保持高性能和灵活性。 +GreptimeDB 与你的现有可观测性技术栈无缝集成，同时保持高性能和灵活性。 ![Greptime Ecosystem](/greptime-ecosystem.png) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md index 49aa67d0d0..cfd6d4d2f8 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/deployments-administration/configuration.md @@ -117,86 +117,75 @@ GREPTIMEDB_METASRV__META_CLIENT__METASRV_ADDRS=127.0.0.1:3001,127.0.0.1:3002,127 HTTP 协议配置适用于所有 GreptimeDB 组件：`frontend`、`datanode`、`flownode` 和 `metasrv`。 ::: -下面的示例配置包含了所有协议选项的默认值。 +下面的示例配置包含了所有协议的最常用的并且需要重点关注的选项。 你可以在配置文件中更改这些值或禁用某些协议。 例如禁用 OpenTSDB 协议支持，可以将 `enable` 参数设置为 `false`。 请注意，为了保障数据库的正常工作，无法禁用 HTTP 和 gRPC 协议。 ```toml [http] +# 服务器 HTTP 绑定地址，默认为 "127.0.0.1:4000"，仅监听本地回环地址，外部主机无法访问 +# 生产环境请根据实际网络情况调整为可访问的 IP 地址或者域名 addr = "127.0.0.1:4000" -timeout = "30s" -body_limit = "64MB" +# 所有并发 HTTP 请求体的最大总内存（字节），超过限制时立即拒绝请求，默认为 "0"（无限制） +# 限制单节点 HTTP 写入时的内存占用，避免写入过快触发 OOM，仅统计 HTTP body 大小，不含解压和内部处理内存 +# 建议结合 `greptime_servers_http_memory_usage_bytes` 监控评估后再调整，此选项仅提供辅助防护 +max_total_body_memory = "0" [grpc] +# gRPC 服务绑定地址，默认为 "127.0.0.1:4001" +# 默认配置仅监听本地回环地址，外部主机无法访问。生产环境请根据实际网络配置调整为合适的 IP 地址或域名 bind_addr = "127.0.0.1:4001" -runtime_size = 8 +# 向 metasrv 注册的通信地址，供其他节点通过 metasrv 获取并建立连接，可配置 IP 地址或域名 +# 该值仅用于对外注册，不影响本地监听的 `grpc.bind_addr`；默认为 127.0.0.1，其他节点无法根据 metasrv 返回的地址连接 +# 若留空或未设置，服务器将自动使用主机上第一个网络接口的 IP 地址，端口号与 `grpc.bind_addr` 保持一致 +server_addr = "127.0.0.1:4001" +# 所有并发 gRPC 请求消息的最大总内存（字节），超过限制时立即拒绝请求，默认为 "0"（无限制） +# 限制单节点 gRPC 写入时的内存占用，避免写入过快触发 OOM，仅统计 gRPC 消息大小，不含解压和内部处理内存 +# 建议结合 `greptime_servers_grpc_memory_usage_bytes` 监控评估后再调整，此选项仅提供辅助防护 +max_total_message_memory = "0" +# 如需启用 TLS，可添加 `[grpc.tls]` 段，字段说明见下方 TLS 配置表 [mysql] +# 是否启用 MySQL 协议，默认为 true enable = true +# MySQL 协议服务器地址，默认为 "127.0.0.1:4002"，仅监听本地回环地址，外部主机无法访问 +# 生产环境请根据实际网络情况调整为可访问的 IP 地址或域名 addr = "127.0.0.1:4002" -runtime_size = 2 - -[mysql.tls] -mode = "disable" -cert_path = "" -key_path = "" +# 如需启用 TLS，可添加 `[mysql.tls]` 段，字段说明见下方 TLS 配置表 [postgres] +# 是否启用 PostgreSQL 协议，默认为 true enable = true +# PostgreSQL 协议服务器地址，默认为 "127.0.0.1:4003"，仅监听本地回环地址，外部主机无法访问 +# 生产环境请根据实际网络情况调整为可访问的 IP 地址或域名 addr = "127.0.0.1:4003" -runtime_size = 2 - -[postgres.tls] -mode = "disable" -cert_path = "" -key_path = "" +# 如需启用 TLS，可添加 `[postgres.tls]` 段，字段说明见下方 TLS 配置表 [opentsdb] +# 是否在 HTTP API 中启用 OpenTSDB 协议，默认为 true enable = true [influxdb] +# 是否在 HTTP API 中启用 InfluxDB Line Protocol，默认为 true +enable = true + +[jaeger] +# 是否在 HTTP API 中启用 Jaeger Trace 写入，默认为 true enable = true [prom_store] +# 是否在 HTTP API 中启用 Prometheus 远程读写，默认为 true enable = true +# 是否在 Prometheus 远程写入中使用 Metric Engine，默认为 true +with_metric_engine = true ``` -下表描述了每个选项的详细信息： - -| 选项 | 键 | 类型 | 描述 | -| ---------- | ------------------ | ------ | ------------------------------------------------------------ | -| http | | | HTTP 服务器选项 | -| | addr | 字符串 | 服务器地址，默认为 "127.0.0.1:4000" | -| | timeout | 字符串 | HTTP 请求超时时间，默认为 "30s" | -| | body_limit | 字符串 | HTTP 最大体积大小，默认为 "64MB" | -| | prom_validation_mode | 字符串 | 在 Prometheus Remote Write 协议中中是否检查字符串是否为有效的 UTF-8 字符串。可用选项：`strict`（拒绝任何包含无效 UTF-8 字符串的请求），`lossy`（用 [UTF-8 REPLACEMENT CHARACTER](https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-23/#G24272)（即 `�` ） 替换无效字符），`unchecked`（不验证字符串有效性）。 | -| grpc | | | gRPC 服务器选项 | -| | bind_addr | 字符串 | gRPC 服务绑定地址，默认为 "127.0.0.1:4001" | -| | runtime_size | 整数 | 服务器工作线程数量，默认为 8 | -| | max_connection_age | 字符串 | gRPC 连接在服务端保持的最长时间。参见 ["MAX_CONNECTION_AGE"](https://grpc.io/docs/guides/keepalive/)。默认不设置。示例："1h" 表示 1 小时，"30m" 表示 30 分钟 | -| | flight_compression | 字符串 | Frontend 的 Arrow IPC 服务的压缩模式。可用选项：`none`：禁用所有压缩，`transport`：仅启用 gRPC 传输压缩（zstd），`arrow_ipc`：仅启用 Arrow IPC 压缩（lz4），`all`：启用所有压缩。默认值为 `none`。| -| mysql | | | MySQL 服务器选项 | -| | enable | 布尔值 | 是否启用 MySQL 协议，默认为 true | -| | addr | 字符串 | 服务器地址，默认为 "127.0.0.1:4002" | -| | runtime_size | 整数 | 服务器工作线程数量，默认为 2 | -| influxdb | | | InfluxDB 协议选项 | -| | enable | 布尔值 | 是否在 HTTP API 中启用 InfluxDB 协议，默认为 true | -| opentsdb | | | OpenTSDB 协议选项 | -| | enable | 布尔值 | 是否启用 OpenTSDB 协议，默认为 true | -| prom_store | | | Prometheus 远程存储选项 | -| | enable | 布尔值 | 是否在 HTTP API 中启用 Prometheus 远程读写，默认为 true | -| | with_metric_engine | 布尔值 | 是否在 Prometheus 远程写入中使用 Metric Engine，默认为 true | -| postgres | | | PostgresSQL 服务器选项 | -| | enable | 布尔值 | 是否启用 PostgresSQL 协议，默认为 true | -| | addr | 字符串 | 服务器地址，默认为 "127.0.0.1:4003" | -| | runtime_size | 整数 | 服务器工作线程数量，默认为 2 | - -对 MySQL，Postgres 和 gRPC 接口，我们支持 TLS 配置 +对 MySQL、Postgres 和 gRPC 接口，可以通过在配置中添加 `[mysql.tls]`、`[postgres.tls]` 或 `[grpc.tls]` 段来启用 TLS，以下表格列出了这些 TLS 段共享的字段 | Option | Key | Type | Description | |------------------------------------------|-------------|---------|--------------------------------------------------| -| `mysql.tls`，`postgres.tls` 或 `grpc.tls` | | | MySQL 或 Postgres 的 TLS 配置 | +| `mysql.tls`，`postgres.tls` 或 `grpc.tls` | | | MySQL / Postgres / gRPC 的 TLS 配置段 | | | `mode` | String | TLS 模式，支持 `disable`, `prefer` and `require` | | | `cert_path` | String | TLS 证书文件路径 | | | `key_path` | String | TLS 私钥文件路径 | @@ -225,13 +214,16 @@ parallelism = 0 GreptimeDB 支持将数据保存在本地文件系统，AWS S3 以及其兼容服务（比如 MinIO、digitalocean space、腾讯 COS、百度对象存储（BOS）等），Azure Blob Storage 和阿里云 OSS。 -| 选项 | 键 | 类型 | 描述 | -| ------- | ----------------- | ------ | --------------------------------------------------- | -| storage | | | 存储选项 | -| | type | 字符串 | 存储类型，支持 "File"，"S3" 和 "Oss" 等。 | -| File | | | 本地文件存储选项，当 type="File" 时有效 | -| | data_home | 字符串 | 数据库存储根目录，默认为 "./greptimedb_data" | -| S3 | | | AWS S3 存储选项，当 type="S3" 时有效 | +| 选项 | 键 | 类型 | 描述 | +| ------- | ----------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| storage | | | 存储选项 | +| | type | 字符串 | 存储类型，支持 "File"、"S3"、"Gcs"、"Azblob" 和 "Oss" | +| | enable_read_cache | 布尔值 | 是否启用读缓存。使用对象存储时默认启用。推荐启用以获得更好的性能 | +| | cache_path | 字符串 | 对象存储的读缓存目录路径。默认为 `{data_home}/cache`。设置为空字符串可禁用缓存 | +| | cache_capacity | 字符串 | 最大缓存容量。如果磁盘空间充足，建议设置更大的值。示例："10GB"、"512MB" | +| File | | | 本地文件存储选项，当 type="File" 时有效 | +| | data_home | 字符串 | 数据库存储根目录，默认为 "./greptimedb_data" | +| S3 | | | AWS S3 存储选项，当 type="S3" 时有效 | | | name | 字符串 | 存储提供商名字，默认为 `S3` | | | bucket | 字符串 | S3 桶名称 | | | root | 字符串 | S3 桶中的根路径 | @@ -271,7 +263,7 @@ type = "File" data_home = "./greptimedb_data/" ``` -s3 配置范例： +S3 配置范例： ```toml [storage] @@ -280,6 +272,11 @@ bucket = "test_greptimedb" root = "/greptimedb" access_key_id = "" secret_access_key = "" + +# 读缓存配置（推荐启用以获得更好的性能） +enable_read_cache = true +#+ cache_path = "" +cache_capacity = "10GB" ``` ### 存储服务的 http 客户端 @@ -288,12 +285,13 @@ secret_access_key = "" 仅当存储服务类型是“S3”，“Oss”，“Azblob”或“Gcs”时生效。 -| Key | 类型 | 默认值 | 含义 | -|--------------------------|-----|------------|-------------------------------------------------------------| -| `pool_max_idle_per_host` | 数字 | 1024 | http 连接池中对每个 host 的最大空闲连接数。 | -| `connect_timeout` | 字符串 | “30s”（30 秒） | http 客户端在进行连接时的超时 | -| `timeout` | 字符串 | “30s”（30 秒） | 总的 http 请求超时，包括了从建立连接到接收完返回值为止的时间。也可视为一个请求从开始到结束的一个完整的截止时间。 | -| `pool_idle_timeout` | 字符串 | “90s”（90 秒） | 对空闲连接进行保活（ "keep-alive" ）的超时。 | +| Key | 类型 | 默认值 | 含义 | +| ------------------------ | ------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `pool_max_idle_per_host` | 整数 | 1024 | http 连接池中对每个 host 的最大空闲连接数 | +| `connect_timeout` | 字符串 | "30s"（30 秒） | http 客户端在进行连接时的超时 | +| `timeout` | 字符串 | "30s"（30 秒） | 总的 http 请求超时，包括了从建立连接到接收完返回值为止的时间。也可视为一个请求从开始到结束的一个完整的截止时间 | +| `pool_idle_timeout` | 字符串 | "90s"（90 秒） | 对空闲连接进行保活（"keep-alive"）的超时 | +| `skip_ssl_validation` | 布尔值 | `false` | 是否跳过 SSL 证书验证。**安全提示**：设置 `skip_ssl_validation = true` 会禁用证书验证，使连接容易受到中间人攻击。仅在开发环境或可信的私有网络中使用此选项。默认值：false | ### 存储引擎提供商 @@ -343,15 +341,28 @@ cache_capacity = "10GiB" `cache_path` 指定存储缓存文件的本地目录，而 `cache_capacity` 则决定缓存目录中允许的最大文件总大小（以字节为单位）。你可以通过将 `cache_path` 设置为空字符串来禁用读取缓存。默认的缓存目录位于 `{data_home}` 目录下。我们建议你不用配置 `cache_path`，因为数据库会自动设置该目录。 -自 `v0.12` 之后，写入缓存不再是实验性的功能。你可以通过修改 mito 的配置调整缓存的大小 +自 `v0.12` 之后，写入缓存不再是实验性的功能。使用对象存储时，写入缓存会自动启用。你可以在 mito 配置中进行调整： ```toml [[region_engine]] [region_engine.mito] +# 使用对象存储时会自动启用写入缓存 +enable_write_cache = true +#+ write_cache_path = "" write_cache_size = "10GiB" +#+ write_cache_ttl = "" ``` +可用的写入缓存选项： + +| 键 | 类型 | 默认值 | 描述 | +| -------------------- | ------ | ------- | ----------------------------------------------------------------------------- | +| `enable_write_cache` | 布尔值 | `false` | 是否启用写入缓存。使用对象存储时会自动启用。推荐启用以获得更好的性能 | +| `write_cache_path` | 字符串 | `""` | 写入缓存的文件系统路径。默认为 `{data_home}`。留空则自动选择路径 | +| `write_cache_size` | 字符串 | `5GiB` | 写入缓存容量。如果磁盘空间充足，建议设置更大的值 | +| `write_cache_ttl` | 字符串 | 未设置 | 写入缓存条目的 TTL（生存时间）。示例："7d" 表示 7 天 | + 更详细的信息请参阅[性能调优技巧](/user-guide/deployments-administration/performance-tuning/performance-tuning-tips.md)。 @@ -368,18 +379,30 @@ GreptimeDB 支持两种 WAL 存储方式：本地 WAL 和 Remote WAL。关于它 dir = "./greptimedb_data/logs" level = "info" enable_otlp_tracing = false -otlp_endpoint = "localhost:4317" +otlp_endpoint = "http://localhost:4318/v1/traces" +otlp_export_protocol = "http" append_stdout = true +log_format = "text" +max_log_files = 720 + [logging.tracing_sample_ratio] default_ratio = 1.0 ``` -- `dir`: log 输出目录。 -- `level`: log 输出的日志等级，日志等级有 `info`, `debug`, `error`, `warn`，默认等级为 `info`。 -- `enable_otlp_tracing`：是否打开分布式追踪，默认不开启。 -- `otlp_endpoint`：使用基于 gRPC 的 OTLP 协议导出 tracing 的目标端点，默认值为 `localhost:4317`。 -- `append_stdout`：是否将日志打印到 stdout。默认是`true`。 -- `tracing_sample_ratio`：该字段可以配置 tracing 的采样率，如何使用 `tracing_sample_ratio`，请参考 [如何配置 tracing 采样率](/user-guide/deployments-administration/monitoring/tracing.md#指南如何配置-tracing-采样率)。 +可用的 logging 选项： + +| 键 | 类型 | 默认值 | 描述 | +| ------------------------------------- | ------ | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | +| `dir` | 字符串 | `./greptimedb_data/logs` | 日志输出目录。设置为空字符串可禁用文件日志 | +| `level` | 字符串 | `info` | 日志等级。可用值：`info`、`debug`、`warn`、`error` | +| `log_format` | 字符串 | `text` | 日志格式。可用值：`text`、`json` | +| `max_log_files` | 整数 | `720` | 保留的最大日志文件数量 | +| `append_stdout` | 布尔值 | `true` | 是否将日志输出到 stdout | +| `enable_otlp_tracing` | 布尔值 | `false` | 是否启用 OTLP tracing | +| `otlp_endpoint` | 字符串 | `http://localhost:4318/v1/traces` | OTLP tracing 端点 URL | +| `otlp_export_protocol` | 字符串 | `http` | OTLP 导出协议。可用值：`http`、`grpc` | +| `tracing_sample_ratio` | -- | -- | Tracing 采样配置。参见 [tracing 采样率](/user-guide/deployments-administration/monitoring/tracing.md#指南如何配置-tracing-采样率) | +| `tracing_sample_ratio.default_ratio` | 浮点数 | `1.0` | 默认采样比率。有效范围：[0, 1]。1 表示采样所有 trace，0 表示不采样 | 如何使用分布式追踪，请参考 [Tracing](/user-guide/deployments-administration/monitoring/tracing.md#教程使用-jaeger-追踪-greptimedb-调用链路) @@ -393,30 +416,54 @@ datanode 和 standalone 在 `[region_engine]` 部分可以配置不同存储引 [[region_engine]] [region_engine.mito] num_workers = 8 +worker_channel_size = 128 +worker_request_batch_size = 64 manifest_checkpoint_distance = 10 +#+ compress_manifest = false max_background_jobs = 4 +#+ max_background_flushes = 4 +#+ max_background_compactions = 2 +#+ max_background_purges = 8 auto_flush_interval = "1h" global_write_buffer_size = "1GB" global_write_buffer_reject_size = "2GB" sst_meta_cache_size = "128MB" vector_cache_size = "512MB" page_cache_size = "512MB" +selector_result_cache_size = "512MB" sst_write_buffer_size = "8MB" +parallel_scan_channel_size = 32 +max_concurrent_scan_files = 384 scan_parallelism = 0 +#+ min_compaction_interval = "0m" +#+ allow_stale_entries = false [region_engine.mito.index] aux_path = "" staging_size = "2GB" +staging_ttl = "7d" metadata_cache_size = "64MiB" content_cache_size = "128MiB" content_cache_page_size = "64KiB" +result_cache_size = "128MiB" [region_engine.mito.inverted_index] create_on_flush = "auto" create_on_compaction = "auto" apply_on_query = "auto" -mem_threshold_on_create = "64M" -intermediate_path = "" +mem_threshold_on_create = "auto" + +[region_engine.mito.fulltext_index] +create_on_flush = "auto" +create_on_compaction = "auto" +apply_on_query = "auto" +mem_threshold_on_create = "auto" + +[region_engine.mito.bloom_filter_index] +create_on_flush = "auto" +create_on_compaction = "auto" +apply_on_query = "auto" +mem_threshold_on_create = "auto" [region_engine.mito.memtable] type = "time_series" @@ -437,8 +484,14 @@ fork_dictionary_bytes = "1GiB" | 键 | 类型 | 默认值 | 描述 | | ---------------------------------------- | ------ | ------------- | ---------------------------------------------------------------------------------------------------------------------- | | `num_workers` | 整数 | `8` | 写入线程数量 | +| `worker_channel_size` | 整数 | `128` | 每个 worker 的请求通道大小 | +| `worker_request_batch_size` | 整数 | `64` | worker 处理请求的最大批量大小 | | `manifest_checkpoint_distance` | 整数 | `10` | 每写入 `manifest_checkpoint_distance` 个 manifest 文件创建一次 checkpoint | +| `compress_manifest` | 布尔值 | `false` | 是否用 gzip 压缩 manifest 和 checkpoint 文件 | | `max_background_jobs` | 整数 | `4` | 后台线程数量 | +| `max_background_flushes` | 整数 | 自动 | 后台 flush 任务的最大数量。默认为 CPU 核数的 1/2 | +| `max_background_compactions` | 整数 | 自动 | 后台 compaction 任务的最大数量。默认为 CPU 核数的 1/4 | +| `max_background_purges` | 整数 | 自动 | 后台 purge 任务的最大数量。默认为 CPU 核数 | | `auto_flush_interval` | 字符串 | `1h` | 自动 flush 超过 `auto_flush_interval` 没 flush 的 region | | `global_write_buffer_size` | 字符串 | `1GB` | 写入缓冲区大小，默认值为内存总量的 1/8，但不会超过 1GB | | `global_write_buffer_reject_size` | 字符串 | `2GB` | 写入缓冲区内数据的大小超过 `global_write_buffer_reject_size` 后拒绝写入请求，默认为 `global_write_buffer_size` 的 2 倍 | @@ -447,18 +500,31 @@ fork_dictionary_bytes = "1GiB" | `page_cache_size` | 字符串 | `512MB` | SST 数据页的缓存。设为 0 可关闭该缓存
默认为内存的 1/8 | | `selector_result_cache_size` | 字符串 | `512MB` | `last_value()` 等时间线检索结果的缓存。设为 0 可关闭该缓存
默认为内存的 1/16，不超过 512MB | | `sst_write_buffer_size` | 字符串 | `8MB` | SST 的写缓存大小 | +| `parallel_scan_channel_size` | 整数 | `32` | 从并行扫描任务向主任务发送数据的通道容量 | +| `max_concurrent_scan_files` | 整数 | `384` | 并发扫描的 SST 文件的最大数量 | | `scan_parallelism` | 整数 | `0` | 扫描并发度 (默认 1/4 CPU 核数)
- `0`: 使用默认值 (1/4 CPU 核数)
- `1`: 单线程扫描
- `n`: 按并行度 n 扫描 | +| `min_compaction_interval` | 字符串 | `0m` | 两次 compaction 之间的最小时间间隔。默认为 0（无限制） | +| `allow_stale_entries` | 布尔值 | `false` | 是否允许在重放时读取过期的 WAL 条目 | | `index` | -- | -- | Mito 引擎中索引的选项。 | | `index.aux_path` | 字符串 | `""` | 文件系统中索引的辅助目录路径，用于存储创建索引的中间文件和搜索索引的暂存文件，默认为 `{data_home}/index_intermediate`。为了向后兼容，该目录的默认名称为 `index_intermediate`。此路径包含两个子目录：- `__intm`: 用于存储创建索引时使用的中间文件。- `staging`: 用于存储搜索索引时使用的暂存文件。 | | `index.staging_size` | 字符串 | `2GB` | 暂存目录的最大容量。 | | `index.metadata_cache_size` | 字符串 | `64MiB` | 索引元数据的缓存大小。 | | `index.content_cache_size` | 字符串 | `128MiB` | 索引内容的缓存大小。 | | `index.content_cache_page_size` | 字符串 | `64KiB` | 倒排索引内容缓存的页大小。 | +| `index.staging_ttl` | 字符串 | `7d` | 暂存目录的 TTL。默认为 7 天。设置为 "0s" 可禁用 TTL | +| `index.result_cache_size` | 字符串 | `128MiB` | 索引结果的缓存大小 | | `inverted_index.create_on_flush` | 字符串 | `auto` | 是否在 flush 时构建索引
- `auto`: 自动
- `disable`: 从不 | | `inverted_index.create_on_compaction` | 字符串 | `auto` | 是否在 compaction 时构建索引
- `auto`: 自动
- `disable`: 从不 | | `inverted_index.apply_on_query` | 字符串 | `auto` | 是否在查询时使用索引
- `auto`: 自动
- `disable`: 从不 | -| `inverted_index.mem_threshold_on_create` | 字符串 | `64M` | 创建索引时如果超过该内存阈值则改为使用外部排序
设置为空会关闭外排，在内存中完成所有排序 | -| `inverted_index.intermediate_path` | 字符串 | `""` | 存放外排临时文件的路径 (默认 `{data_home}/index_intermediate`). | +| `inverted_index.mem_threshold_on_create` | 字符串 | `auto` | 创建索引时如果超过该内存阈值则改为使用外部排序。- `auto`: 自动确定阈值
- 自定义值: 例如 `64M`
- 空字符串: 关闭外排，在内存中完成所有排序 | +| `fulltext_index.create_on_flush` | 字符串 | `auto` | 是否在 flush 时构建全文索引
- `auto`: 自动
- `disable`: 从不 | +| `fulltext_index.create_on_compaction` | 字符串 | `auto` | 是否在 compaction 时构建全文索引
- `auto`: 自动
- `disable`: 从不 | +| `fulltext_index.apply_on_query` | 字符串 | `auto` | 是否在查询时使用全文索引
- `auto`: 自动
- `disable`: 从不 | +| `fulltext_index.mem_threshold_on_create` | 字符串 | `auto` | 创建全文索引时如果超过该内存阈值则改为使用外部排序。- `auto`: 自动确定阈值
- 自定义值: 例如 `64M`
- 空字符串: 关闭外排，在内存中完成所有排序 | +| `bloom_filter_index.create_on_flush` | 字符串 | `auto` | 是否在 flush 时构建 Bloom filter 索引
- `auto`: 自动
- `disable`: 从不 | +| `bloom_filter_index.create_on_compaction` | 字符串 | `auto` | 是否在 compaction 时构建 Bloom filter 索引
- `auto`: 自动
- `disable`: 从不 | +| `bloom_filter_index.apply_on_query` | 字符串 | `auto` | 是否在查询时使用 Bloom filter 索引
- `auto`: 自动
- `disable`: 从不 | +| `bloom_filter_index.mem_threshold_on_create` | 字符串 | `auto` | 创建 Bloom filter 索引时如果超过该内存阈值则改为使用外部排序。- `auto`: 自动确定阈值
- 自定义值: 例如 `64M`
- 空字符串: 关闭外排，在内存中完成所有排序 | | `memtable.type` | 字符串 | `time_series` | Memtable type.
- `time_series`: time-series memtable
- `partition_tree`: partition tree memtable (实验性功能) | | `memtable.index_max_keys_per_shard` | 整数 | `8192` | 一个 shard 内的主键数
只对 `partition_tree` memtable 生效 | | `memtable.data_freeze_threshold` | 整数 | `32768` | 一个 shard 内写缓存可容纳的最大行数
只对 `partition_tree` memtable 生效 | @@ -472,18 +538,26 @@ fork_dictionary_bytes = "1GiB" [meta_client] metasrv_addrs = ["127.0.0.1:3002"] timeout = "3s" +heartbeat_timeout = "500ms" connect_timeout = "1s" ddl_timeout = "10s" tcp_nodelay = true +metadata_cache_max_capacity = 100000 +metadata_cache_ttl = "10m" +metadata_cache_tti = "5m" ``` 通过 `meta_client` 配置 metasrv 客户端，包括： - `metasrv_addrs`，Metasrv 地址列表，对应 Metasrv 启动配置的 server address。 - `timeout`，操作超时时长，默认为 3 秒。 +- `heartbeat_timeout`，心跳请求的超时时长，默认为 500 毫秒。 - `connect_timeout`，连接服务器超时时长，默认为 1 秒。 - `ddl_timeout`，DDL 执行的超时时间，默认 10 秒。 - `tcp_nodelay`，接受连接时的 `TCP_NODELAY` 选项，默认为 true。 +- `metadata_cache_max_capacity`，元数据缓存的最大容量，默认为 100000。 +- `metadata_cache_ttl`，元数据缓存的 TTL（生存时间），默认为 10 分钟。 +- `metadata_cache_tti`，元数据缓存的 TTI（空闲超时时间），默认为 5 分钟。 ### 指标监控选项