Skip to main content

Time partition

About 2 min

Time partition

Features

Time partition divides data according to time, and a time partition is used to save all data within a certain time range. The time partition number is represented by a natural number. Number 0 means January 1, 1970, it will increase by one every partition_interval seconds. Time partition number's calculation formula is timestamp / partition_interval. The main configuration items are as follows:

Notice: Currently, it's not recommend to open this function. If open, please calculate appropriate concurrent_writing_time_partition and wal_buffer_size, you can calculate wal_buffer_size = MaxDirectMemorySizeInBytes * 0.3 / (storage_group_num * virtual_storage_group_num) / concurrent_writing_time_partition

  • enable_partition
Nameenable_partition
DescriptionWhether enable time partition for data, if disabled, all data belongs to partition 0 (It's not recommend to open this function. If open, please calculate appropriate concurrent_writing_time_partition and wal_buffer_size)
TypeBool
Defaultfalse
EffectiveOnly allowed to be modified in first start up
  • partition_interval
Namepartition_interval
DescriptionTime range for dividing storage group, time series data will be divided into groups by this time range
TypeInt64
Default604800
EffectiveOnly allowed to be modified in first start up

Configuration example

Enable time partition and set partition_interval to 86400 (one day), then the data distribution is shown as the following figure:

time partition example
  • Insert one datapoint with timestamp 0, calculate 0/86400 = 0, then this datapoint will be stored in TsFile under folder 0

  • Insert one datapoint with timestamp 1609459200010, calculate 1609459200010/86400 = 18628, then this datapoint will be stored in TsFile under folder 18628

Suggestions

When enabling time partition, it is better to enable timed flush memtable and timed close tsfile, configuration params are detailed in Config manual for timed flush and timed close.

  • enable_timed_flush_unseq_memtable: Whether to enable timed flush unsequence memtable, enabled by default.

  • enable_timed_flush_seq_memtable: Whether to enable timed flush sequence memtable, disabled by default. It should be enabled when time partition is enabled, so inactive time partition's memtable can be flushed regularly to prepare for the timed close of TsFileProcessor.

  • enable_timed_close_tsfile: Whether to enable timed close tsfile, disabled by default. It should be enabled when time partition is enabled, so inactive time partition's TsFileProcessor can be closed regularly to reduce memory usage.

Copyright © 2024 The Apache Software Foundation.
Apache and the Apache feather logo are trademarks of The Apache Software Foundation

Have a question? Connect with us on QQ, WeChat, or Slack. Join the community now.