Release 2.18.0

This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity. **Highlighted features in TimescaleDB v2.18.0** * The ability to add secondary indexes to the columnstore through the new hypercore table access method. * Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore. * Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests. * Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression. **Dropping support for Bitnami images** After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha) **Deprecation Notice** We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below: | Deprecated | Replacement | Type | | --- | --- | --- | | decompress_chunk | convert_to_rowstore | Procedure | | compress_chunk | convert_to_columnstore | Procedure | | add_compression_policy | add_columnstore_policy | Function | | remove_compression_policy | remove_columnstore_policy | Function | | hypertable_compression_stats | hypertable_columnstore_stats | Function | | chunk_compression_stats | chunk_columnstore_stats | Function | | hypertable_compression_settings | hypertable_columnstore_settings | View | | chunk_compression_settings | chunk_columnstore_settings | View | | compression_settings | columnstore_settings | View | | timescaledb.compress | timescaledb.enable_columnstore | Parameter | | timescaledb.compress_segmentby | timescaledb.segmentby | Parameter | | timescaledb.compress_orderby | timescaledb.orderby | Parameter | **Features** * timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types). * timescale#7104: Hypercore table access method. * timescale#6901: Add hypertable support for transition tables. * timescale#7482: Optimize recompression of partially compressed chunks. * timescale#7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable. * timescale#7433: Add support for merging chunks. * timescale#7271: Push down `order by` in real-time continuous aggregate queries. * timescale#7455: Support `drop not null` on compressed hypertables. * timescale#7295: Support `alter table set access method` on hypertable. * timescale#7411: Change parameter name to enable hypercore table access method. * timescale#7436: Add index creation on `order by` columns. * timescale#7443: Add hypercore function and view aliases. * timescale#7521: Add optional `force` argument to `refresh_continuous_aggregate`. * timescale#7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases. * timescale#7565: Add hint when hypertable creation fails. * timescale#7390: Disable custom `hashagg` planner code. * timescale#7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API. * timescale#7486: Prevent building against PostgreSQL versions with broken ABI. * timescale#7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default. * timescale#7413: Add GUC for segmentwise recompression. **Bugfixes** * timescale#7378: Remove obsolete job referencing `policy_job_error_retention`. * timescale#7409: Update `bgw_job` table when altering procedure. * timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query. * timescale#7426: Fix `datetime` parsing error in chunk constraint creation. * timescale#7432: Verify that the heap tuple is valid before using. * timescale#7434: Fix the segfault when internally setting the replica identity for a given chunk. * timescale#7488: Emit error for transition table trigger on chunks. * timescale#7514: Fix the error: `invalid child of chunk append`. * timescale#7517: Fix the performance regression on the `cagg_migrate` procedure. * timescale#7527: Restart scheduler on error. * timescale#7557: Fix null handling for in-memory tuple filtering. * timescale#7566: Improve transaction check in CAGG refresh. * timescale#7584: Fix NaN-handling for vectorized aggregation. * timescale#7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables. **Thanks** * @bharrisau for reporting the segfault when creating chunks. * @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables. * @k-rus for suggesting that we add a hint when hypertable creation fails. * @staticlibs for sending the pull request that improves the transaction check in CAGG refresh. * @uasiddiqi for reporting the `aggregated compressed column not found` error.
pallavisontakke · Jan 24, 2025 · 996eb05 · 996eb05
1 parent dc67130
commit 996eb05
Show file tree

Hide file tree

Showing 41 changed files with 352 additions and 308 deletions.
diff --git a/.unreleased/compressed-sort-transform b/.unreleased/compressed-sort-transform
diff --git a/.unreleased/nan-vectorized-filters b/.unreleased/nan-vectorized-filters
diff --git a/.unreleased/pr_6901 b/.unreleased/pr_6901
diff --git a/.unreleased/pr_7104 b/.unreleased/pr_7104
diff --git a/.unreleased/pr_7271 b/.unreleased/pr_7271
diff --git a/.unreleased/pr_7295 b/.unreleased/pr_7295
diff --git a/.unreleased/pr_7378 b/.unreleased/pr_7378
diff --git a/.unreleased/pr_7390 b/.unreleased/pr_7390
diff --git a/.unreleased/pr_7409 b/.unreleased/pr_7409
diff --git a/.unreleased/pr_7411 b/.unreleased/pr_7411
diff --git a/.unreleased/pr_7412 b/.unreleased/pr_7412
diff --git a/.unreleased/pr_7413 b/.unreleased/pr_7413
diff --git a/.unreleased/pr_7426 b/.unreleased/pr_7426
diff --git a/.unreleased/pr_7432 b/.unreleased/pr_7432
diff --git a/.unreleased/pr_7433 b/.unreleased/pr_7433
diff --git a/.unreleased/pr_7434 b/.unreleased/pr_7434
diff --git a/.unreleased/pr_7436 b/.unreleased/pr_7436
diff --git a/.unreleased/pr_7443 b/.unreleased/pr_7443
diff --git a/.unreleased/pr_7455 b/.unreleased/pr_7455
diff --git a/.unreleased/pr_7482 b/.unreleased/pr_7482
diff --git a/.unreleased/pr_7486 b/.unreleased/pr_7486
diff --git a/.unreleased/pr_7488 b/.unreleased/pr_7488
diff --git a/.unreleased/pr_7514 b/.unreleased/pr_7514
diff --git a/.unreleased/pr_7517 b/.unreleased/pr_7517
diff --git a/.unreleased/pr_7521 b/.unreleased/pr_7521
diff --git a/.unreleased/pr_7527 b/.unreleased/pr_7527
diff --git a/.unreleased/pr_7557 b/.unreleased/pr_7557
diff --git a/.unreleased/pr_7565 b/.unreleased/pr_7565
diff --git a/.unreleased/pr_7566 b/.unreleased/pr_7566
diff --git a/.unreleased/pr_7584 b/.unreleased/pr_7584
diff --git a/.unreleased/pr_7587 b/.unreleased/pr_7587
diff --git a/.unreleased/resolve-vars b/.unreleased/resolve-vars
diff --git a/.unreleased/vectorized-agg-filter b/.unreleased/vectorized-agg-filter
diff --git a/.unreleased/vectorized-grouping-one-fixed b/.unreleased/vectorized-grouping-one-fixed
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,87 @@
 `psql` with the `-X` flag to prevent any `.psqlrc` commands from
 accidentally triggering the load of a previous DB version.**
 
+
+## 2.18.0 (2025-01-23)
+
+This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity.
+
+**Highlighted features in TimescaleDB v2.18.0**
+
+* The ability to add secondary indexes to the columnstore through the new hypercore table access method.
+* Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore.
+* Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests.
+* Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression.
+
+**Dropping support for Bitnami images**
+
+After the recent change in Bitnami’s [LTS support policy](https://github.com/bitnami/containers/issues/75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha)
+
+**Deprecation Notice**
+
+We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below:
+
+| Deprecated | Replacement | Type |
+| --- | --- | --- |
+| decompress_chunk | convert_to_rowstore | Procedure |
+| compress_chunk | convert_to_columnstore | Procedure | 
+| add_compression_policy | add_columnstore_policy | Function | 
+| remove_compression_policy | remove_columnstore_policy | Function | 
+| hypertable_compression_stats | hypertable_columnstore_stats | Function | 
+| chunk_compression_stats | chunk_columnstore_stats | Function | 
+| hypertable_compression_settings | hypertable_columnstore_settings | View | 
+| chunk_compression_settings | chunk_columnstore_settings | View | 
+| compression_settings | columnstore_settings | View | 
+| timescaledb.compress | timescaledb.enable_columnstore | Parameter | 
+| timescaledb.compress_segmentby | timescaledb.segmentby | Parameter | 
+| timescaledb.compress_orderby  | timescaledb.orderby | Parameter | 
+
+**Features**
+* #7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types).
+* #7104: Hypercore table access method.
+* #6901: Add hypertable support for transition tables.
+* #7482: Optimize recompression of partially compressed chunks.
+* #7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable.
+* #7433: Add support for merging chunks.
+* #7271: Push down `order by` in real-time continuous aggregate queries.
+* #7455: Support `drop not null` on compressed hypertables.
+* #7295: Support `alter table set access method` on hypertable.
+* #7411: Change parameter name to enable hypercore table access method.
+* #7436: Add index creation on `order by` columns.
+* #7443: Add hypercore function and view aliases.
+* #7521: Add optional `force` argument to `refresh_continuous_aggregate`.
+* #7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
+* #7565: Add hint when hypertable creation fails.
+* #7390: Disable custom `hashagg` planner code.
+* #7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API.
+* #7486: Prevent building against PostgreSQL versions with broken ABI.
+* #7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default.
+* #7413: Add GUC for segmentwise recompression.
+
+**Bugfixes**
+* #7378: Remove obsolete job referencing `policy_job_error_retention`.
+* #7409: Update `bgw_job` table when altering procedure.
+* #7410: Fix the `aggregated compressed column not found` error on aggregation query.
+* #7426: Fix `datetime` parsing error in chunk constraint creation.
+* #7432: Verify that the heap tuple is valid before using.
+* #7434: Fix the segfault when internally setting the replica identity for a given chunk.
+* #7488: Emit error for transition table trigger on chunks.
+* #7514: Fix the error: `invalid child of chunk append`.
+* #7517: Fix the performance regression on the `cagg_migrate` procedure.
+* #7527: Restart scheduler on error.
+* #7557: Fix null handling for in-memory tuple filtering.
+* #7566: Improve transaction check in CAGG refresh.
+* #7584: Fix NaN-handling for vectorized aggregation.
+* #7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables.
+
+**Thanks**
+* @bharrisau for reporting the segfault when creating chunks.
+* @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables.
+* @k-rus for suggesting that we add a hint when hypertable creation fails.
+* @staticlibs for sending the pull request that improves the transaction check in CAGG refresh.
+* @uasiddiqi for reporting the `aggregated compressed column not found` error.
+
+
 ## 2.17.2 (2024-11-06)
 
 This release contains bug fixes since the 2.17.1 release. We recommend that you

diff --git a/sql/CMakeLists.txt b/sql/CMakeLists.txt
@@ -45,7 +45,8 @@ set(MOD_FILES
     updates/2.16.0--2.16.1.sql
     updates/2.16.1--2.17.0.sql
     updates/2.17.0--2.17.1.sql
-    updates/2.17.1--2.17.2.sql)
+    updates/2.17.1--2.17.2.sql
+    updates/2.17.2--2.18.0.sql)
 
 # The downgrade file to generate a downgrade script for the current version, as
 # specified in version.config
@@ -90,7 +91,8 @@ set(OLD_REV_FILES
     2.16.1--2.16.0.sql
     2.17.0--2.16.1.sql
     2.17.1--2.17.0.sql
-    2.17.2--2.17.1.sql)
+    2.17.2--2.17.1.sql
+    2.18.0--2.17.2.sql)
 
 set(MODULE_PATHNAME "$libdir/timescaledb-${PROJECT_VERSION_MOD}")
 set(LOADER_PATHNAME "$libdir/timescaledb")

diff --git a/sql/updates/2.17.2--2.18.0.sql b/sql/updates/2.17.2--2.18.0.sql
@@ -0,0 +1,172 @@
+-- remove obsolete job
+DELETE FROM _timescaledb_config.bgw_job WHERE id = 2;
+
+-- Hypercore updates
+CREATE FUNCTION _timescaledb_debug.is_compressed_tid(tid) RETURNS BOOL
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder' LANGUAGE C STRICT;
+
+DROP FUNCTION IF EXISTS @[email protected]_chunk(uncompressed_chunk REGCLASS,	if_not_compressed BOOLEAN, recompress BOOLEAN);
+
+CREATE FUNCTION @[email protected]_chunk(
+    uncompressed_chunk REGCLASS,
+    if_not_compressed BOOLEAN = true,
+    recompress BOOLEAN = false,
+    hypercore_use_access_method BOOL = NULL
+) RETURNS REGCLASS AS '@MODULE_PATHNAME@', 'ts_update_placeholder' LANGUAGE C VOLATILE;
+
+DROP FUNCTION IF EXISTS @[email protected]_compression_policy(hypertable REGCLASS, compress_after "any", if_not_exists BOOL, schedule_interval INTERVAL, initial_start TIMESTAMPTZ, timezone TEXT, compress_created_before INTERVAL);
+
+CREATE FUNCTION @[email protected]_compression_policy(
+    hypertable REGCLASS,
+    compress_after "any" = NULL,
+    if_not_exists BOOL = false,
+    schedule_interval INTERVAL = NULL,
+    initial_start TIMESTAMPTZ = NULL,
+    timezone TEXT = NULL,
+    compress_created_before INTERVAL = NULL,
+    hypercore_use_access_method BOOL = NULL
+)
+RETURNS INTEGER
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder'
+LANGUAGE C VOLATILE;
+
+DROP FUNCTION IF EXISTS timescaledb_experimental.add_policies(relation REGCLASS, if_not_exists BOOL, refresh_start_offset "any", refresh_end_offset "any", compress_after "any", drop_after "any");
+
+CREATE FUNCTION timescaledb_experimental.add_policies(
+    relation REGCLASS,
+    if_not_exists BOOL = false,
+    refresh_start_offset "any" = NULL,
+    refresh_end_offset "any" = NULL,
+    compress_after "any" = NULL,
+    drop_after "any" = NULL,
+    hypercore_use_access_method BOOL = NULL)
+RETURNS BOOL
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder'
+LANGUAGE C VOLATILE;
+
+DROP PROCEDURE IF EXISTS _timescaledb_functions.policy_compression_execute(job_id INTEGER, htid INTEGER, lag ANYELEMENT, maxchunks INTEGER, verbose_log BOOLEAN, recompress_enabled  BOOLEAN, use_creation_time BOOLEAN);
+
+DROP PROCEDURE IF EXISTS _timescaledb_functions.policy_compression(job_id INTEGER, config JSONB);
+
+CREATE PROCEDURE @[email protected]_to_columnstore(
+    chunk REGCLASS,
+    if_not_columnstore BOOLEAN = true,
+    recompress BOOLEAN = false,
+    hypercore_use_access_method BOOL = NULL)
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder'
+LANGUAGE C;
+
+CREATE PROCEDURE @[email protected]_to_rowstore(
+    chunk REGCLASS,
+    if_columnstore BOOLEAN = true)
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder'
+LANGUAGE C;
+
+CREATE PROCEDURE @[email protected]_columnstore_policy(
+    hypertable REGCLASS,
+    after "any" = NULL,
+    if_not_exists BOOL = false,
+    schedule_interval INTERVAL = NULL,
+    initial_start TIMESTAMPTZ = NULL,
+    timezone TEXT = NULL,
+    created_before INTERVAL = NULL,
+    hypercore_use_access_method BOOL = NULL
+) LANGUAGE C AS '@MODULE_PATHNAME@', 'ts_update_placeholder';
+
+CREATE PROCEDURE @[email protected]_columnstore_policy(
+       hypertable REGCLASS,
+       if_exists BOOL = false
+) LANGUAGE C AS '@MODULE_PATHNAME@', 'ts_update_placeholder';
+
+CREATE FUNCTION @[email protected]_columnstore_stats (hypertable REGCLASS)
+    RETURNS TABLE (
+        chunk_schema name,
+        chunk_name name,
+        compression_status text,
+        before_compression_table_bytes bigint,
+        before_compression_index_bytes bigint,
+        before_compression_toast_bytes bigint,
+        before_compression_total_bytes bigint,
+        after_compression_table_bytes bigint,
+        after_compression_index_bytes bigint,
+        after_compression_toast_bytes bigint,
+        after_compression_total_bytes bigint,
+        node_name name)
+    LANGUAGE SQL
+    STABLE STRICT
+    AS 'SELECT * FROM @[email protected]_compression_stats($1)'
+    SET search_path TO pg_catalog, pg_temp;
+
+CREATE FUNCTION @[email protected]_columnstore_stats (hypertable REGCLASS)
+    RETURNS TABLE (
+        total_chunks bigint,
+        number_compressed_chunks bigint,
+        before_compression_table_bytes bigint,
+        before_compression_index_bytes bigint,
+        before_compression_toast_bytes bigint,
+        before_compression_total_bytes bigint,
+        after_compression_table_bytes bigint,
+        after_compression_index_bytes bigint,
+        after_compression_toast_bytes bigint,
+        after_compression_total_bytes bigint,
+        node_name name)
+    LANGUAGE SQL
+    STABLE STRICT
+    AS 'SELECT * FROM @[email protected]_compression_stats($1)'
+    SET search_path TO pg_catalog, pg_temp;
+
+-- Recreate `refresh_continuous_aggregate` procedure to add `force` argument
+DROP PROCEDURE IF EXISTS @[email protected]_continuous_aggregate (continuous_aggregate REGCLASS, window_start "any", window_end "any");
+
+CREATE PROCEDURE @[email protected]_continuous_aggregate(
+    continuous_aggregate     REGCLASS,
+    window_start             "any",
+    window_end               "any",
+    force                    BOOLEAN = FALSE
+) LANGUAGE C AS '@MODULE_PATHNAME@', 'ts_update_placeholder';
+
+-- Add `include_tiered_data` argument to `add_continuous_aggregate_policy`
+DROP FUNCTION @[email protected]_continuous_aggregate_policy(
+    continuous_aggregate REGCLASS, start_offset "any",
+    end_offset "any", schedule_interval INTERVAL,
+    if_not_exists BOOL,
+    initial_start TIMESTAMPTZ,
+    timezone TEXT
+);
+CREATE FUNCTION @[email protected]_continuous_aggregate_policy(
+    continuous_aggregate REGCLASS, start_offset "any",
+    end_offset "any", schedule_interval INTERVAL,
+    if_not_exists BOOL = false,
+    initial_start TIMESTAMPTZ = NULL,
+    timezone TEXT = NULL,
+	include_tiered_data BOOL = NULL
+)
+RETURNS INTEGER
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder'
+LANGUAGE C VOLATILE;
+
+-- Merge chunks
+CREATE PROCEDURE @[email protected]_chunks(
+    chunk1 REGCLASS, chunk2 REGCLASS
+) LANGUAGE C AS '@MODULE_PATHNAME@', 'ts_update_placeholder';
+
+CREATE PROCEDURE @[email protected]_chunks(
+    chunks REGCLASS[]
+) LANGUAGE C AS '@MODULE_PATHNAME@', 'ts_update_placeholder';
+
+CREATE FUNCTION @[email protected]_hypercore_handler(internal) RETURNS table_am_handler
+AS '@MODULE_PATHNAME@', 'ts_hypercore_handler' LANGUAGE C;
+
+CREATE FUNCTION @[email protected]_hypercore_proxy_handler(internal) RETURNS index_am_handler
+AS '@MODULE_PATHNAME@', 'ts_hypercore_proxy_handler' LANGUAGE C;
+
+CREATE ACCESS METHOD hypercore TYPE TABLE HANDLER @[email protected]_hypercore_handler;
+COMMENT ON ACCESS METHOD hypercore IS 'Storage engine using hybrid row/columnar compression';
+
+CREATE ACCESS METHOD hypercore_proxy TYPE INDEX HANDLER @[email protected]_hypercore_proxy_handler;
+COMMENT ON ACCESS METHOD hypercore_proxy IS 'Hypercore proxy index access method';
+
+CREATE OPERATOR CLASS int4_ops
+DEFAULT FOR TYPE int4 USING hypercore_proxy AS
+       OPERATOR 1 = (int4, int4),
+       FUNCTION 1 hashint4(int4);