Release 2.18.0

This release improves `group by` performance of compressing data and filters, introduces the ability to add secondary indexes, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity. **Highlighted features in this release** * TAM * Significant performance improvements for aggregations using a `group by` with one column and/or using a filter clause. SIMD * Transition tables **Dropping support for Bitnami images** After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. **Features** * #7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types). * #7104: Hypercore table access method. * #6901: Add hypertable support for transition tables. * #7482: Optimize recompression of partially compressed chunks. * #7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable. * #7433: Add support for merging chunks. * #7271: Push down `order by` in real-time continuous aggregate queries. * #7455: Support `drop not null` on compressed hypertables. * #7295: Support `alter table set access method` on hypertable. * #7411: Change parameter name to enable hypercore table access method. * #7436: Add index creation on `order by` columns. * #7443: Add hypercore function and view aliases. * #7521: Add optional `force` argument to `refresh_continuous_aggregate`. * #7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases. * #7565: Add hint when hypertable creation fails. * #7390: Disable custom `hashagg` planner code. * #7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API. * #7486: Prevent building against PostgreSQL versions with broken ABI. * #7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default. * #7413: Add GUC for segmentwise recompression. **Bugfixes** * #7378: Remove obsolete job referencing `policy_job_error_retention`. * #7409: Update `bgw_job` table when altering procedure. * #7410: Fix the `aggregated compressed column not found` error on aggregation query. * #7426: Fix `datetime` parsing error in chunk constraint creation. * #7432: Verify that the heap tuple is valid before using. * #7434: Fix the segfault when internally setting the replica identity for a given chunk. * #7488: Emit error for transition table trigger on chunks. * #7514: Fix the error: `invalid child of chunk append`. * #7517: Fix the performance regression on the `cagg_migrate` procedure. * #7527: Restart scheduler on error. * #7557: Fix null handling for in-memory tuple filtering. * #7566: Improve transaction check in CAGG refresh. * #7584: Fix NaN-handling for vectorized aggregation. **Thanks** * @bharrisau for reporting the segfault when creating chunks. * @k-rus for suggesting that we add a hint when hypertable creation fails. * @pgloader for reporting the issue in an internal background job. * @staticlibs for sending the pull request that improves the transaction check in CAGG refresh. * @uasiddiqi for reporting the `aggregated compressed column not found` error.
timescale · Jan 23, 2025 · 0d90d1b · 0d90d1b
1 parent 193431b
commit 0d90d1b
Show file tree

Hide file tree

Showing 40 changed files with 316 additions and 290 deletions.
diff --git a/.unreleased/compressed-sort-transform b/.unreleased/compressed-sort-transform
diff --git a/.unreleased/pr_6901 b/.unreleased/pr_6901
diff --git a/.unreleased/pr_7104 b/.unreleased/pr_7104
diff --git a/.unreleased/pr_7271 b/.unreleased/pr_7271
diff --git a/.unreleased/pr_7295 b/.unreleased/pr_7295
diff --git a/.unreleased/pr_7378 b/.unreleased/pr_7378
diff --git a/.unreleased/pr_7390 b/.unreleased/pr_7390
diff --git a/.unreleased/pr_7409 b/.unreleased/pr_7409
diff --git a/.unreleased/pr_7411 b/.unreleased/pr_7411
diff --git a/.unreleased/pr_7412 b/.unreleased/pr_7412
diff --git a/.unreleased/pr_7413 b/.unreleased/pr_7413
diff --git a/.unreleased/pr_7426 b/.unreleased/pr_7426
diff --git a/.unreleased/pr_7432 b/.unreleased/pr_7432
diff --git a/.unreleased/pr_7433 b/.unreleased/pr_7433
diff --git a/.unreleased/pr_7434 b/.unreleased/pr_7434
diff --git a/.unreleased/pr_7436 b/.unreleased/pr_7436
diff --git a/.unreleased/pr_7443 b/.unreleased/pr_7443
diff --git a/.unreleased/pr_7455 b/.unreleased/pr_7455
diff --git a/.unreleased/pr_7482 b/.unreleased/pr_7482
diff --git a/.unreleased/pr_7486 b/.unreleased/pr_7486
diff --git a/.unreleased/pr_7488 b/.unreleased/pr_7488
diff --git a/.unreleased/pr_7514 b/.unreleased/pr_7514
diff --git a/.unreleased/pr_7517 b/.unreleased/pr_7517
diff --git a/.unreleased/pr_7521 b/.unreleased/pr_7521
diff --git a/.unreleased/pr_7527 b/.unreleased/pr_7527
diff --git a/.unreleased/pr_7557 b/.unreleased/pr_7557
diff --git a/.unreleased/pr_7565 b/.unreleased/pr_7565
diff --git a/.unreleased/pr_7566 b/.unreleased/pr_7566
diff --git a/.unreleased/pr_7584 b/.unreleased/pr_7584
diff --git a/.unreleased/pr_7587 b/.unreleased/pr_7587
diff --git a/.unreleased/resolve-vars b/.unreleased/resolve-vars
diff --git a/.unreleased/vectorized-agg-filter b/.unreleased/vectorized-agg-filter
diff --git a/.unreleased/vectorized-grouping-one-fixed b/.unreleased/vectorized-grouping-one-fixed
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,68 @@
 `psql` with the `-X` flag to prevent any `.psqlrc` commands from
 accidentally triggering the load of a previous DB version.**
 
+
+## 2.18.0 (2025-01-23)
+
+This release improves `group by` performance of compressing data and filters,
+introduces the ability to add secondary indexes, and contains the highly upvoted
+community request of transition table support. We recommend that you upgrade at the
+next available opportunity.
+
+**Highlighted features in this release**
+* TAM
+* Significant performance improvements for aggregations using a `group by` with one column
+and/or using a filter clause. SIMD
+* Transition tables
+
+**Dropping support for Bitnami images**
+After the recent change in Bitnami’s [LTS support policy](https://github.com/bitnami/containers/issues/75671),
+we are no longer building Bitnami images for TimescaleDB.
+
+**Features**
+* #7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types).
+* #7104: Hypercore table access method.
+* #6901: Add hypertable support for transition tables.
+* #7482: Optimize recompression of partially compressed chunks.
+* #7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable.
+* #7433: Add support for merging chunks.
+* #7271: Push down `order by` in real-time continuous aggregate queries.
+* #7455: Support `drop not null` on compressed hypertables.
+* #7295: Support `alter table set access method` on hypertable.
+* #7411: Change parameter name to enable hypercore table access method.
+* #7436: Add index creation on `order by` columns.
+* #7443: Add hypercore function and view aliases.
+* #7521: Add optional `force` argument to `refresh_continuous_aggregate`.
+* #7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
+* #7565: Add hint when hypertable creation fails.
+* #7390: Disable custom `hashagg` planner code.
+* #7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API.
+* #7486: Prevent building against PostgreSQL versions with broken ABI.
+* #7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default.
+* #7413: Add GUC for segmentwise recompression.
+
+**Bugfixes**
+* #7378: Remove obsolete job referencing `policy_job_error_retention`.
+* #7409: Update `bgw_job` table when altering procedure.
+* #7410: Fix the `aggregated compressed column not found` error on aggregation query.
+* #7426: Fix `datetime` parsing error in chunk constraint creation.
+* #7432: Verify that the heap tuple is valid before using.
+* #7434: Fix the segfault when internally setting the replica identity for a given chunk.
+* #7488: Emit error for transition table trigger on chunks.
+* #7514: Fix the error: `invalid child of chunk append`.
+* #7517: Fix the performance regression on the `cagg_migrate` procedure.
+* #7527: Restart scheduler on error.
+* #7557: Fix null handling for in-memory tuple filtering.
+* #7566: Improve transaction check in CAGG refresh.
+* #7584: Fix NaN-handling for vectorized aggregation.
+
+**Thanks**
+* @bharrisau for reporting the segfault when creating chunks.
+* @k-rus for suggesting that we add a hint when hypertable creation fails.
+* @pgloader for reporting the issue in an internal background job.
+* @staticlibs for sending the pull request that improves the transaction check in CAGG refresh.
+* @uasiddiqi for reporting the `aggregated compressed column not found` error.
+
 ## 2.17.2 (2024-11-06)
 
 This release contains bug fixes since the 2.17.1 release. We recommend that you

diff --git a/sql/CMakeLists.txt b/sql/CMakeLists.txt
@@ -45,11 +45,12 @@ set(MOD_FILES
     updates/2.16.0--2.16.1.sql
     updates/2.16.1--2.17.0.sql
     updates/2.17.0--2.17.1.sql
-    updates/2.17.1--2.17.2.sql)
+    updates/2.17.1--2.17.2.sql
+    updates/2.17.2--2.18.0.sql)
 
 # The downgrade file to generate a downgrade script for the current version, as
 # specified in version.config
-set(CURRENT_REV_FILE reverse-dev.sql)
+set(CURRENT_REV_FILE 2.18.0--2.17.2.sql)
 # Files for generating old downgrade scripts. This should only include files for
 # downgrade from one version to its previous version since we do not support
 # skipping versions when downgrading.
@@ -90,7 +91,8 @@ set(OLD_REV_FILES
     2.16.1--2.16.0.sql
     2.17.0--2.16.1.sql
     2.17.1--2.17.0.sql
-    2.17.2--2.17.1.sql)
+    2.17.2--2.17.1.sql
+    2.18.0--2.17.2.sql)
 
 set(MODULE_PATHNAME "$libdir/timescaledb-${PROJECT_VERSION_MOD}")
 set(LOADER_PATHNAME "$libdir/timescaledb")

diff --git a/sql/updates/2.17.2--2.18.0.sql b/sql/updates/2.17.2--2.18.0.sql
@@ -0,0 +1,155 @@
+-- remove obsolete job
+DELETE FROM _timescaledb_config.bgw_job WHERE id = 2;
+
+-- Hypercore updates
+CREATE FUNCTION _timescaledb_debug.is_compressed_tid(tid) RETURNS BOOL
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder' LANGUAGE C STRICT;
+
+DROP FUNCTION IF EXISTS @[email protected]_chunk(uncompressed_chunk REGCLASS,	if_not_compressed BOOLEAN, recompress BOOLEAN);
+
+CREATE FUNCTION @[email protected]_chunk(
+    uncompressed_chunk REGCLASS,
+    if_not_compressed BOOLEAN = true,
+    recompress BOOLEAN = false,
+    hypercore_use_access_method BOOL = NULL
+) RETURNS REGCLASS AS '@MODULE_PATHNAME@', 'ts_update_placeholder' LANGUAGE C VOLATILE;
+
+DROP FUNCTION IF EXISTS @[email protected]_compression_policy(hypertable REGCLASS, compress_after "any", if_not_exists BOOL, schedule_interval INTERVAL, initial_start TIMESTAMPTZ, timezone TEXT, compress_created_before INTERVAL);
+
+CREATE FUNCTION @[email protected]_compression_policy(
+    hypertable REGCLASS,
+    compress_after "any" = NULL,
+    if_not_exists BOOL = false,
+    schedule_interval INTERVAL = NULL,
+    initial_start TIMESTAMPTZ = NULL,
+    timezone TEXT = NULL,
+    compress_created_before INTERVAL = NULL,
+    hypercore_use_access_method BOOL = NULL
+)
+RETURNS INTEGER
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder'
+LANGUAGE C VOLATILE;
+
+DROP FUNCTION IF EXISTS timescaledb_experimental.add_policies(relation REGCLASS, if_not_exists BOOL, refresh_start_offset "any", refresh_end_offset "any", compress_after "any", drop_after "any");
+
+CREATE FUNCTION timescaledb_experimental.add_policies(
+    relation REGCLASS,
+    if_not_exists BOOL = false,
+    refresh_start_offset "any" = NULL,
+    refresh_end_offset "any" = NULL,
+    compress_after "any" = NULL,
+    drop_after "any" = NULL,
+    hypercore_use_access_method BOOL = NULL)
+RETURNS BOOL
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder'
+LANGUAGE C VOLATILE;
+
+DROP PROCEDURE IF EXISTS _timescaledb_functions.policy_compression_execute(job_id INTEGER, htid INTEGER, lag ANYELEMENT, maxchunks INTEGER, verbose_log BOOLEAN, recompress_enabled  BOOLEAN, use_creation_time BOOLEAN);
+
+DROP PROCEDURE IF EXISTS _timescaledb_functions.policy_compression(job_id INTEGER, config JSONB);
+
+CREATE PROCEDURE @[email protected]_to_columnstore(
+    chunk REGCLASS,
+    if_not_columnstore BOOLEAN = true,
+    recompress BOOLEAN = false,
+    hypercore_use_access_method BOOL = NULL)
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder'
+LANGUAGE C;
+
+CREATE PROCEDURE @[email protected]_to_rowstore(
+    chunk REGCLASS,
+    if_columnstore BOOLEAN = true)
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder'
+LANGUAGE C;
+
+CREATE PROCEDURE @[email protected]_columnstore_policy(
+    hypertable REGCLASS,
+    after "any" = NULL,
+    if_not_exists BOOL = false,
+    schedule_interval INTERVAL = NULL,
+    initial_start TIMESTAMPTZ = NULL,
+    timezone TEXT = NULL,
+    created_before INTERVAL = NULL,
+    hypercore_use_access_method BOOL = NULL
+) LANGUAGE C AS '@MODULE_PATHNAME@', 'ts_update_placeholder';
+
+CREATE PROCEDURE @[email protected]_columnstore_policy(
+       hypertable REGCLASS,
+       if_exists BOOL = false
+) LANGUAGE C AS '@MODULE_PATHNAME@', 'ts_update_placeholder';
+
+CREATE FUNCTION @[email protected]_columnstore_stats (hypertable REGCLASS)
+    RETURNS TABLE (
+        chunk_schema name,
+        chunk_name name,
+        compression_status text,
+        before_compression_table_bytes bigint,
+        before_compression_index_bytes bigint,
+        before_compression_toast_bytes bigint,
+        before_compression_total_bytes bigint,
+        after_compression_table_bytes bigint,
+        after_compression_index_bytes bigint,
+        after_compression_toast_bytes bigint,
+        after_compression_total_bytes bigint,
+        node_name name)
+    LANGUAGE SQL
+    STABLE STRICT
+    AS 'SELECT * FROM @[email protected]_compression_stats($1)'
+    SET search_path TO pg_catalog, pg_temp;
+
+CREATE FUNCTION @[email protected]_columnstore_stats (hypertable REGCLASS)
+    RETURNS TABLE (
+        total_chunks bigint,
+        number_compressed_chunks bigint,
+        before_compression_table_bytes bigint,
+        before_compression_index_bytes bigint,
+        before_compression_toast_bytes bigint,
+        before_compression_total_bytes bigint,
+        after_compression_table_bytes bigint,
+        after_compression_index_bytes bigint,
+        after_compression_toast_bytes bigint,
+        after_compression_total_bytes bigint,
+        node_name name)
+    LANGUAGE SQL
+    STABLE STRICT
+    AS 'SELECT * FROM @[email protected]_compression_stats($1)'
+    SET search_path TO pg_catalog, pg_temp;
+
+-- Recreate `refresh_continuous_aggregate` procedure to add `force` argument
+DROP PROCEDURE IF EXISTS @[email protected]_continuous_aggregate (continuous_aggregate REGCLASS, window_start "any", window_end "any");
+
+CREATE PROCEDURE @[email protected]_continuous_aggregate(
+    continuous_aggregate     REGCLASS,
+    window_start             "any",
+    window_end               "any",
+    force                    BOOLEAN = FALSE
+) LANGUAGE C AS '@MODULE_PATHNAME@', 'ts_update_placeholder';
+
+-- Add `include_tiered_data` argument to `add_continuous_aggregate_policy`
+DROP FUNCTION @[email protected]_continuous_aggregate_policy(
+    continuous_aggregate REGCLASS, start_offset "any",
+    end_offset "any", schedule_interval INTERVAL,
+    if_not_exists BOOL,
+    initial_start TIMESTAMPTZ,
+    timezone TEXT
+);
+CREATE FUNCTION @[email protected]_continuous_aggregate_policy(
+    continuous_aggregate REGCLASS, start_offset "any",
+    end_offset "any", schedule_interval INTERVAL,
+    if_not_exists BOOL = false,
+    initial_start TIMESTAMPTZ = NULL,
+    timezone TEXT = NULL,
+	include_tiered_data BOOL = NULL
+)
+RETURNS INTEGER
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder'
+LANGUAGE C VOLATILE;
+
+-- Merge chunks
+CREATE PROCEDURE @[email protected]_chunks(
+    chunk1 REGCLASS, chunk2 REGCLASS
+) LANGUAGE C AS '@MODULE_PATHNAME@', 'ts_update_placeholder';
+
+CREATE PROCEDURE @[email protected]_chunks(
+    chunks REGCLASS[]
+) LANGUAGE C AS '@MODULE_PATHNAME@', 'ts_update_placeholder';