Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up counting past 30 day traffic via ingest_counters #4841

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

macobo
Copy link
Contributor

@macobo macobo commented Nov 20, 2024

Companion to #4839

A projection speeds up ClickHouse queries by storing data in a different order in each part. In this case, it pre-aggregates results by site, speeding up the query and requiring less resources at query-time.

Query stats before:

-- 10 rows in set. Elapsed: 1.134 sec. Processed 1.45 billion rows, 21.65 GB (1.28 billion rows/s., 19.09 GB/s.)
-- Peak memory usage: 53.41 MiB.

After:

-- 10 rows in set. Elapsed: 0.321 sec. Processed 39.70 million rows, 1.07 GB (123.51 million rows/s., 3.33 GB/s.)
-- Peak memory usage: 97.43 MiB.

Companion to #4839

A projection speeds up ClickHouse queries by storing data in a different
order in each part. In this case, it pre-aggregates results by site,
speeding up the query and requiring less resources at query-time.

Query stats before:
```
-- 10 rows in set. Elapsed: 1.134 sec. Processed 1.45 billion rows, 21.65 GB (1.28 billion rows/s., 19.09 GB/s.)
-- Peak memory usage: 53.41 MiB.
```

After:
```
-- 10 rows in set. Elapsed: 0.321 sec. Processed 39.70 million rows, 1.07 GB (123.51 million rows/s., 3.33 GB/s.)
-- Peak memory usage: 97.43 MiB.
```
@macobo macobo requested review from apata and RobertJoonas November 20, 2024 06:52
@macobo macobo marked this pull request as draft November 20, 2024 06:58
@macobo macobo marked this pull request as ready for review November 20, 2024 08:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants