Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add statique materialized view #305

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

jmaupetit
Copy link
Member

@jmaupetit jmaupetit commented Dec 18, 2024

Purpose

Having a SQL materialized view for "statique" data would ease recurrent database queries joining the same tables frequently. Moreover this should speed up database queries and response time for API endpoints using them.

Proposal

  • add sqlalchemy_utils dependency to ease materialized view management
  • add statique materialized view
  • test the materialized view
  • integrate a refresh strategy

@jmaupetit jmaupetit self-assigned this Dec 18, 2024
@jmaupetit jmaupetit marked this pull request as draft December 18, 2024 18:31
@jmaupetit jmaupetit added critical improvement WIP API Item related to the API service labels Dec 18, 2024
@jmaupetit jmaupetit force-pushed the api-add-statique-materialized-view branch from 36bcd7a to a9602f0 Compare January 16, 2025 16:15
@MTES-MCT MTES-MCT deleted a comment from github-actions bot Jan 16, 2025
@MTES-MCT MTES-MCT deleted a comment from github-actions bot Jan 16, 2025
Copy link

Current benchmark

Type Name Request Count Failure Count Median Response Time Average Response Time Min Response Time Max Response Time Average Content Size Requests/s Failures/s 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% git timestamp version
GET /auth/whoami 26 0 440.0 654.9960846153875 2.5614129999951274 2003.010771000021 160.0 0.8988042172954325 0.0 460 850 940 1300 1600 1700 2000 2000 2000 2000 2000 8e36440 2025-01-17 10:55:56.078856+00:00 0.16.0
POST /dynamique/session/ 21 0 1100.0 966.445110095239 51.3447259999964 1909.327248000011 45.0 0.7259572524309262 0.0 1100 1100 1300 1400 1800 1800 1900 1900 1900 1900 1900 8e36440 2025-01-17 10:55:56.078856+00:00 0.16.0
POST /dynamique/session/bulk 26 0 940.0 1092.8425574615376 18.8574860000017 3377.787278999989 411.0 0.8988042172954325 0.0 940 1300 1800 1900 2400 2400 3400 3400 3400 3400 3400 8e36440 2025-01-17 10:55:56.078856+00:00 0.16.0
GET /dynamique/status/ 25 0 850.0 922.2937881199972 18.623044000008804 2393.150949999977 285.0 0.8642348243225313 0.0 850 1100 1100 1300 2400 2400 2400 2400 2400 2400 2400 8e36440 2025-01-17 10:55:56.078856+00:00 0.16.0
POST /dynamique/status/ 27 0 280.0 692.6267656296303 18.153559000012365 2653.307245000008 45.0 0.9333736102683338 0.0 280 940 1400 1400 1800 1800 2700 2700 2700 2700 2700 8e36440 2025-01-17 10:55:56.078856+00:00 0.16.0
GET /dynamique/status/FRALLEGO002006P3 21 0 950.0 939.4849056666654 35.07820000001516 2443.810198999984 283.0 0.7259572524309262 0.0 950 1100 1300 1400 1800 1800 2400 2400 2400 2400 2400 8e36440 2025-01-17 10:55:56.078856+00:00 0.16.0
GET /dynamique/status/FRALLEGO002006P3/history 24 0 890.0 1015.1485303750007 12.572855000001937 3364.9994339999976 64542.833333333336 0.82966543134963 0.0 910 1100 1400 1500 2000 2100 3400 3400 3400 3400 3400 8e36440 2025-01-17 10:55:56.078856+00:00 0.16.0
POST /dynamique/status/bulk 35 0 880.0 866.4592271428604 12.527720000008458 2653.949668999985 411.0 1.2099287540515438 0.0 880 1000 1500 1700 1800 2400 2700 2700 2700 2700 2700 8e36440 2025-01-17 10:55:56.078856+00:00 0.16.0
POST /statique/ 18 7 940.0 1173.8402273333356 33.48681900001793 3368.151293000011 57.05555555555556 0.6222490735122225 0.2419857508103087 1100 1200 1600 1700 3400 3400 3400 3400 3400 3400 3400 8e36440 2025-01-17 10:55:56.078856+00:00 0.16.0
GET /statique/?limit=10 25 0 950.0 925.9586292000004 21.47623600001225 2384.5099400000154 13826.48 0.8642348243225313 0.0 950 1300 1300 1400 1700 1900 2400 2400 2400 2400 2400 8e36440 2025-01-17 10:55:56.078856+00:00 0.16.0
GET /statique/?limit=100 28 0 970.0 986.2816288214252 142.84350200000517 3363.868915000012 129006.53571428572 0.967943003241235 0.0 980 1100 1300 1400 1800 1800 3400 3400 3400 3400 3400 8e36440 2025-01-17 10:55:56.078856+00:00 0.16.0
PUT /statique/FR073EP7QU5233 23 0 940.0 963.7401336956508 46.05530800000679 2656.052921999986 1371.0 0.7950960383767287 0.0 940 1100 1500 1600 1800 1800 2700 2700 2700 2700 2700 8e36440 2025-01-17 10:55:56.078856+00:00 0.16.0
GET /statique/FRALLEGO002006P3 35 0 980.0 910.221689714286 18.237545000005184 2656.3762989999877 1202.0 1.2099287540515438 0.0 980 1100 1400 1400 1700 1800 2700 2700 2700 2700 2700 8e36440 2025-01-17 10:55:56.078856+00:00 0.16.0
POST /statique/bulk 33 0 1600.0 1473.622342212121 369.7135550000041 3378.620521000016 1126.878787878788 1.1407899681057412 0.0 1600 1800 1800 1900 2200 3400 3400 3400 3400 3400 3400 8e36440 2025-01-17 10:55:56.078856+00:00 0.16.0
Aggregated 367 7 940.0 971.2902861471392 2.5614129999951274 3378.620521000016 15430.92370572207 12.68696722105476 0.2419857508103087 940 1100 1400 1600 1800 2400 2700 3400 3400 3400 3400 8e36440 2025-01-17 10:55:56.078856+00:00 0.16.0

Comparison with the latest previous benchmark

A lower (negative) value means the current version performs better than the previous one.

Type Name Request Count Failure Count Median Response Time Average Response Time Min Response Time Max Response Time Average Content Size Requests/s
GET /auth/whoami
POST /dynamique/session/
POST /dynamique/session/bulk
GET /dynamique/status/
POST /dynamique/status/
GET /dynamique/status/FRALLEGO002006P3
GET /dynamique/status/FRALLEGO002006P3/history
POST /dynamique/status/bulk
POST /statique/
GET /statique/?limit=10
GET /statique/?limit=100
PUT /statique/FR073EP7QU5233
GET /statique/FRALLEGO002006P3
POST /statique/bulk
Aggregated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API Item related to the API service critical improvement WIP
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant