[ECO] Use new Entity v2 endpoints in Inventory #203452

rmyz · 2024-12-09T14:58:44Z

Summary

Closes #202295
Closes #202296

This PR adapts Inventory to use the new Entity v2 endpoints.

Testing

Use any synthtrace scenario that loads service/hosts/containers data
Navigate and make sure everything works as expected (navigation to Discovery/Infra/Services pages, interacting with the table, searching for some specific entity, interacting with the type filter)
To check the alerts work, it's easier to connect to a remote cluster.

rmyz · 2024-12-09T16:31:17Z

/ci

…fix'

rmyz · 2024-12-10T16:21:33Z

/ci

# Conflicts: # x-pack/plugins/observability_solution/inventory/server/routes/entities/route.ts

…er issue

elasticmachine · 2024-12-14T12:43:17Z

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

rmyz · 2025-01-07T15:53:26Z

Hey @iblancof, I took a look at the points that were not working:

Missing alerts ✅
This has been fixed, so it should work from now on. Thanks for catching this up!
Old entities index in request present ✅
When I search any term, it no longer shows that request, so maybe there was leftover code that had been removed.

Error on Kubernetes Clusters entity detail ✅

This is because locally, we don't have the dashboards created by default, as depending on the entity, we use a hardcoded ID to show one dashboard or another, so it fails. To make it work, you need to install the Kubernetes integration to have the dashboards locally.

jennypavlova

LGTM 🚀 It works as expected :)
🔽 Just some nits

...utions/observability/plugins/inventory/server/routes/entities/get_group_by_terms_agg.test.ts

x-pack/solutions/observability/plugins/inventory/server/routes/entities/route.ts

...rm/plugins/shared/entity_manager/server/lib/v2/definitions/identity_fields_by_source.test.ts

...solutions/observability/plugins/inventory/public/components/entity_group_accordion/index.tsx

...s/observability/plugins/inventory/public/components/search_bar/entity_types_multi_select.tsx

iblancof · 2025-01-08T09:30:54Z

Hi @rmyz, thanks for the update!
I tested locally again the PR to double check.

Missing alerts

This has been fixed, so it should work from now on. Thanks for catching this up!

Got the alerts now, nice! 🚀

Old entities index in request present

When I search any term, it no longer shows that request, so maybe there was leftover code that had been removed.

I don't see the requests anymore but it seems the search got broken.
The suggested filters are not appearing anymore:

Current	Expected

The request seems to be made to /internal/inventory/entities/types and it feels strange to query that endpoint from the search bar.

Error on Kubernetes Clusters entity detail

This is because locally, we don't have the dashboards created by default, as depending on the entity, we use a hardcoded ID to show one dashboard or another, so it fails. To make it work, you need to install the Kubernetes integration to have the dashboards locally.

I did not try it, but that makes sense, thanks!

rmyz · 2025-01-08T10:36:35Z

Thanks again for testing it @iblancof

The request seems to be made to /internal/inventory/entities/types and it feels strange to query that endpoint from the search bar.

This is expected, by how the whole flow is built, we first query the entity types, then the entities according to those types, in any case, if we query something that's exclusively for one entity, we would only get that entity type and get the entity once we open the dropdown.

I don't see the requests anymore but it seems the search got broken.
The suggested filters are not appearing anymore

I'm currently investigating this, no changes were made to the search component, so this may be broken for some time.

rmyz · 2025-01-08T12:01:57Z

I don't see the requests anymore but it seems the search got broken.
The suggested filters are not appearing anymore

I found the issue @iblancof, we are using a wrong dataView, causing the search component to not show filter suggestions.

As the whole Inventory (also search) cannot be used because it's currently broken (EEM did their changes, which are not compatible with what we have) until we merge this PR, we will fix the search in #204026.

…apping'

elasticmachine · 2025-01-08T13:14:37Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: a9dc255
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-203452-a9dc255c5110

Failed CI Steps

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`inventory`	237	236	-1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`entityManager`	43	45	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`inventory`	231.7KB	231.5KB	-249.0B

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`inventory`	4	3	-1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`entityManager`	8.5KB	8.5KB	-5.0B
`observabilityShared`	93.3KB	93.7KB	+442.0B
total			+437.0B

Unknown metric groups

API count

id	before	after	diff
`entityManager`	43	45	+2

History

💛 Build #265028 was flaky 326df03
💚 Build #264872 succeeded da9218d
💚 Build #264746 succeeded 7612f15
💚 Build #264626 succeeded f580557
💛 Build #263808 was flaky 68b6f3c

cc @rmyz

iblancof · 2025-01-08T13:25:18Z

I found the issue @iblancof, we are using a wrong dataView, causing the search component to not show filter suggestions.

As the whole Inventory (also search) cannot be used because it's currently broken (EEM did their changes, which are not compatible with what we have) until we merge this PR, we will fix the search in #204026.

Thanks for all the updates @rmyz!

kibanamachine · 2025-01-08T14:01:40Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12672105450

kibanamachine · 2025-01-08T14:06:46Z

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 203452

Questions ?

Please refer to the Backport tool documentation

## Summary Closes elastic#202295 Closes elastic#202296 This PR adapts Inventory to use the new Entity v2 endpoints. ## Testing - Use any synthtrace scenario that loads service/hosts/containers data - Navigate and make sure everything works as expected (navigation to Discovery/Infra/Services pages, interacting with the table, searching for some specific entity, interacting with the type filter) - To check the alerts work, it's easier to connect to a remote cluster. --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Jenny <[email protected]>

rmyz · 2025-01-08T14:46:41Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

## Summary Closes elastic#202295 Closes elastic#202296 This PR adapts Inventory to use the new Entity v2 endpoints. ## Testing - Use any synthtrace scenario that loads service/hosts/containers data - Navigate and make sure everything works as expected (navigation to Discovery/Infra/Services pages, interacting with the table, searching for some specific entity, interacting with the type filter) - To check the alerts work, it's easier to connect to a remote cluster. --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Jenny <[email protected]> (cherry picked from commit eb919c9) # Conflicts: # x-pack/solutions/observability/plugins/inventory/server/routes/entities/get_group_by_terms_agg.test.ts # x-pack/solutions/observability/plugins/inventory/server/routes/entities/get_group_by_terms_agg.ts # x-pack/solutions/observability/plugins/inventory/server/routes/entities/get_latest_entities_alerts.ts # x-pack/solutions/observability/plugins/inventory/server/routes/entities/route.ts

#205534) Closes #203095 Closes #204263 ## Summary This PR removes the page for the enabling functionality and welcome screen. After the migration to v2 API we don't need to enable it anymore as we are not using transforms ### Before: ![image](https://github.com/user-attachments/assets/0c8d5841-9189-4551-a1be-87801cfcf57d) ![image](https://github.com/user-attachments/assets/75a73dd8-6b16-452b-abcf-41e1fcca6645) ### After: The same without the extra step to enable the entities and the welcome screen: - No data: ![image](https://github.com/user-attachments/assets/dbfdf501-36e7-4b6f-a8b6-ed0a6748ab62) - With data: ![image](https://github.com/user-attachments/assets/0bbb2983-4e0a-477c-ac11-d2256b5ff854) ## Testing - In a local environment enable the entities feature flag ( it should be a clean env as the entities should not be enabled before ): <img width="1911" alt="image" src="https://github.com/user-attachments/assets/75d6f77d-5039-41ca-80ca-34c3bf99844e" /> - Go to Inventory - Check before and after ingesting data - Synthtrace: `node scripts/synthtrace logs_traces_hosts.ts` - ~⚠️ this case can't be checked before #203452 is merged~ - #203452 is merged 🎉

elastic#205534) Closes elastic#203095 Closes elastic#204263 ## Summary This PR removes the page for the enabling functionality and welcome screen. After the migration to v2 API we don't need to enable it anymore as we are not using transforms ### Before: ![image](https://github.com/user-attachments/assets/0c8d5841-9189-4551-a1be-87801cfcf57d) ![image](https://github.com/user-attachments/assets/75a73dd8-6b16-452b-abcf-41e1fcca6645) ### After: The same without the extra step to enable the entities and the welcome screen: - No data: ![image](https://github.com/user-attachments/assets/dbfdf501-36e7-4b6f-a8b6-ed0a6748ab62) - With data: ![image](https://github.com/user-attachments/assets/0bbb2983-4e0a-477c-ac11-d2256b5ff854) ## Testing - In a local environment enable the entities feature flag ( it should be a clean env as the entities should not be enabled before ): <img width="1911" alt="image" src="https://github.com/user-attachments/assets/75d6f77d-5039-41ca-80ca-34c3bf99844e" /> - Go to Inventory - Check before and after ingesting data - Synthtrace: `node scripts/synthtrace logs_traces_hosts.ts` - ~⚠️ this case can't be checked before elastic#203452 is merged~ - elastic#203452 is merged 🎉 (cherry picked from commit d6e28f7)

…y model (#205534) (#205958) # Backport This will backport the following commits from `main` to `8.x`: - [[ECO][Inventory v2] Remove the landing page to enable the entity model (#205534)](#205534)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: jennypavlova <[email protected]>

rmyz added 2 commits December 9, 2024 12:17

refactor: use new entity v2 endpoints

35a4661

refactor: rename file

5b77cbb

rmyz self-assigned this Dec 9, 2024

refactor: fix get entities

5db6bfd

rmyz changed the title ~~202295 eco use new entity apis for inventory~~ [ECO] Use new Entity v2 endpoints in Inventory Dec 9, 2024

refactor: add again type

1f35e17

kibanamachine and others added 2 commits December 9, 2024 16:41

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

2cb0673

…fix'

Merge branch 'main' into 202295-eco-use-new-entity-apis-for-inventory

154e573

rmyz added 11 commits December 12, 2024 12:57

Merge branch 'main' into 202295-eco-use-new-entity-apis-for-inventory

b0baef6

Merge branch 'main' into 202295-eco-use-new-entity-apis-for-inventory

2df26b8

test: fix tests

1b44d42

Merge branch 'main' into 202295-eco-use-new-entity-apis-for-inventory

48738b9

Merge branch 'main' into 202295-eco-use-new-entity-apis-for-inventory

4d79f00

Merge branch 'main' into 202295-eco-use-new-entity-apis-for-inventory

d4900ce

# Conflicts: # x-pack/plugins/observability_solution/inventory/server/routes/entities/route.ts

Merge branch 'main' into 202295-eco-use-new-entity-apis-for-inventory

6d57935

refactor: use display_name

d53dd0f

refactor: use count endpoint

fe9f305

refactor: temporary remove alerts as they will be added back in anoth…

802b405

…er issue

Merge branch 'main' into 202295-eco-use-new-entity-apis-for-inventory

2272a27

rmyz marked this pull request as ready for review December 14, 2024 12:41

rmyz requested review from a team as code owners December 14, 2024 12:41

rmyz added release_note:skip Skip the PR/issue when compiling release notes backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team labels Dec 14, 2024

[CI] Auto-commit changed files from 'node scripts/yarn_deduplicate'

4d74740

botelastic bot added the ci:project-deploy-observability Create an Observability project label Dec 14, 2024

jennypavlova mentioned this pull request Jan 3, 2025

[ECO][Inventory v2] Remove the landing page to enable the entity model #205534

Merged

rmyz added 4 commits January 7, 2025 10:15

Merge branch 'main' into 202295-eco-use-new-entity-apis-for-inventory

f580557

refactor: PR comments

5c1d41f

refactor: remove unneeded cast

7612f15

fix: flatten alerts

cf6def8

Merge branch 'main' into 202295-eco-use-new-entity-apis-for-inventory

da9218d

jennypavlova approved these changes Jan 7, 2025

View reviewed changes

rmyz added 2 commits January 8, 2025 10:21

refactor: fix CR comments

d1d20bc

Merge branch 'main' into 202295-eco-use-new-entity-apis-for-inventory

326df03

rmyz added 2 commits January 8, 2025 12:55

refactor: remove unused files

365ffa6

refactor: use const instead of hardcoded value

483d0b2

[CI] Auto-commit changed files from 'node scripts/styled_components_m…

a9dc255

…apping'

rmyz merged commit eb919c9 into elastic:main Jan 8, 2025
8 checks passed

kibanamachine added the v9.0.0 label Jan 8, 2025

rmyz deleted the 202295-eco-use-new-entity-apis-for-inventory branch January 8, 2025 14:39

rmyz mentioned this pull request Jan 8, 2025

[8.x] [ECO] Use new Entity v2 endpoints in Inventory (#203452) #205911

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ECO] Use new Entity v2 endpoints in Inventory #203452

[ECO] Use new Entity v2 endpoints in Inventory #203452

rmyz commented Dec 9, 2024 •

edited by kibanamachine

Loading

rmyz commented Dec 9, 2024

rmyz commented Dec 10, 2024

elasticmachine commented Dec 14, 2024

rmyz commented Jan 7, 2025 •

edited

Loading

jennypavlova left a comment

iblancof commented Jan 8, 2025

rmyz commented Jan 8, 2025

rmyz commented Jan 8, 2025 •

edited

Loading

elasticmachine commented Jan 8, 2025 •

edited

Loading

API count

iblancof commented Jan 8, 2025

kibanamachine commented Jan 8, 2025

kibanamachine commented Jan 8, 2025

rmyz commented Jan 8, 2025

[ECO] Use new Entity v2 endpoints in Inventory #203452

[ECO] Use new Entity v2 endpoints in Inventory #203452

Conversation

rmyz commented Dec 9, 2024 • edited by kibanamachine Loading

Summary

Testing

rmyz commented Dec 9, 2024

rmyz commented Dec 10, 2024

elasticmachine commented Dec 14, 2024

rmyz commented Jan 7, 2025 • edited Loading

jennypavlova left a comment

Choose a reason for hiding this comment

iblancof commented Jan 8, 2025

rmyz commented Jan 8, 2025

rmyz commented Jan 8, 2025 • edited Loading

elasticmachine commented Jan 8, 2025 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Public APIs missing exports

Page load bundle

API count

History

iblancof commented Jan 8, 2025

kibanamachine commented Jan 8, 2025

kibanamachine commented Jan 8, 2025

💔 All backports failed

Manual backport

Questions ?

rmyz commented Jan 8, 2025

💚 All backports created successfully

Questions ?

rmyz commented Dec 9, 2024 •

edited by kibanamachine

Loading

rmyz commented Jan 7, 2025 •

edited

Loading

rmyz commented Jan 8, 2025 •

edited

Loading

elasticmachine commented Jan 8, 2025 •

edited

Loading