[ECO][Inventory v2] Ad hoc data view: Add get entities definition endpoint using sources #204026

jennypavlova · 2024-12-12T13:48:53Z

Closes #202298

This PR changes the way we get the entity index patterns to v2. It creates an endpoint part of the inventory API which returns the index patterns by entity type.

Testing

Test the endpoint:

Open Dev tools and add
GET kbn:/internal/inventory/entity/definitions/sources
Response:

Test in the UI

After the previous steps add some host data (oblt cluster / metricbeat) or use synthtrace (for example use node scripts/synthtrace infra_hosts_with_apm_hosts --scenarioOpts.numInstances=10 or node scripts/synthtrace logs_traces_hosts.ts)
Go to Inventory and expand the host group
Click on the actions button for any host and click on the Discover link
The correct dataview should be selected based on the index patterns in the source definition
The same can be done for other entity types
Test the search bar as well (the suggestions should be visible) and now we should have 1 request to get the sources (instead of doing it on click)

Link_Search_Test.mov

… the fetch hook

…se_discover_redirect.ts Co-authored-by: Carlos Crespo <[email protected]>

elasticmachine · 2024-12-12T13:48:56Z

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

github-actions · 2024-12-12T13:49:07Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

...s/observability_solution/inventory/server/routes/entity_definition/get_entity_definitions.ts

x-pack/plugins/observability_solution/inventory/public/hooks/use_discover_redirect.ts

...s/observability_solution/inventory/server/routes/entity_definition/get_entity_definitions.ts

kpatticha · 2024-12-18T20:15:27Z

x-pack/plugins/observability_solution/inventory/public/hooks/use_discover_redirect.ts

-    entity.entityDefinitionId
-  );
+  const { entityDefinitionIndexPatterns, isEntityDefinitionIndexPatternsLoading } =
+    useFetchEntityDefinitionIndexPattern(entity.entityType);


Entities of the same type are expected to have the same index pattern, right?

Calling useFetchEntityDefinitionIndexPattern in use_discover_redirect means that for each row a request to inventory/entity/definitions/sources?type={type} will be triggered.

What do you think of sending the request only when the panel is expanded? Alternatively, we could fetch the existing entity types, such as services and hosts, in a single request:

and request inventory/entity/definitions/sources?types=services,hosts

This could return a response like:

definitionIndexPatterns: { "services": "metrics-apm*, logs-apm*", "hosts": "metrics-infra*" }

This would reduce redundant requests and improve scalability. What do you think?

That's a good idea! I would totally move the fetching outside of the hook to a higher level and use it once per type.

Alternatively, we could fetch the existing entity types, such as services and hosts, in a single request:

and request inventory/entity/definitions/sources?types=services,hosts

This could return a response like:

definitionIndexPatterns: {
"services": "metrics-apm*, logs-apm*",
"hosts": "metrics-infra*"
}

I tried to do it as suggested with multiple types (here) but the EEM API expects a single type so even if we combine our API response we will still need to call the EEM API several times which is not necessary for the types before we expand the group (Which will make requests to EEM which are not needed at this point as some of the groups are not opened)

What do you think of sending the request only when the panel is expanded?

I would prefer that option rather than the combined response for multiple types and get only the index patterns for the selected group by it's type

EEM API expects a single type

As the main consumers of the API, we have the privilege of requesting an extension if needed. 😊

I would prefer that option rather than the combined response for multiple types and get only the index patterns for the selected group by it's type

sounds good to me.

I moved the logic to get the index patterns to the grouped_entities_grid.tsx and passed it down to the grid so we will do one request per group. Currently, it's tricky to test without the changes from #203452 I will retest it once we merge it.

Update: I ended up changing to multiple types having the type_id as key and index patterns as value like:

{ built_in_hosts_from_ecs_data: [ 'filebeat-*', 'logs-*', 'metrics-*', 'metricbeat-*' ] }

EEM API expects a single type

As the main consumers of the API, we have the privilege of requesting an extension if needed. 😊

I tried first to use what we have but I will comment on the issue based on my findings here to request that 😁

…ntory-v2-adhoc-data-view

kpatticha · 2025-01-13T11:18:38Z

...ns/observability/plugins/inventory/server/routes/entity_definition/get_entity_definitions.ts

+    return {
+      definitionIndexPatterns: {
+        ...Object.fromEntries(
+          Array.from(new Set(allEntityDefinitionIndexPatterns)).flatMap(Object.entries)


Is this necessary? Since the array is already structured in line 24, couldn't be formatted there?

I don't think I understand the issue. The idea of this is to format the data in a format where all the unique values are extracted and converted to an object like: { unique type name: [ all unique array values (without duplicates)] } - the mapping will return an array were here we want to get the unique key/values.

For context: we can have more than one source per type (comment) so from the sources we can have for example: type n with 2 different index pattern arrays: [index1], [index2] ... and [indexN] and we want to transform [{n: [index1]},....] to {n: [index1, index2....]} - that's why we need the set conversion/flatmap conversion of the allEntityDefinitionIndexPatterns

Does it make sense now?

Update: Based on the suggestions I see that that wasn't super clear so I refactored that part and added a test for it in this commit 68d076c

I didn't know that one type can have 1 or n sources. Thanks 💯

kpatticha · 2025-01-13T11:22:29Z

x-pack/solutions/observability/plugins/inventory/public/components/entity_actions/index.tsx

@@ -24,32 +24,28 @@ export const EntityActions = ({ entity, setShowActions }: Props) => {
    ? `inventoryEntityActionsButton-${entity.entityDisplayName}`
    : 'inventoryEntityActionsButton';

-  const { getDiscoverEntitiesRedirectUrl, isEntityDefinitionLoading } = useDiscoverRedirect(entity);
+  const { getDiscoverEntitiesRedirectUrl } = useDiscoverRedirect(entity);


I noticed that the ad hoc data view is created for each row causing redundant requests

Screen.Recording.2025-01-13.at.13.20.38.mov

Could you please take a look

Thanks for spotting that! I missed that after the refactoring - so I moved the discover link dataview to the context and set the entity type once the grid is opened so now it should work as expected:

discover_link_fix.mov

Can you please check again?

...ns/observability/plugins/inventory/server/routes/entity_definition/get_entity_definitions.ts

.../inventory/server/routes/entity_definition/extract_entity_index_patterns_from_definitions.ts

kpatticha · 2025-01-14T13:16:20Z

...ntory/server/routes/entity_definition/extract_entity_index_patterns_from_definitions.test.ts

+import type { EntitySourceDefinition } from '@kbn/entityManager-plugin/server/lib/v2/types';
+import { extractEntityIndexPatternsFromDefinitions } from './extract_entity_index_patterns_from_definitions';
+
+describe('extractEntityIndexPatternsFromDefinitions', () => {


x-pack/solutions/observability/plugins/inventory/public/hooks/use_discover_redirect.ts

kpatticha

Code LGTM, thanks for addressing the comments.

elasticmachine · 2025-01-14T14:29:02Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: a3ddc74
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-204026-a3ddc742a4f0

Failed CI Steps

x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts

Test Failures

[job] [logs] x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts / Serverless Observability - Deployment-agnostic API integration tests observability AI Assistant When there are knowledge base entries (from 8.15 or earlier) that does not contain semantic_text embeddings "after all" hook in "When there are knowledge base entries (from 8.15 or earlier) that does not contain semantic_text embeddings"
[job] [logs] x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts / Serverless Observability - Deployment-agnostic API integration tests observability AI Assistant When there are knowledge base entries (from 8.15 or earlier) that does not contain semantic_text embeddings "before all" hook in "When there are knowledge base entries (from 8.15 or earlier) that does not contain semantic_text embeddings"

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`inventory`	227.5KB	228.0KB	+446.0B

History

💚 Build #266679 succeeded 01976b7
💛 Build #266529 was flaky b378fa5
💚 Build #266450 succeeded d9ad30c
💔 Build #266338 failed 45144df
💔 Build #266131 failed ffee535

cc @jennypavlova

kibanamachine · 2025-01-14T15:30:56Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12770972776

…point using sources (elastic#204026) Closes elastic#202298 This PR changes the way we get the entity index patterns to v2. It creates an endpoint part of the inventory API which returns the index patterns by entity type. ## Testing ### Test the endpoint: - Open Dev tools and add ` GET kbn:/internal/inventory/entity/definitions/sources` - Response: ![image](https://github.com/user-attachments/assets/3346c36e-dbc2-4e56-9ed6-d3d3a8f7d1a5) ### Test in the UI - After the previous steps add some host data (oblt cluster / metricbeat) or use synthtrace (for example use `node scripts/synthtrace infra_hosts_with_apm_hosts --scenarioOpts.numInstances=10` or `node scripts/synthtrace logs_traces_hosts.ts`) - Go to Inventory and expand the host group - Click on the actions button for any host and click on the Discover link - The correct dataview should be selected based on the index patterns in the source definition The same can be done for other entity types - Test the search bar as well (the suggestions should be visible) and now we should have 1 request to get the sources (instead of doing it on click) https://github.com/user-attachments/assets/93b5ac6c-9d64-44e0-b26e-6133477e0840  --------- Co-authored-by: Carlos Crespo <[email protected]> Co-authored-by: Sergi Romeu <[email protected]> Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit ffccfdc)

kibanamachine · 2025-01-14T15:36:07Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…on endpoint using sources (#204026) (#206610) # Backport This will backport the following commits from `main` to `8.x`: - [[ECO][Inventory v2] Ad hoc data view: Add get entities definition endpoint using sources (#204026)](#204026)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) \r\n\r\n---------\r\n\r\nCo-authored-by: Carlos Crespo <[email protected]>\r\nCo-authored-by: Sergi Romeu <[email protected]>\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"ffccfdc62cd1da5baf54568cbee20a9a3466178e","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-infra_services"],"title":"[ECO][Inventory v2] Ad hoc data view: Add get entities definition endpoint using sources","number":204026,"url":"https://github.com/elastic/kibana/pull/204026","mergeCommit":{"message":"[ECO][Inventory v2] Ad hoc data view: Add get entities definition endpoint using sources (#204026)\n\nCloses #202298 \r\n\r\nThis PR changes the way we get the entity index patterns to v2. It\r\ncreates an endpoint part of the inventory API which returns the index\r\npatterns by entity type.\r\n\r\n## Testing\r\n\r\n### Test the endpoint: \r\n- Open Dev tools and add\r\n` GET kbn:/internal/inventory/entity/definitions/sources`\r\n- Response: \r\n\r\n\r\n![image](https://github.com/user-attachments/assets/3346c36e-dbc2-4e56-9ed6-d3d3a8f7d1a5)\r\n\r\n\r\n### Test in the UI\r\n- After the previous steps add some host data (oblt cluster /\r\nmetricbeat) or use synthtrace (for example use `node scripts/synthtrace\r\ninfra_hosts_with_apm_hosts --scenarioOpts.numInstances=10` or `node\r\nscripts/synthtrace logs_traces_hosts.ts`)\r\n- Go to Inventory and expand the host group\r\n- Click on the actions button for any host and click on the Discover\r\nlink\r\n- The correct dataview should be selected based on the index patterns in\r\nthe source definition\r\nThe same can be done for other entity types\r\n- Test the search bar as well (the suggestions should be visible) and\r\nnow we should have 1 request to get the sources (instead of doing it on\r\nclick)\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/93b5ac6c-9d64-44e0-b26e-6133477e0840\r\n\r\n\r\n\r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: Carlos Crespo <[email protected]>\r\nCo-authored-by: Sergi Romeu <[email protected]>\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"ffccfdc62cd1da5baf54568cbee20a9a3466178e"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/204026","number":204026,"mergeCommit":{"message":"[ECO][Inventory v2] Ad hoc data view: Add get entities definition endpoint using sources (#204026)\n\nCloses #202298 \r\n\r\nThis PR changes the way we get the entity index patterns to v2. It\r\ncreates an endpoint part of the inventory API which returns the index\r\npatterns by entity type.\r\n\r\n## Testing\r\n\r\n### Test the endpoint: \r\n- Open Dev tools and add\r\n` GET kbn:/internal/inventory/entity/definitions/sources`\r\n- Response: \r\n\r\n\r\n![image](https://github.com/user-attachments/assets/3346c36e-dbc2-4e56-9ed6-d3d3a8f7d1a5)\r\n\r\n\r\n### Test in the UI\r\n- After the previous steps add some host data (oblt cluster /\r\nmetricbeat) or use synthtrace (for example use `node scripts/synthtrace\r\ninfra_hosts_with_apm_hosts --scenarioOpts.numInstances=10` or `node\r\nscripts/synthtrace logs_traces_hosts.ts`)\r\n- Go to Inventory and expand the host group\r\n- Click on the actions button for any host and click on the Discover\r\nlink\r\n- The correct dataview should be selected based on the index patterns in\r\nthe source definition\r\nThe same can be done for other entity types\r\n- Test the search bar as well (the suggestions should be visible) and\r\nnow we should have 1 request to get the sources (instead of doing it on\r\nclick)\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/93b5ac6c-9d64-44e0-b26e-6133477e0840\r\n\r\n\r\n\r\n\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: Carlos Crespo <[email protected]>\r\nCo-authored-by: Sergi Romeu <[email protected]>\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"ffccfdc62cd1da5baf54568cbee20a9a3466178e"}}]}] BACKPORT--> Co-authored-by: jennypavlova <[email protected]>

davismcphee · 2025-01-16T05:26:32Z

x-pack/solutions/observability/plugins/inventory/public/hooks/use_discover_redirect.ts

@@ -37,19 +24,19 @@ export const useDiscoverRedirect = (entity: InventoryEntity) => {
        })
      : '';

-    return application.capabilities.discover?.show
+    return application.capabilities.discover?.show || !dataView


@jennypavlova Sorry to show up on this PR after it's already merged, but I just encountered this line when resolving a merge conflict and thought it was worth verifying. If I'm reading this right, a Discover link will display if the user has the show capability or if there is no dataView. But couldn't the || !dataView here cause the link to show in some cases even when the user doesn't have the required Discover capability?

Thank you for the comment, good catch! I think we can remove the || !dataView or make it an AND condition - this can be addressed as part of #207064 as it will be touching this part.

jennypavlova and others added 8 commits December 9, 2024 14:53

[ECO][Inventory v2]Add get entities definition endpoint and use it in…

0ceb955

… the fetch hook

Merge branch 'main' into 202298-eco-inventory-v2-adhoc-data-view

88692c3

Fix variable name

07248de

Merge branch 'main' into 202298-eco-inventory-v2-adhoc-data-view

e3d2997

Update x-pack/plugins/observability_solution/inventory/public/hooks/u…

93ce402

…se_discover_redirect.ts Co-authored-by: Carlos Crespo <[email protected]>

Update x-pack/plugins/observability_solution/inventory/public/hooks/u…

a74f992

…se_discover_redirect.ts Co-authored-by: Carlos Crespo <[email protected]>

Merge branch 'main' into 202298-eco-inventory-v2-adhoc-data-view

2e112cd

Lint fix

7b96483

jennypavlova added release_note:skip Skip the PR/issue when compiling release notes backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team labels Dec 12, 2024

jennypavlova self-assigned this Dec 12, 2024

jennypavlova requested a review from a team as a code owner December 12, 2024 13:48

botelastic bot added the ci:project-deploy-observability Create an Observability project label Dec 12, 2024

Merge branch 'main' into 202298-eco-inventory-v2-adhoc-data-view

5783a73

rmyz approved these changes Dec 12, 2024

View reviewed changes

Merge branch 'main' into 202298-eco-inventory-v2-adhoc-data-view

4d58139

rmyz reviewed Dec 18, 2024

View reviewed changes

...s/observability_solution/inventory/server/routes/entity_definition/get_entity_definitions.ts Outdated Show resolved Hide resolved

rmyz added 4 commits December 18, 2024 12:17

refactor: remove unused const

7e1a469

Merge branch 'main' into 202298-eco-inventory-v2-adhoc-data-view

e57e287

Merge branch 'main' into 202298-eco-inventory-v2-adhoc-data-view

8853421

Merge branch 'main' into 202298-eco-inventory-v2-adhoc-data-view

3e06456

klacabane reviewed Dec 18, 2024

View reviewed changes

x-pack/plugins/observability_solution/inventory/public/hooks/use_discover_redirect.ts Outdated Show resolved Hide resolved

klacabane reviewed Dec 18, 2024

View reviewed changes

...s/observability_solution/inventory/server/routes/entity_definition/get_entity_definitions.ts Outdated Show resolved Hide resolved

rmyz mentioned this pull request Dec 18, 2024

[ECO] Use new Entity v2 endpoints in Inventory #203452

Merged

kpatticha reviewed Dec 18, 2024

View reviewed changes

Merge branch 'main' of github.com:elastic/kibana into 202298-eco-inve…

0d0504e

…ntory-v2-adhoc-data-view

jennypavlova requested a review from a team as a code owner January 2, 2025 10:43

klacabane approved these changes Jan 13, 2025

View reviewed changes

kpatticha reviewed Jan 13, 2025

View reviewed changes

Merge branch 'main' into 202298-eco-inventory-v2-adhoc-data-view

d9ad30c

klacabane reviewed Jan 13, 2025

View reviewed changes

...ns/observability/plugins/inventory/server/routes/entity_definition/get_entity_definitions.ts Outdated Show resolved Hide resolved

jennypavlova added 3 commits January 13, 2025 13:57

Move discover data view to the context

b5a2389

Refactor index patterns extraction

68d076c

Merge branch 'main' into 202298-eco-inventory-v2-adhoc-data-view

b378fa5

jennypavlova requested a review from kpatticha January 13, 2025 14:27

klacabane reviewed Jan 13, 2025

View reviewed changes

.../inventory/server/routes/entity_definition/extract_entity_index_patterns_from_definitions.ts Outdated Show resolved Hide resolved

jennypavlova force-pushed the 202298-eco-inventory-v2-adhoc-data-view branch from 5afc869 to 03f968b Compare January 13, 2025 18:30

CR changes

1f03ce0

jennypavlova force-pushed the 202298-eco-inventory-v2-adhoc-data-view branch from 03f968b to 1f03ce0 Compare January 13, 2025 18:31

Merge branch 'main' into 202298-eco-inventory-v2-adhoc-data-view

01976b7

kpatticha reviewed Jan 14, 2025

View reviewed changes

x-pack/solutions/observability/plugins/inventory/public/hooks/use_discover_redirect.ts Outdated Show resolved Hide resolved

kpatticha approved these changes Jan 14, 2025

View reviewed changes

jennypavlova added 2 commits January 14, 2025 14:22

CR changes

cc9d860

Merge branch 'main' into 202298-eco-inventory-v2-adhoc-data-view

a3ddc74

jennypavlova merged commit ffccfdc into elastic:main Jan 14, 2025
9 checks passed

kibanamachine added the v9.0.0 label Jan 14, 2025

kibanamachine mentioned this pull request Jan 14, 2025

[8.x] [ECO][Inventory v2] Ad hoc data view: Add get entities definition endpoint using sources (#204026) #206610

Merged

kibanamachine added the v8.18.0 label Jan 14, 2025

davismcphee reviewed Jan 16, 2025

View reviewed changes

davismcphee mentioned this pull request Jan 18, 2025

[Inventory v2] Open Explore in Discover link in a new tab #207064

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ECO][Inventory v2] Ad hoc data view: Add get entities definition endpoint using sources #204026

[ECO][Inventory v2] Ad hoc data view: Add get entities definition endpoint using sources #204026

jennypavlova commented Dec 12, 2024 •

edited by kibanamachine

Loading

elasticmachine commented Dec 12, 2024

github-actions bot commented Dec 12, 2024

kpatticha Dec 18, 2024

jennypavlova Jan 2, 2025

kpatticha Jan 3, 2025

jennypavlova Jan 3, 2025

jennypavlova Jan 10, 2025

kpatticha Jan 13, 2025

jennypavlova Jan 13, 2025

jennypavlova Jan 13, 2025

kpatticha Jan 14, 2025 •

edited

Loading

kpatticha Jan 13, 2025

jennypavlova Jan 13, 2025

kpatticha Jan 14, 2025

kpatticha left a comment •

edited

Loading

elasticmachine commented Jan 14, 2025 •

edited

Loading

kibanamachine commented Jan 14, 2025

kibanamachine commented Jan 14, 2025

davismcphee Jan 16, 2025

jennypavlova Jan 20, 2025

[ECO][Inventory v2] Ad hoc data view: Add get entities definition endpoint using sources #204026

[ECO][Inventory v2] Ad hoc data view: Add get entities definition endpoint using sources #204026

Conversation

jennypavlova commented Dec 12, 2024 • edited by kibanamachine Loading

Testing

Test the endpoint:

Test in the UI

elasticmachine commented Dec 12, 2024

github-actions bot commented Dec 12, 2024

🤖 GitHub comments

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kpatticha Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kpatticha left a comment • edited Loading

Choose a reason for hiding this comment

elasticmachine commented Jan 14, 2025 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Async chunks

History

kibanamachine commented Jan 14, 2025

kibanamachine commented Jan 14, 2025

💚 All backports created successfully

Questions ?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jennypavlova commented Dec 12, 2024 •

edited by kibanamachine

Loading

kpatticha Jan 14, 2025 •

edited

Loading

kpatticha left a comment •

edited

Loading

elasticmachine commented Jan 14, 2025 •

edited

Loading