[Security Solution] Users can Customize Prebuilt Detection Rules: Milestone 3

[banderror]

Epic: https://github.com/elastic/security-team/issues/1974 (internal)
Milestones: << • >>

Status: In development.

Summary

Milestone 3: Add support for customizing prebuilt rules. Extend the rule upgrade workflow with full support for 3-way diffs and conflict resolution. Allow users to:

• Edit and customize prebuilt rules
• Export and import prebuilt rules, including customized ones
• Upgrade prebuilt rules while keeping the user customizations whenever possible

User stories

Prebuilt rule customization workflow:

• User can edit a single prebuilt rule
  • User can click "edit" button for prebuilt rules and customize (almost) any field on the Rule Editing page, just like it's possible to do with custom rules
  • User can't edit the Author and License fields
• User can bulk edit multiple prebuilt rules via bulk actions
• User can see if the rule is customized on the Rule Details page
• User can see which rules are customized on the Rule Management page

Prebuilt rule upgrade workflow:

• User can upgrade a single prebuilt rule to its latest version with previewing the incoming updates
  • User can preview updates from Elastic, for each rule field that has an update from Elastic
  • User can preview their customizations, for each rule field that was customized
  • User can compare their customizations with updates from Elastic and see if there are any conflicts between them, per each rule field
  • User can manually resolve conflicts between their customizations and updates from Elastic, per each rule field
  • User can edit the final field values before submitting the update
  • User can upgrade a rule if its type has been changed by Elastic in the latest version, but can only accept the incoming changes
• User can upgrade a single prebuilt rule to its latest version without previewing the incoming updates
• User can bulk upgrade multiple prebuilt rules to their latest versions

Prebuilt rule export/import workflow:

• User can export a single prebuilt rule
  • Pages: Rule Details, Rule Management
  • It can be a prebuilt non-customized or prebuilt customized rule
• User can bulk export multiple prebuilt rules via bulk actions
  • Pages: Rule Management
  • We support exporting prebuilt non-customized, prebuilt customized, and custom rules in any combination
• User can bulk import multiple prebuilt rules
  • Pages: Rule Management
  • We support importing prebuilt non-customized, prebuilt customized, and custom rules - in any combination

Useful info

• RFC: Prebuilt Rules Customization - a new technical design document describing in detail how prebuilt rule customization should work and affect other existing features.
• Prebuilt Rules Customization Technical Design - an older document from Milestone 1.

Design

Technical design

Preview Give feedback

1. [Security Solution] Write an RFC for customizing prebuilt rules #171309
   8.14 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +5
   
2. [Security Solution] Discuss rule fields in the context of the prebuilt rule upgrade workflow #147239
   8.14 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp discuss +6
   

   
3. [Security Solution] Discuss rule field diffs and conflicts to handle in the UI #148913
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp discuss release_note:skip skip-ci +7
   

UI/UX design

Preview Give feedback

1. [Security Solution] Discuss UI/UX designs for customizing prebuilt rules #178211
   8.14 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp design +6
2. Design rule management changes (Rule Management and Details pages)
3. Design rule customization changes (Rule Management and Editing pages)
4. Design rule installation and upgrade changes (Add Rules page, Rule updates table, rule upgrade details flyout)
5. Design rule upgrade UI for rule type changes (implementation ticket)

Preparatory changes

Preparatory changes is something we can work on before starting to hide functionality behind a feature flag. This will reduce the overall complexity introduced by feature toggling.

Missing UI for editing certain rule fields

Preview Give feedback

1. [Security Solution] Allow users to edit setup field for custom rules #173626
   8.14 candidate Feature:Rule Creation Feature:Rule Edit Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.14.0 +8
   
2. [Security Solution] Allow users to edit max_signals field for custom rules #173593
   8.15 candidate Feature:Rule Creation Feature:Rule Details Feature:Rule Edit Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0 +9
   
3. [Security Solution] Allow users to edit related_integrations field for custom rules #173595
   8.15 candidate Feature:Rule Creation Feature:Rule Edit Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0 +8
   
4. [Security Solution] Allow users to edit required_fields field for custom rules #173594
   8.15 candidate Feature:Rule Creation Feature:Rule Edit Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0 +8
   

Missing UI for editing certain rule fields (docs)

Preview Give feedback

1. Edit setup guide field for custom rules in UI and API security-docs#4917
   3 of 3
   Docset: ESS Docset: Serverless Feature: Rules Team: Detections/Response v8.14.0 +5
   
2. Edit max_signals field for custom rules in UI and API security-docs#5029
   3 of 3
   Docset: ESS Docset: Serverless Feature: Rules Team: Detections/Response v8.15.0 +5
   
3. Edit related_integrations field for custom rules in UI and API security-docs#5099
   3 of 3
   Docset: ESS Docset: Serverless Feature: Rules Team: Detections/Response v8.15.0 +5
   
4. Edit required_fields field for custom rules in UI and API security-docs#5131
   3 of 3
   Docset: ESS Docset: Serverless Feature: Rules Team: Detections/Response v8.15.0 +5
   

Schema-related changes

Preview Give feedback

1. [Security Solution] Add rule_source to the API schema #180122
   8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.15.0 +6
   
2. [Security Solution] Make changes in the internal rule schema #180124
   8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.15.0 +6
   
3. [Security Solution] Replaced the incorrect runtime type used for ruleSource #184004
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp backport:skip bug impact:critical release_note:skip v8.15.0 +9
   
4. [Security Solution] Write the rule_source field together with immutable #180141
   8.15 candidate Feature:Prebuilt Detection Rules Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.15.0 +7
   
5. [Security Solution] RulesManagementClient refactoring. Part 1 #180128
   8.15 candidate Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp refactoring v8.15.0 +7
   
6. [Security Solution] DetectionRulesClient refactoring. Part 2 #184364
   8.16 candidate Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp refactoring v8.15.0 v8.16.0 +8
   
7. [Security Solution] Implement normalization on read for rule_source and immutable fields #180140
   8.16 candidate Feature:Prebuilt Detection Rules Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +7
   

Rule customization, API changes

Preview Give feedback

1. [Security Solution] Calculate and save ruleSource.isCustomized in API endpoint handlers #180145
   8.16 candidate Feature:Prebuilt Detection Rules Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +7
   
2. [Security Solution] Calculate and save ruleSource.isCustomized in bulk edit API #187706
   8.16 candidate Feature:Prebuilt Detection Rules Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +7
   
3. [Security Solution] Make rule_source field required in RuleResponse #180270
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +6
   
4. [Security Solution] Prevent from updating "non-customizable" fields in prebuilt rules #180273
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +6
   

Rule upgrade, API changes

Preview Give feedback

1. [Security Solution] Return user-customized fields from the POST /prebuilt_rules/upgrade/_review API endpoint even if they haven't been updated by Elastic in the target version #180154
   8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.15.0 +6
   
2. [Security Solution] Extend the POST /upgrade/_review API endpoint's contract and functionality #180153
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +6
   
3. [Security Solution] Handle specific fields in /upgrade/_review upgrade workflow #180393
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +6
   
4. [Security Solution] Substitute MissingVersion symbol in the ThreeWayDiff object with a boolean #188277
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp refactoring +6
   
5. [Security Solution] Extend the POST /upgrade/_perform API endpoint's contract and functionality #166376
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0 +7
   
6. [Security Solution] Handle specific fields in /upgrade/_perform endpoint upgrade workflow #186544
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +6
   
7. [Security Solution] Add InvestigationFields and AlertSuppression fields to the upgrade workflow #190597
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +5
   
8. [Security Solution] Remove exceptions_list, author and license from Diffable Rule #196213
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +6
   

Rule upgrade, diff algorithms

Preview Give feedback

1. [Security Solution] Implement number diff algorithm #180160
   8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0 +7
   
2. [Security Solution] Implement single-line string diff algorithm #180158
   8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0 +7
   
3. [Security Solution] Implement multi-line string diff algorithm #180159
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0 +7
   
4. [Security Solution] Implement array of scalar values diff algorithm #180162
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0 +7
   
5. [Security Solution] Implement data source fields diff algorithm #187659
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0 +7
   
6. [Security Solution] Implement query fields diff algorithms #187658
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0 +7
   
7. [Security Solution] Implement concurrent_searches and items_per_search fields diff algorithms #188061
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +6
   
8. [Security Solution] Implement rule type diff algorithm #190482
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0 +7
   

Fleet package with prebuilt rules

Preview Give feedback

1. [Security Solution] OOMs during prebuilt rules package installation #187969
   3 of 3
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet v8.16.0 +7
   
2. [Fleet] All package SOs are being read into memory during package upgrade or force installation #187975
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet technical debt +7
   
3. [Fleet] Split bulk delete into chunks to avoid high memory consumption #188208
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet technical debt +7
   
4. [Fleet] Split asset installation into chunks to avoid high memory pressure #189043
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet +6
   
5. [Security Solution] API endpoint for the package with prebuilt rules #187647
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +6
   
6. [Fleet] Stream-based programmatic API for installing packages #187646
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet +5
   
7. [Security Solution] Stream-based installation of the package with prebuilt rules #192350
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet performance v8.17.0 +8
   
8. [Security Solution] Smart limits for the package with prebuilt rules #187645
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.17.0 v8.18.0 +7
   

Changes hidden behind the feature flag

These are changes that will need to be hidden behind the prebuiltRulesCustomizationEnabled feature flag.

Rule customization, UI changes

Preview Give feedback

1. [Security Solution] Add a new feature flag prebuiltRulesCustomizationEnabled #180130
   8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.15.0 +6
   
2. [Security Solution] Show if a rule is customized on the Rule Details page #180170
   8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0 +7
   
3. [Security Solution] Support filtering by "Modified rules" and "Unmodified rules" #180169
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +6
   
4. [Security Solution] Add support for editing prebuilt rules to the Rule Management and Rule Details pages #180171
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +5
   
5. [Security Solution] Add support for editing prebuilt rules to the Rule Editing page #180172
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +5
   

Rule upgrade, UI changes

Preview Give feedback

1. [Security Solution] Handle conflicting rules in the Rule Upgrade table #180589
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp needs design +6
   

   
2. [Security Solution] Rework Update flyout to display all field updates and build Three-Way-Diff field component #171520
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0 v8.17.0 v8.18.0 +9
   
3. [Security Solution] Handle specific fields in the upgrade workflow's UI #188065
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0 +6
   
4. [Security Solution] Implement UI for updating prebuilt rule to a new rule type (MVP) #180395
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0 +6
   
5. [Security Solution] Show Modified badge for fields customized by users #203718
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +6
   

Rule export and import, API and UI changes

Preview Give feedback

1. [Security Solution] Allow exporting prebuilt rules at the API level #180167
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +6
   
2. [Security Solution] Allow importing prebuilt rules at the API level #180168
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +6
   
3. [Security Solution] Support exporting prebuilt rules from the Rule Management page #180173
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +5
   
4. [Security Solution] Support exporting prebuilt rules from the Rule Details page #180176
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +5
   
5. [Security Solution] Benchmark performance of importing a large number of prebuilt rules #195632
   8.18 candidate Feature:Prebuilt Detection Rules Feature:Rule Import/Export Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0 +7
   

Licensing

Preview Give feedback

1. https://github.com/elastic/security-team/issues/10410
2. https://github.com/elastic/security-team/issues/11502
3. https://github.com/elastic/security-team/issues/11504

Telemetry

Preview Give feedback

1. [Security Solution] Collect usage statistics for prebuilt rule customization #140369
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp needs product telemetry +7
   

Before release

Bugs

Preview Give feedback

1. [Security Solution] Users can Customize Prebuilt Detection Rules: Milestone 3 bugs #201502
   44 of 55
   8.18 candidate Feature:Prebuilt Detection Rules Meta Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 v8.17.0 v8.18.0 +9
   

   

Testing

Preview Give feedback

1. [Security Solution] Write tests for "Customized Elastic rule" badge on the Rule Details page #186916
   8.18 candidate Feature:Prebuilt Detection Rules Feature:Rule Details Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp technical debt test-coverage v8.18.0 +9
   
2. [Security Solution] Tests for prebuilt rule customization workflow #202068
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test test-coverage test-plan v8.18.0 +9
   
3. [Security Solution] Tests for prebuilt rule upgrade workflow #202078
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test test-coverage test-plan v8.18.0 +9
   
4. [Security Solution] Tests for prebuilt rule import/export workflow #202079
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test test-coverage test-plan v8.18.0 +9
   
5. [Security Solution] Acceptance testing of prebuilt rule customization workflows #180397
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0 +6
   
6. [Security Solution] Exploratory testing of prebuilt rule customization workflows #180398
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0 +6
   

Documentation

Preview Give feedback

1. [Request] Prebuilt rule customization and upgrade workflows (DRAFT) security-docs#5061
   v8.18.0 +1
   
2. [Request] Document new rule_source property for rules in the API schema (DRAFT) security-docs#5063
   v8.18.0 +1
   

Release

Preview Give feedback

1. [Security Solution] Release prebuilt rule customization feature #180267
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0 +6

After release

Last changes after releasing the feature

Preview Give feedback

1. [Security Solution] Exploratory testing of prebuilt rule customization workflows in production #180399
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0 +6
   
2. [Security Solution] Remove the prebuiltRulesCustomizationEnabled feature flag #180272
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +5

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[banderror]

Draft plan for Milestone 3

UPD: the plan has been moved to the ticket description.

[nikitaindik]

Posting leftover and optional work from the Upgrade flyout ticket as a comment here, so that @banderror can prioritize it and create necessary tickets.

Product enhancements

• Try to diff JSONs instead of individual subfields
• Research necessity of state persistence in local/session storage
• Add Prefill component that displays a dropdown with versions available for prefill.
• Update RuleDetailsFlyoutHeader flyout header with components that are shown in designs, for example a badge for Customized Elastic rules.
• Add a component that displays the "Modified" badge for the fields that were edited
• Add unsaved changes modal confirmation upon flyout closing
• Show non-diff view of the changes (target rule field)

Refactoring

• Make sure the table and the whole flyout don't re-render on every "save" of a field value
• Extract editable components from Rule creation/editing UI into a separate directory
• (If necessary) Reiterate in React Context after integration to fix/improve the implementation
• Simplify generic types
• Storybook: Sort simple cases before advanced ones
• Storybook: Find a way to add explainer comments in Storybook