Create HIP for Adoption of CUE #364
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,120 @@ | ||
| --- | ||
| hip: "0019" | ||
| title: "Adoption of CUE for Values Processing and Validation in Helm" | ||
| authors: ["Jiří Altman <[email protected]>", "Jan von Löwenstein <[email protected]>"] | ||
| created: "2024-08-07" | ||
| type: "feature" | ||
| status: "draft" | ||
| --- | ||
|
|
||
| ## Abstract | ||
|
|
||
| This HIP proposes the integration of CUE (Configuration Unification Engine) as the primary tool for processing and validating Helm values files, replacing the existing JSON Schema support. As a superset of YAML, JSON, and JSON Schema, CUE extends existing capabilities while maintaining complete backward compatibility. This ensures that all current Helm charts and configurations remain fully supported, with the added benefits of enhanced validation, improved defaulting mechanisms, and easier schema management. | ||
|
There was a problem hiding this comment. Also, I always thought CUE stood for "Configure Unify Execute" (and not: "Configuration Unification Engine"). Where does this alternate come from? |
||
|
|
||
| ## Motivation | ||
|
There was a problem hiding this comment. One significant benefit to utilizing CUE for values processing, is that CUE would allow "pre-processing" of input values (beyond validation). A frequent pain-point of Helm charts is that values tend to be very "raw". It is difficult for values to not simply be the exact inputs to substitute in templates. As any business logic that expands or modifies the values needs to be written in gotemplate, either as a "named template" (function) or inline with the template logic. And gotemplate is both a terrible language to write configuration "business" logic. And the actual rendering of charts IMHO should contain as little business logic as possible (should be relegated as much as possible to dumb value substitution). One motivation to me would be to enable mutation of the input values (which CUE excels at). In order to "expand" the user provided values into values suitable for the (gotemplate) rendering phase. This HIP could just limit CUE to validation only. But it might be worth considering if CUE could be used beyond validation? There was a problem hiding this comment. I think, you might be thinking about the above in this point? If so, I would recommend expanding that
|
||
|
|
||
| The current JSON Schema implementation for validating Helm values files has several notable limitations: | ||
|
|
||
| 1. **Outdated Implementation**: The existing JSON Schema support in Helm is based on Draft-07 and has not been actively maintained. It supports only a subset of the specification and contains non-standard issues, leading to inconsistent behavior and limited validation capabilities. | ||
|
There was a problem hiding this comment. minor note: helm/helm#13283 would mean Helm can support newer json schema versions (if that PR lands, we would want to remove this point I think) |
||
|
|
||
| 2. **Complexity in Schema Management**: Writing and maintaining JSON schemas for Helm values files often requires deep expertise, making it challenging for users unfamiliar with JSON Schema to create and manage their configurations effectively. | ||
|
|
||
| 3. **Limited Features**: Helm values lack advanced features such as defaulting, value referencing, and flexible schema definitions, which are increasingly necessary for handling complex Helm charts. | ||
|
|
||
| CUE addresses these issues by providing: | ||
|
|
||
| - **Support for Modern JSON Schema Specifications**: CUE offers out-of-the-box support for newer JSON Schema specifications, ensuring compatibility with modern features and addressing the limitations of the current implementation. | ||
|
|
||
| - **Simplified Schema Management**: CUE simplifies the process of writing and managing schemas, making it more accessible to users without extensive JSON Schema knowledge. This lowers the barrier to entry and enables more users to leverage Helm's validation capabilities effectively. | ||
|
|
||
| - **Enhanced Features**: CUE natively supports advanced features such as defaulting, flexible referencing of values, and other configuration management capabilities critical for complex Helm deployments. | ||
|
|
||
| ## Rationale | ||
|
|
||
| Integrating CUE as the primary validation tool in Helm is based on several key considerations: | ||
|
|
||
| 1. **Backward Compatibility**: CUE's nature as a superset of YAML, JSON, and JSON Schema ensures that all existing Helm configurations and JSON Schemas will continue to work without modifications. This seamless transition is a significant advantage for users. | ||
|
|
||
| 2. **Support for Modern Standards**: CUE's support for the latest JSON Schema specifications enables Helm users to benefit from modern features and bug fixes that are unavailable in the current implementation. | ||
|
|
||
| 3. **Ease of Use**: CUE's design simplifies schema writing and understanding, eliminating the need for deep expertise in JSON Schema. This democratizes the use of Helm and makes it accessible to a broader audience. | ||
|
|
||
| 4. **Advanced Features**: CUE's built-in capabilities, such as defaulting, value referencing, and flexible schema definitions, offer a more powerful validation system, enabling users to manage complex configurations with greater ease. | ||
|
|
||
| 5. **Consistency and Flexibility**: CUE's support for multiple data formats and its seamless integration with YAML and JSON make it an ideal choice for Helm's needs. It allows for more sophisticated configurations while ensuring consistency across different environments. | ||
|
|
||
| 6. **Community Demand**: The Helm community has expressed the need for a more capable validation system. CUE meets these needs without compromising the stability or compatibility of existing charts. | ||
|
|
||
| ## Specification | ||
|
|
||
| The implementation will involve the following key changes: | ||
|
|
||
| 1. **CUE Integration**: Helm will integrate CUE as a new validation engine for values files, embedding CUE's runtime into Helm and providing command-line options to enable CUE-based validation. | ||
|
|
||
| 2. **Dual Support for Validation Schemas**: While CUE will be introduced as the primary tool for new validations, existing JSON Schema support will remain, allowing users to choose the most suitable tool for their needs. | ||
|
|
||
| 3. **Configuration Files**: Support for CUE files (e.g., `values.cue`) will be introduced alongside `values.yaml`, allowing users to define configurations and schemas using CUE syntax. | ||
|
|
||
| 4. **Documentation and Tooling**: Comprehensive documentation and tooling support will be provided to assist users in understanding how to use CUE for validation, including examples and best practices. | ||
|
|
||
| 5. **Backward Compatibility Assurance**: As CUE is fully backward compatible with YAML, JSON, and JSON Schema, no changes will be required for existing charts. Full support will be maintained for all current validation schemas and formats. | ||
|
|
||
| 6. **Community Feedback Mechanism**: A feedback mechanism will be established to gather user experiences and address issues related to the adoption of CUE, ensuring a smooth transition. | ||
|
|
||
| ## Backward Compatibility | ||
|
|
||
| The introduction of CUE will not disrupt existing Helm configurations, as CUE is a superset of YAML, JSON, and JSON Schema. This means that all current `values.yaml` files, JSON Schema validations, and related configurations will continue to function as expected. Users can adopt CUE at their own pace, with the assurance that their current setups will remain fully operational. Moreover, CUE will extend support for newer JSON Schema specifications, ensuring continued relevance and support for evolving standards. | ||
|
|
||
| ## Security Implications | ||
|
|
||
| Integrating CUE introduces a new dependency into Helm, which must be carefully evaluated for security implications. Potential security considerations include: | ||
|
|
||
| - **Code Injection Risks**: Ensuring that CUE file processing does not introduce vulnerabilities, such as code injection or execution of arbitrary code. | ||
|
|
||
| - **Denial of Service**: Evaluating CUE processing performance to prevent potential denial-of-service attacks due to resource exhaustion. | ||
|
|
||
| - **Dependency Management**: Regularly updating the CUE library to incorporate security patches and updates. | ||
|
|
||
| A thorough security audit will be conducted to identify and mitigate any potential risks associated with CUE integration. | ||
|
|
||
| ## How to Teach This | ||
|
|
||
| To educate users about this change: | ||
|
|
||
| - **Documentation**: Update the Helm documentation to include comprehensive sections on using CUE, with examples, tutorials, and best practices. | ||
|
|
||
| - **Migration Guides**: Provide detailed migration guides for users transitioning from JSON Schema to CUE, highlighting benefits and considerations. | ||
|
|
||
| - **Workshops and Webinars**: Host online workshops and webinars to demonstrate the benefits of CUE and its effective use in Helm charts. | ||
|
|
||
| - **Community Engagement**: Engage with the community through forums, GitHub discussions, and social media to answer questions, gather feedback, and provide support during the transition. | ||
|
|
||
| - **Sample Charts**: Provide sample Helm charts utilizing CUE for validation, serving as references for users. | ||
|
|
||
| ## Reference Implementation | ||
|
|
||
| A proof-of-concept implementation has been developed and is available in [PR #13236](https://github.com/helm/helm/pull/13236). This implementation demonstrates the integration of CUE with Helm and includes basic examples of values file validation using CUE. | ||
|
|
||
| ## Rejected Ideas | ||
|
|
||
| - **Enhancing JSON Schema Support**: Extending the existing JSON Schema implementation to support newer drafts and features was considered but rejected due to the complexity and maintenance burden, as well as the limitations of JSON Schema in handling non-JSON formats like YAML. | ||
|
|
||
| - **Developing a Custom Validation System**: Creating a custom validation system specifically for Helm was considered but deemed unnecessary given CUE's existing capabilities and ecosystem support. | ||
|
|
||
| - **Using Alternative Tools**: Other configuration languages and validation tools were evaluated but did not offer the same level of compatibility, features, or community support as CUE. | ||
|
There was a problem hiding this comment. On this note, we have considered Pkl (https://pkl-lang.org/). I'm more familiar with CUE than Pkl. But Pkl would I believe provide both validation and rendering functionality. That said, there is certainly rationale to keep value processing/validation and manifest rendering separate concerns. Which does mean we could choose separate technologies for each. ie. for values processing/validation we could support (JSON schema, CUE, Pkl), and for manifest rendering we could support (Pkl, gotemplate) Overall, Helm I think needs to make a fairly strategic choice in how to approach these problem spaces. Personally, I would like to see better support in both. JSON schema (to the point of this HIP) is limited. And using unstructured text formatting ala. gotemplate for producing structured documents (Kubernetes object manifests) is at best an awkward solution. |
||
|
|
||
| ## Open Issues | ||
|
|
||
| - **Performance Impact**: The performance impact of integrating CUE into Helm is under evaluation. Benchmarks and performance testing will be conducted to ensure that CUE does not negatively affect Helm's performance. | ||
|
|
||
| - **User Adoption**: Encouraging the community to adopt CUE and transition away from JSON Schema may require additional efforts, including education, tooling support, and extended assistance during the transition period. | ||
|
|
||
| - **Integration with Existing Tools**: Assess how CUE integration will interact with existing Helm plugins and extensions to ensure compatibility and minimal disruption. | ||
|
|
||
| ## References | ||
|
|
||
| - [CUE Language Documentation](https://cuelang.org/docs/) | ||
| - [Helm Documentation](https://helm.sh/docs/) | ||
| - [JSON Schema Documentation](https://json-schema.org/) | ||
| - [Helm PR #13236](https://github.com/helm/helm/pull/13236) | ||
| - [Helm Issue #13260](https://github.com/helm/helm/issues/13260) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I would use e.g. superseding rather than replacing, and implementation rather than support. To me, "replacing" and "support" reads like JSON schema support will go away in lieu of using CUE. But I think the HIP means to utilizing CUE to implement JSON schema functionality.