[NL-to-ESQL] update internal documentation #205853
Merged
+1,307
−194
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/ci |
pgayvallet
added
release_note:skip
|
Pinging @elastic/appex-ai-infra (Team:AI Infra) |
pgayvallet
commented
Jan 8, 2025
There was a problem hiding this comment.
- added new (and missing) functions
- added one-sentence description to most functions (except type conversion)
- added more examples
- added comments explaining reasoning on some examples
legrego
approved these changes
Jan 8, 2025
| @@ -13,6 +13,11 @@ const suggestions: Suggestion[] = [ | |||
| return ['BUCKET']; | |||
| } | |||
| }, | |||
| (keywords) => { | |||
| if (keywords.includes('TO_DATETIME')) { | |||
There was a problem hiding this comment.
I always suspected that AI was just a bunch of if statements 😄
There was a problem hiding this comment.
Sometimes we have to cheat a bit!
x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-byte_length.txt
Outdated
Show resolved
Hide resolved
x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-categorize.txt
Outdated
Show resolved
Hide resolved
x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-hash.txt
Outdated
Show resolved
Hide resolved
x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-hash.txt
Outdated
Show resolved
Hide resolved
x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-hash.txt
Outdated
Show resolved
Hide resolved
x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-limit.txt
Outdated
Show resolved
Hide resolved
x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-st_xmax.txt
Outdated
Show resolved
Hide resolved
x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-st_ymin.txt
Outdated
Show resolved
Hide resolved
|
/ci |
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]
History
|
|
Starting backport for target branches: 8.x |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jan 9, 2025
## Summary Fix elastic#205606 - Re-generate the internal ES|QL documentation using the generation script (+ human review) - Add more scenario to the NL-to-ESQL evaluation suite - Some prompt engineering - improving the system instructions / functions summary - add more examples to the summary - adapt a few opinionated examples for some specific functions ## Evaluation - average based on 4 runs for each model/branch tuple - the new tests were locally added to main to run against the same suite and properly evaluate the difference | Model | before (main) | after (PR) | delta | | ------------- | ------------- | ------------- | ------------- | | GPT-4o | 90.9 | 97.74 | + 6.84 | | Claude 3.5 Sonnet v2 | 88.58 | 96.49 | +7.91 | | Gemini 1.5-pro-002 | 88.17 | 94.19 | +6.02 | Overall, the prompt engineering somewhat significantly improved the generation efficiency. (cherry picked from commit 5b96912)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Jan 9, 2025
# Backport This will backport the following commits from `main` to `8.x`: - [[NL-to-ESQL] update internal documentation (#205853)](#205853) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Pierre Gayvallet","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-01-09T07:04:29Z","message":"[NL-to-ESQL] update internal documentation (#205853)\n\n## Summary\r\n\r\nFix https://github.com/elastic/kibana/issues/205606\r\n\r\n- Re-generate the internal ES|QL documentation using the generation\r\nscript (+ human review)\r\n- Add more scenario to the NL-to-ESQL evaluation suite \r\n- Some prompt engineering\r\n - improving the system instructions / functions summary\r\n - add more examples to the summary\r\n - adapt a few opinionated examples for some specific functions \r\n\r\n## Evaluation\r\n\r\n- average based on 4 runs for each model/branch tuple\r\n- the new tests were locally added to main to run against the same suite\r\nand properly evaluate the difference\r\n\r\n| Model | before (main) | after (PR) | delta |\r\n| ------------- | ------------- | ------------- | ------------- |\r\n| GPT-4o | 90.9 | 97.74 | + 6.84 |\r\n| Claude 3.5 Sonnet v2 | 88.58 | 96.49 | +7.91 |\r\n| Gemini 1.5-pro-002 | 88.17 | 94.19 | +6.02 |\r\n\r\nOverall, the prompt engineering somewhat significantly improved the\r\ngeneration efficiency.","sha":"5b9691278165417d4cc12853f58de728aaeff011","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:version","Team:AI Infra","v8.18.0"],"title":"[NL-to-ESQL] update internal documentation","number":205853,"url":"https://github.com/elastic/kibana/pull/205853","mergeCommit":{"message":"[NL-to-ESQL] update internal documentation (#205853)\n\n## Summary\r\n\r\nFix https://github.com/elastic/kibana/issues/205606\r\n\r\n- Re-generate the internal ES|QL documentation using the generation\r\nscript (+ human review)\r\n- Add more scenario to the NL-to-ESQL evaluation suite \r\n- Some prompt engineering\r\n - improving the system instructions / functions summary\r\n - add more examples to the summary\r\n - adapt a few opinionated examples for some specific functions \r\n\r\n## Evaluation\r\n\r\n- average based on 4 runs for each model/branch tuple\r\n- the new tests were locally added to main to run against the same suite\r\nand properly evaluate the difference\r\n\r\n| Model | before (main) | after (PR) | delta |\r\n| ------------- | ------------- | ------------- | ------------- |\r\n| GPT-4o | 90.9 | 97.74 | + 6.84 |\r\n| Claude 3.5 Sonnet v2 | 88.58 | 96.49 | +7.91 |\r\n| Gemini 1.5-pro-002 | 88.17 | 94.19 | +6.02 |\r\n\r\nOverall, the prompt engineering somewhat significantly improved the\r\ngeneration efficiency.","sha":"5b9691278165417d4cc12853f58de728aaeff011"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/205853","number":205853,"mergeCommit":{"message":"[NL-to-ESQL] update internal documentation (#205853)\n\n## Summary\r\n\r\nFix https://github.com/elastic/kibana/issues/205606\r\n\r\n- Re-generate the internal ES|QL documentation using the generation\r\nscript (+ human review)\r\n- Add more scenario to the NL-to-ESQL evaluation suite \r\n- Some prompt engineering\r\n - improving the system instructions / functions summary\r\n - add more examples to the summary\r\n - adapt a few opinionated examples for some specific functions \r\n\r\n## Evaluation\r\n\r\n- average based on 4 runs for each model/branch tuple\r\n- the new tests were locally added to main to run against the same suite\r\nand properly evaluate the difference\r\n\r\n| Model | before (main) | after (PR) | delta |\r\n| ------------- | ------------- | ------------- | ------------- |\r\n| GPT-4o | 90.9 | 97.74 | + 6.84 |\r\n| Claude 3.5 Sonnet v2 | 88.58 | 96.49 | +7.91 |\r\n| Gemini 1.5-pro-002 | 88.17 | 94.19 | +6.02 |\r\n\r\nOverall, the prompt engineering somewhat significantly improved the\r\ngeneration efficiency.","sha":"5b9691278165417d4cc12853f58de728aaeff011"}},{"branch":"8.x","label":"v8.18.0","branchLabelMappingKey":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Pierre Gayvallet <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fix #205606
Evaluation
Overall, the prompt engineering somewhat significantly improved the generation efficiency.