Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci(ftr): notify owners in the slack message when failures #205260

Merged
merged 7 commits into from
Jan 7, 2025
Merged

ci(ftr): notify owners in the slack message when failures #205260

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d470111
ci(ftr): notify owners in the slack message when failures
v1v Dec 30, 2024
4f0069c
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine Dec 30, 2024
d30ec5f
use env variable to ping the slack team
v1v Dec 30, 2024
2e28915
refactor
v1v Dec 30, 2024
d2cc67c
fix: unbound variable
v1v Dec 30, 2024
aa29b60
simplify
v1v Jan 7, 2025
ba3d202
Merge branch 'main' into feature/ping-errors-eams
v1v Jan 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .buildkite/pipeline-utils/test-failures/annotate.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,3 +188,16 @@ export const annotateTestFailures = async () => {
);
}
};

export const pingTeam = async () => {
if (process.env.ELASTIC_SLACK_NOTIFICATIONS_ENABLED === 'true') {
const reportSlackDir = 'target/report-slack-team';
const team: string =
recursiveReadDir(reportSlackDir).find((file) => file.endsWith('.slack')) || '';

buildkite.setMetadata(
'slack:ping_team:body',
`@${team}, can you please take a look at the test failures?`
);
}
};
1 change: 1 addition & 0 deletions .buildkite/scripts/common/env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ export CI=true
KIBANA_DIR=$(pwd)
export KIBANA_DIR
export XPACK_DIR="$KIBANA_DIR/x-pack"
export REPORT_SLACK_TEAM="target/report-slack-team"

export CACHE_DIR="$HOME/.kibana"
export ES_CACHE_DIR="$HOME/.es-snapshot-cache"
Expand Down
23 changes: 23 additions & 0 deletions .buildkite/scripts/lifecycle/ping_team_ftrs.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the "Elastic License
* 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
* Public License v 1"; you may not use this file except in compliance with, at
* your election, the "Elastic License 2.0", the "GNU Affero General Public
* License v3.0 only", or the "Server Side Public License, v 1".
*/

import { TestFailures } from '#pipeline-utils';

(async () => {
try {
await TestFailures.pingTeam();
} catch (ex) {
console.error('Ping team for the test failures error', ex.message);
if (ex.response) {
console.error('HTTP Error Response Status', ex.response.status);
console.error('HTTP Error Response Body', ex.response.data);
}
process.exit(1);
}
})();
4 changes: 4 additions & 0 deletions .buildkite/scripts/lifecycle/post_command.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,4 +52,8 @@ if [[ "$IS_TEST_EXECUTION_STEP" == "true" ]]; then
buildkite-agent artifact upload 'target/test_failures/**/*'
ts-node .buildkite/scripts/lifecycle/annotate_test_failures.ts
fi

if [[ -d "${REPORT_SLACK_TEAM}" ]]; then
ts-node .buildkite/scripts/lifecycle/ping_team_ftrs.ts
fi
fi
13 changes: 10 additions & 3 deletions .buildkite/scripts/steps/functional/apm_cypress.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,15 @@ fi

echo "--- APM Cypress Tests"

cd "$XPACK_DIR"
pushd "$XPACK_DIR"

node solutions/observability/plugins/apm/scripts/test/e2e.js \
if ! node solutions/observability/plugins/apm/scripts/test/e2e.js \
--kibana-install-dir "$KIBANA_BUILD_LOCATION" \
$CYPRESS_ARGS
$CYPRESS_ARGS ; then

# Report the error to the team using the subfolder matching the Slack Team
popd
mkdir "$REPORT_SLACK_TEAM"
touch "$REPORT_SLACK_TEAM"/obs-ux-infra_services-team.slack
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you help me understand the changes here vs passing a string PING_SLACK_TEAM='obs-ux-infra_services-team?

I'm not opposed to files but it seems to add some additional complexity.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure thing.

I wanted to make pingTeam agnostic to the team name. But read the files under target/report-slack-team, and those files match the name of the slack team to be notified if any errors.

Regarding your question about PING_SLACK_TEAM='obs-ux-infra_services-team', I'm not sure how the post_command can read that value and whether I could use something like ts-node .buildkite/scripts/lifecycle/ping_team_ftrs.ts "$PING_SLACK_TEAM" in here

Maybe I could use the buildkite metadata set command to notify the Slack team to reduce the additional complexity with files under target/report-slack-team.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I've found a better approach:

Pretty much, if the env variable PING_SLACK_TEAM exists and the step fails, then it will notify the slack team.

exit 1
fi
12 changes: 9 additions & 3 deletions .buildkite/scripts/steps/functional/profiling_cypress.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,13 @@ export JOB=kibana-profiling-cypress

echo "--- Profiling Cypress Tests"

cd "$XPACK_DIR"
pushd "$XPACK_DIR"

NODE_OPTIONS=--openssl-legacy-provider node solutions/observability/plugins/profiling/scripts/test/e2e.js \
--kibana-install-dir "$KIBANA_BUILD_LOCATION" \
if ! NODE_OPTIONS=--openssl-legacy-provider node solutions/observability/plugins/profiling/scripts/test/e2e.js \
--kibana-install-dir "$KIBANA_BUILD_LOCATION" ; then
# Report the error to the team using the subfolder matching the Slack Team
popd
mkdir "$REPORT_SLACK_TEAM"
touch "$REPORT_SLACK_TEAM"/obs-ux-infra_services-team.slack
exit 1
fi