From 0cfa64a4a4d0d7380b9feec9ceb8d598c131e176 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Steven!=20Ragnaro=CC=88k?= Date: Fri, 23 Jun 2023 08:07:12 -0700 Subject: [PATCH] Add script to mark build agents offline for shutdown. Because of the way import package jobs are triggered by package build jobs the normal Jenkins shutdown procedures will block triggering the import package jobs required to complete a packaging job, preventing jobs from finishing (until they fail due to timeout). The added script can be used to mark build agents offline in order to drain running packaging jobs while allowing the repo host to continue importing packages to let jobs finish. --- doc/ongoing_operations.rst | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/doc/ongoing_operations.rst b/doc/ongoing_operations.rst index 5ef294b97..61320905c 100644 --- a/doc/ongoing_operations.rst +++ b/doc/ongoing_operations.rst @@ -77,6 +77,31 @@ No arguments means import from the default location(s). We do this for things like new releases of the core ROS python tools. +Stop package building jobs without blocking package imports +----------------------------------------------------------- + +Package (source and binary) build jobs for deb rely on a subordinate import package job to run on the repo host before the job can complete. +If the build farm needs to be brought into shutdown the standard shutdown procedure won't work because import package jobs will not be allowed to start. +In order to do a clean shutdown the build agents must be brought offline while allowing the repo host to continue running. For build farms with fixed agent pools this can be accomplished using the following groovy script + +.. code-block:: groovy + + import hudson.model.labels.LabelAtom + import hudson.slaves.OfflineCause.UserCause + + cause = new UserCause(Jenkins.get().getUser("nuclearsandwich"), "Allow package imports to catch up") + + label1 = new LabelAtom("buildagent") + label2 = new LabelAtom("buildagent_arm64") + + for (comp in Jenkins.get().getComputers()) { + if (comp.getAssignedLabels().contains(label1) || comp.getAssignedLabels().contains(label2)) { + comp.setTemporarilyOffline(true, cause) + } + } + +Once all source and binary packaging jobs have completed the farm can be put into the standard shutdown mode to complete the process. + Perform action on a set of jobs -------------------------------