SSHLauncher tries to launch Windows agent for more than 13 hours #359
Comments
|
Thanks for the report! I am not sure if this is a problem of
|
|
@olivergondza thanks for getting back to me so quickly
No, node is N/A (highlighted in red) and a job run is cancelled after timeout, but node is never terminated. I have a reason why I believe it's an Also I'd like to stress out this happen quite "randomly" (meaning I can't related it to the cause) as sometimes there is 20 nodes provisioned correctly in row, sometimes it's couple of this in row. On average, the "launch" phase is done within 14 seconds.
I can't disclose details, but we have quite a volume of linux-based runs and I can't see it there.
I'll try that next time I see the problem and get back to you here. In short - ATM no, but I'll be watching it. |
|
@olivergondza, me and @pematous have encountered this issue recently. We decided to workaround it by not running Jenkins Agent on the Windows machines. Instead, we spawn Windows machines using openstackMachine step, which works reliably, and then connect to the machine by its IP using SSH/Ansible, as needed. It is less convenient than having the Jenkins agent available, but it works! |
Jenkins and plugins versions report
Environment
What Operating System are you using (both controller, and any agents involved in the problem)?
agent - Windows Server 2019 (CYGWIN_NT-10.0-17763)
Reproduction steps
Provision Windows nodes again and again until you hit the problem.
Expected Results
One of:
SSHLaunchershould automatically re-launchJCloudsLauncheron Windows (re-try mechanism)Actual Results
Sometimes ("randomly") provisioning fails and node is stuck in state "launch", in cloud stats I can see 13 hr and counting (then I terminated node manually), however there is no issue with the agent. I can access node from my computer with
sshand there are no logs in/remoting(the directory does not exist, but remoting.jar is there). Using "relaunch agent" button does not help.Here is edited log:
SSHLauncher{host='xx.xx.xx.xx', port=22, credentialsId='xxxxxx', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=600, maxNumRetries=5, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true} [01/26/23 20:59:31] [SSH] Opening SSH connection to xx.xx.xx.xx:22. [01/26/23 20:59:39] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection. [01/26/23 20:59:39] [SSH] Authentication successful. [01/26/23 20:59:39] [SSH] The remote user's environment is: .. Checking Java version in the PATH openjdk version "11.0.18" 2023-01-17 OpenJDK Runtime Environment GraalVM CE 22.3.1 (build 11.0.18+10-jvmci-22.3-b13) OpenJDK 64-Bit Server VM GraalVM CE 22.3.1 (build 11.0.18+10-jvmci-22.3-b13, mixed mode, sharing) [01/26/23 20:59:42] [SSH] Checking java version of /<<some-path>>/jdk/bin/java Couldn't figure out the Java version of /<<some-path>>/jdk/bin/java bash: /<<some-path>>/jdk/bin/java: No such file or directory [01/26/23 20:59:42] [SSH] Checking java version of java [01/26/23 20:59:43] [SSH] java -version returned 11.0.18. [01/26/23 20:59:43] [SSH] Starting sftp client. [01/26/23 20:59:44] [SSH] Copying latest remoting.jar... [01/26/23 20:59:45] [SSH] Copied 1,339,992 bytes. Expanded the channel window size to 4MB [01/26/23 20:59:45] [SSH] Starting agent process: cd "<<some_hudson_workspace_path>>" && java -jar remoting.jar -workDir <<some_hudson_workspace_path>> -jar-cache <<some_hudson_workspace_path>>/jarCache led 26, 2023 8:59:46 ODP. org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using <<some_hudson_workspace_path>>\remoting as a remoting work directory led 26, 2023 8:59:46 ODP. org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to <<some_hudson_workspace_path>>\remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 3044.vb_940a_a_e4f72e Launcher: JCloudsLauncher Communication Protocol: Standard in/out This is a Windows agentAnything else?
No response
The text was updated successfully, but these errors were encountered: