#171 - Package Jenkinsfile Runner as a single JAR file with bundled dependencies + Include SSHD Module to the bundle to fix warning by the Git Server Plugin (Pipeline: Global CPS Library Dependency) #184

jainsahab · 2019-10-08T07:03:08Z

Fix for #171

jainsahab · 2019-10-08T07:05:08Z

bootstrap/src/main/java/io/jenkins/jenkinsfile/runner/bootstrap/Bootstrap.java

-    /**
-     * This system property is set by the bootstrap script created by appassembler Maven plugin
-     * to point to a local Maven repository.
-     */
-    public final File appRepo = new File(System.getProperty("app.repo"));
-


I believe in case of fat jar, the app.repo property will not be provided since repo is part of the jar itself. We've made it on-demand initialization with the help of a function below which will be executed only when collecting jars from repo is required.

jainsahab · 2019-10-08T07:06:11Z

bootstrap/src/main/java/io/jenkins/jenkinsfile/runner/bootstrap/Bootstrap.java

+        if (hasClass(appClassName)) {
+            Class<?> c = Class.forName(appClassName);
+            return ((IApp) c.newInstance()).run(this);
+        }


We first check if a particular class is present in class path(in case of fat jar), otherwise the old flow will be initiated. i.e. collecting jars and then load the class

oleg-nenashev · 2019-10-08T08:35:27Z

Thanks! It looks right. I need to test it locally before merging, I hope to do so by the end of the week

oleg-nenashev · 2019-10-09T06:12:06Z

setup/src/main/java/io/jenkins/jenkinsfile/runner/JenkinsfileRunnerLauncher.java

-                    .make();
-
-            Class<?> c = cl.loadClass("io.jenkins.jenkinsfile.runner.Runner");
+            Class<?> c = bootstrap.hasClass(RUNNER_CLASS_NAME)? Class.forName(RUNNER_CLASS_NAME) : getRunnerClassFromJar();


Suggested change

Class<?> c = bootstrap.hasClass(RUNNER_CLASS_NAME)? Class.forName(RUNNER_CLASS_NAME) : getRunnerClassFromJar();

Class<?> c = bootstrap.hasClass(RUNNER_CLASS_NAME) ? Class.forName(RUNNER_CLASS_NAME) : getRunnerClassFromJar();

You fetch your class twice here. IMHO it is fine for now, because I do not see a practical future for the old mode, except performance impact which needs to be measured. But this is a case where catching an exception would be better

Hi @oleg-nenashev , do you want me to change it?

oleg-nenashev · 2019-10-09T12:27:50Z

@jainsahab So I have merged it with master and tested locally with the Declarative Pipeline sample script

$demo/declarative-pipeline$ java -jar /c/Users/Oleg/Documents/jenkins/jfr/jenkinsfile-runner/app/target/jenkinsfile-runner-standalone.jar --plugins ../../vanilla-package/target/plugins/ --jenkins-war ../../vanilla-package/target/war/jenkins.war --runWorkspace work -f .

The Pipeline itself passes successfully. It is already great, thanks a lot!

Some discoveries...

There is analytics exception being thrown in this mode. "Filter instance is registered too early: jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter@". I doubt we want to setup analytics in this case at all, and I wonder whether we should just disable it inside JFR. CC @daniel-beck just in case

2019-10-09 12:08:28.525+0000 [id=32]    WARNING hudson.util.PluginServletFilter#addFilter: Filter instance is registered too early: jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter@2a
java.lang.Exception
        at hudson.util.PluginServletFilter.addFilter(PluginServletFilter.java:108)
        at jenkins.telemetry.impl.UserLanguages.setUpFilter(UserLanguages.java:102)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
        at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Git Server plugin startup, because the SSHD module classes cannot be located by the classloader in such mode. Indeed we should just disable the Git Server to optimize performance (comes from Pipeline: Global CPS Library, no use-case for JFR), but it looks like there is a packaging issue either. Some core classes seem to be loaded well from the internal bundles, but not modules. Maybe a classloader glitch

2019-10-09 12:08:30.498+0000 [id=27]    WARNING hudson.ExtensionFinder$Sezpoz#scout: Failed to scout org.jenkinsci.plugins.gitserver.ssh.SshCommandFactoryImpl
java.lang.ClassNotFoundException: org.jenkinsci.main.modules.sshd.SshCommandFactory
        at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1373)
        at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1326)
        at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1079)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
Caused: java.lang.NoClassDefFoundError: org/jenkinsci/main/modules/sshd/SshCommandFactory
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at jenkins.util.AntClassLoader.defineClassFromData(AntClassLoader.java:1139)
        at hudson.ClassicPluginStrategy$AntClassLoader2.defineClassFromData(ClassicPluginStrategy.java:707)
        at jenkins.util.AntClassLoader.getClassFromStream(AntClassLoader.java:1310)
        at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1363)
        at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1326)
        at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at jenkins.ClassLoaderReflectionToolkit.invoke(ClassLoaderReflectionToolkit.java:44)
        at jenkins.ClassLoaderReflectionToolkit._findClass(ClassLoaderReflectionToolkit.java:81)
        at hudson.PluginManager$UberClassLoader.findClass(PluginManager.java:2045)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
Caused: java.lang.InstantiationException
        at net.java.sezpoz.IndexItem.element(IndexItem.java:146)
        at hudson.ExtensionFinder$Sezpoz.scout(ExtensionFinder.java:736)
        at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:333)
        at hudson.ExtensionList.load(ExtensionList.java:381)
        at hudson.ExtensionList.ensureLoaded(ExtensionList.java:317)
        at hudson.ExtensionList.getComponents(ExtensionList.java:183)
        at jenkins.model.Jenkins$6.onInitMilestoneAttained(Jenkins.java:1126)
        at jenkins.InitReactorRunner$1.onAttained(InitReactorRunner.java:83)
        at org.jvnet.hudson.reactor.ReactorListener$Aggregator.lambda$onAttained$3(ReactorListener.java:102)
        at org.jvnet.hudson.reactor.ReactorListener$Aggregator.run(ReactorListener.java:109)
        at org.jvnet.hudson.reactor.ReactorListener$Aggregator.onAttained(ReactorListener.java:102)
        at org.jvnet.hudson.reactor.Reactor$1.run(Reactor.java:177)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

jainsahab · 2019-10-09T13:26:08Z

@jainsahab So I have merged it with master and tested locally with the Declarative Pipeline sample script

$demo/declarative-pipeline$ java -jar /c/Users/Oleg/Documents/jenkins/jfr/jenkinsfile-runner/app/target/jenkinsfile-runner-standalone.jar --plugins ../../vanilla-package/target/plugins/ --jenkins-war ../../vanilla-package/target/war/jenkins.war --runWorkspace work -f .

The Pipeline itself passes successfully. It is already great, thanks a lot!

Some discoveries...

1. There is analytics exception being thrown in this mode. "Filter instance is registered too early: jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter@". I doubt we want to setup analytics in this case at all, and I wonder whether we should just disable it inside JFR. CC @daniel-beck just in case

2019-10-09 12:08:28.525+0000 [id=32]    WARNING hudson.util.PluginServletFilter#addFilter: Filter instance is registered too early: jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter@2a
java.lang.Exception
        at hudson.util.PluginServletFilter.addFilter(PluginServletFilter.java:108)
        at jenkins.telemetry.impl.UserLanguages.setUpFilter(UserLanguages.java:102)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
        at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1095)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

1. Git Server plugin startup, because the SSHD module classes cannot be located by the classloader in such mode. Indeed we should just disable the Git Server to optimize performance (comes from Pipeline: Global CPS Library, no use-case for JFR), but it looks like there is a packaging issue either. Some core classes seem to be loaded well from the internal bundles, but not modules. Maybe a classloader glitch

2019-10-09 12:08:30.498+0000 [id=27]    WARNING hudson.ExtensionFinder$Sezpoz#scout: Failed to scout org.jenkinsci.plugins.gitserver.ssh.SshCommandFactoryImpl
java.lang.ClassNotFoundException: org.jenkinsci.main.modules.sshd.SshCommandFactory
        at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1373)
        at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1326)
        at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1079)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
Caused: java.lang.NoClassDefFoundError: org/jenkinsci/main/modules/sshd/SshCommandFactory
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at jenkins.util.AntClassLoader.defineClassFromData(AntClassLoader.java:1139)
        at hudson.ClassicPluginStrategy$AntClassLoader2.defineClassFromData(ClassicPluginStrategy.java:707)
        at jenkins.util.AntClassLoader.getClassFromStream(AntClassLoader.java:1310)
        at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1363)
        at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1326)
        at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at jenkins.ClassLoaderReflectionToolkit.invoke(ClassLoaderReflectionToolkit.java:44)
        at jenkins.ClassLoaderReflectionToolkit._findClass(ClassLoaderReflectionToolkit.java:81)
        at hudson.PluginManager$UberClassLoader.findClass(PluginManager.java:2045)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
Caused: java.lang.InstantiationException
        at net.java.sezpoz.IndexItem.element(IndexItem.java:146)
        at hudson.ExtensionFinder$Sezpoz.scout(ExtensionFinder.java:736)
        at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:333)
        at hudson.ExtensionList.load(ExtensionList.java:381)
        at hudson.ExtensionList.ensureLoaded(ExtensionList.java:317)
        at hudson.ExtensionList.getComponents(ExtensionList.java:183)
        at jenkins.model.Jenkins$6.onInitMilestoneAttained(Jenkins.java:1126)
        at jenkins.InitReactorRunner$1.onAttained(InitReactorRunner.java:83)
        at org.jvnet.hudson.reactor.ReactorListener$Aggregator.lambda$onAttained$3(ReactorListener.java:102)
        at org.jvnet.hudson.reactor.ReactorListener$Aggregator.run(ReactorListener.java:109)
        at org.jvnet.hudson.reactor.ReactorListener$Aggregator.onAttained(ReactorListener.java:102)
        at org.jvnet.hudson.reactor.Reactor$1.run(Reactor.java:177)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Hi @oleg-nenashev

For 1st point, Let me know what to do, if we want to disable it or not.

For 2nd Point, Could you please tell me where exactly we are specifying that dependency so that i can debug why it's not being packaged as part of Jar itself

oleg-nenashev · 2019-10-09T13:31:24Z

For 2nd Point, Could you please tell me where exactly we are specifying that dependency so that i can debug why it's not being packaged as part of Jar itself

Modules are part of the Jenkins Core, and some of them are included in appassembler under org/jenkins-ci/modules. The problem is that SSHD Module is not there due to whatever reason. We can easily add it an explicit dependency

setup/pom.xml

oleg-nenashev · 2019-10-09T14:48:49Z

👍 I will create a follow-up task to get the Git Server disabled and the module removed again

jainsahab · 2019-10-09T15:07:32Z

@oleg-nenashev let me know if there is anything else remaining to be done from my side.

oleg-nenashev · 2019-10-09T15:55:28Z

Retriggering CI

jainsahab · 2019-10-11T12:31:08Z

@oleg-nenashev Could you please help me understand if the build is failing because of the valid reasons?

oleg-nenashev · 2019-10-11T12:39:32Z

Does not look so. CI was pretty unstable over past months. Since we are in Beta, I tend to merge it since local testing works. But I will make another attempt

jainsahab · 2019-10-11T15:24:14Z

Ahh ok, Thanks for the confirmation. :)

oleg-nenashev · 2019-10-11T15:37:33Z

#194 for the CI fix

oleg-nenashev · 2019-10-11T17:05:18Z

@jainsahab could you please merge it with master?

jainsahab · 2019-10-11T17:58:35Z

You mean, merge it with fat_jar branch?

jainsahab · 2019-10-12T12:45:28Z

Thanks @oleg-nenashev for the quick CI fix. All checks have passed and it's green now.

oleg-nenashev · 2019-10-12T13:44:32Z

Thanks a lot for the patch! Will try to release the new Beta version this weekend

jainsahab added 2 commits October 8, 2019 12:23

pom configuration to generate fat jar

337efa1

preventing recursively loading of jars in case of fat jar

b0faa87

jainsahab commented Oct 8, 2019

View reviewed changes

oleg-nenashev added the hacktoberfest label Oct 8, 2019

oleg-nenashev added the enhancement New feature or request label Oct 8, 2019

oleg-nenashev self-assigned this Oct 8, 2019

oleg-nenashev self-requested a review October 8, 2019 08:35

oleg-nenashev reviewed Oct 9, 2019

View reviewed changes

including sshd module back

14af220

jainsahab commented Oct 9, 2019

View reviewed changes

setup/pom.xml Show resolved Hide resolved

oleg-nenashev changed the title ~~Fix for #171 - Fat jar for Jenkinsfile Runner CLI~~ #171 - Package Jenkinsfile Runner as a single JAR file with bundled dependencies + Include SSHD Module to the bundle to fix warning by the Git Server Plugin (Pipeline: Global CPS Library Dependency) Oct 9, 2019

oleg-nenashev mentioned this pull request Oct 9, 2019

Git Server Plugin should not start the SSH server inside JFR #192

Open

oleg-nenashev mentioned this pull request Oct 9, 2019

Jenkins Telemetry prints warnings on JFR startup #193

Closed

oleg-nenashev closed this Oct 9, 2019

oleg-nenashev reopened this Oct 9, 2019

oleg-nenashev closed this Oct 11, 2019

oleg-nenashev reopened this Oct 11, 2019

oleg-nenashev mentioned this pull request Oct 11, 2019

Jenkinsfile: Increase the build timeout to 60min #194

Merged

Merge remote-tracking branch 'upstream/master' into fat_jar

a24e0d5

oleg-nenashev merged commit 2af569f into jenkinsci:master Oct 12, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#171 - Package Jenkinsfile Runner as a single JAR file with bundled dependencies + Include SSHD Module to the bundle to fix warning by the Git Server Plugin (Pipeline: Global CPS Library Dependency) #184

#171 - Package Jenkinsfile Runner as a single JAR file with bundled dependencies + Include SSHD Module to the bundle to fix warning by the Git Server Plugin (Pipeline: Global CPS Library Dependency) #184

jainsahab commented Oct 8, 2019 •

edited

Loading

jainsahab Oct 8, 2019 •

edited

Loading

jainsahab Oct 8, 2019

oleg-nenashev commented Oct 8, 2019

oleg-nenashev Oct 9, 2019

jainsahab Oct 9, 2019

oleg-nenashev commented Oct 9, 2019

jainsahab commented Oct 9, 2019 •

edited

Loading

oleg-nenashev commented Oct 9, 2019

oleg-nenashev commented Oct 9, 2019

jainsahab commented Oct 9, 2019

oleg-nenashev commented Oct 9, 2019

jainsahab commented Oct 11, 2019

oleg-nenashev commented Oct 11, 2019

jainsahab commented Oct 11, 2019

oleg-nenashev commented Oct 11, 2019

oleg-nenashev commented Oct 11, 2019

jainsahab commented Oct 11, 2019

jainsahab commented Oct 12, 2019

oleg-nenashev commented Oct 12, 2019

	Class<?> c = bootstrap.hasClass(RUNNER_CLASS_NAME)? Class.forName(RUNNER_CLASS_NAME) : getRunnerClassFromJar();
	Class<?> c = bootstrap.hasClass(RUNNER_CLASS_NAME) ? Class.forName(RUNNER_CLASS_NAME) : getRunnerClassFromJar();

#171 - Package Jenkinsfile Runner as a single JAR file with bundled dependencies + Include SSHD Module to the bundle to fix warning by the Git Server Plugin (Pipeline: Global CPS Library Dependency) #184

#171 - Package Jenkinsfile Runner as a single JAR file with bundled dependencies + Include SSHD Module to the bundle to fix warning by the Git Server Plugin (Pipeline: Global CPS Library Dependency) #184

Conversation

jainsahab commented Oct 8, 2019 • edited Loading

jainsahab Oct 8, 2019 • edited Loading

Choose a reason for hiding this comment

jainsahab Oct 8, 2019

Choose a reason for hiding this comment

oleg-nenashev commented Oct 8, 2019

oleg-nenashev Oct 9, 2019

Choose a reason for hiding this comment

jainsahab Oct 9, 2019

Choose a reason for hiding this comment

oleg-nenashev commented Oct 9, 2019

jainsahab commented Oct 9, 2019 • edited Loading

oleg-nenashev commented Oct 9, 2019

oleg-nenashev commented Oct 9, 2019

jainsahab commented Oct 9, 2019

oleg-nenashev commented Oct 9, 2019

jainsahab commented Oct 11, 2019

oleg-nenashev commented Oct 11, 2019

jainsahab commented Oct 11, 2019

oleg-nenashev commented Oct 11, 2019

oleg-nenashev commented Oct 11, 2019

jainsahab commented Oct 11, 2019

jainsahab commented Oct 12, 2019

oleg-nenashev commented Oct 12, 2019

jainsahab commented Oct 8, 2019 •

edited

Loading

jainsahab Oct 8, 2019 •

edited

Loading

jainsahab commented Oct 9, 2019 •

edited

Loading