[Discuss] Re-evaluate the argument for custom Chromium build

[tsullivan]

This issue is to list the pros and cons of having a custom build of Chromium shipped with Kibana to support PDF/PNG reporting.

Background

PhantomJS was the original headless browser used by Kibana to support screenshot reports since Kibana 5.0. That had its share of issues that were hard to reproduce and diagnose. Eventually, the Chromium project released headless support for its browser and Kibana switched to using it as our Reporting browser. As a reaction to Chromium supporting headless, the PhantomJS project deprecated itself. Using Chromium goes back to 6.3, and for the first 2 minors, chrome-remote-interface drove the integration. In 6.5, we switched to Puppeteer.

In first switching away from Phantom, we had the option to use a "stock" Chromium installation from the node module. Instead, we chose the alternative which is to supply our own Chromium installation. The Kibana team created build scripts that compile Chromium from source, using our own arguments files.

There is one main reason why we chose to ship a custom binary rather than use the default Chromium: in Linux, the default Chromium requires x11 which is not normally installed on a Linux server, and has a huge impact on how a server is provisioned in a self-deploy setup. (Note: even with the custom binary, Kibana Reporting imposes a few system requirements for the Linux server, such as fontconfig and nss )

Reasons to re-evaluate

Kibana Reporting has worked the way it has since version 6.3. Versions 6.3 and 6.4 used chrome-remote-interface rather than Puppeteer, but the custom build process. was the same. Looking at the question now, there are unanswered questions:

1. Why is the size of the custom binary shipped with the Kibana distro larger than Linux's default binary for Chromium?
2. How would using the Puppeteer defaults affect the size of the Kibana Docker image?
3. How would using Puppeteer defaults affect general stability of Kibana Reporting?

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[tsullivan]

ping @tylersmalley @kobelb @stacey-gammon

[kobelb]

Why is the size of the custom binary shipped with the Kibana distro larger than Linux's default binary for Chromium?

I'm assuming it's because we statically link everything we can when building the Kibana distro version so that as few as possible dependencies need to be installed on the server. The default binary is dynamically linked, resulting in a smaller binary but more dependencies on system libraries that we would have to instruct customers how to install.

How much would the Reporting team be freed to work on other things?

It's my understanding that we haven't automated the building of Chromium, yet. If we were to do so, wouldn't this free the Reporting team up to work on other things?

[tsullivan]

It's my understanding that we haven't automated the building of Chromium, yet. If we were to do so, wouldn't this free the Reporting team up to work on other things?

Yes. I'm glad you brought it up, but I want to explore that idea separately. If there isn't enough of a reason to continue our custom build, then we don't need to automate it.

(I edited the topic and removed the last bullet point)

[tsullivan]

What is driving this question: we want to see if updating Chromium will help with certain bugs in Reporting. Creating a new build just to see if bugs are fixed is a big ask.

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[tsullivan]

@kobelb I understand the argument for continuing the custom Linux build, and it is good to statically link everything we can.

For Windows and Mac, I think the arguments don't apply. There really aren't extra system dependencies that an admin would need to install for Windows and Mac, correct?

[kobelb]

@tsullivan to be honest, I don't recall.

[tsullivan]

The plan for the next update of Puppeteer in 7.14 will be to ship the "stock" build of Chromium for Mac and Windows. Linux will still require a custom build to lower the barrier of getting started for server OSes.

We'll try to work in a way that developers are not blocked on waiting for the build process while working on change to Kibana that use Puppeteer: we should develop and write tests using the stock build no matter what OS the developer works with.

[tsullivan]

Closing for #90496

[tsullivan]

An observation has come up while working through the next Chromium upgrade. There are a few reasons to continue custom building our own headless Chromium:

1. No builds are easily available for the "headless_shell" application. This application is generated from a build target in Chromium source code that seems almost like a proof-of-concept. Google does not provide any downloads to this application, and it seems that it is not even built regularly nor automatically tested for regressions. (This is a reason to stop using Chromium headless_shell completely)
2. If we use the "stock" builds of Chromium (that is, "full" Chromium), we could end up bundling a LOT of files, especially for Mac. See chore(NA): assure puppeteer_skip_chromium_download is applied across every yarn install situation #88346
3. Our custom build arguments disable kerberos, which seems to have security implications:

   • kibana/x-pack/build_chromium/linux/args.gn

     Line 8 in be1eb4c

     use_kerberos = false

   • kibana/x-pack/build_chromium/windows/args.gn

     Line 22 in be1eb4c

     use_kerberos = false

   • kibana/x-pack/build_chromium/darwin/args.gn

     Line 23 in be1eb4c

     use_kerberos = false