feat: Introduce ecosystem tests for popular plugins

[JoshuaKGoldberg]

Summary

Adding an CI job to the eslint/eslint repo that checks changes against a small selection of third-party plugins.

Related Issues

eslint/eslint#19139

[voxpelli]

That none of the eslint-community plugins are included feels a bit odd? Since it's somewhat operated under the same umbrella as ESLint itself?

At least one of eg eslint-plugin-n, eslint-plugin-promise or eslint-plugin-security would be good to include?

I also note that the selection criteria is similar to those outlined in the suggested eslint-community governance, and which have gotten feedback / objections / concerns there: eslint-community/governance#1 (comment)

[nzakas]

My apologies, I started a review last week, left a bunch of comments, and then forgot to hit "Submit Review".

I think this outlines a good infrastructure, just looking for some clarifications.

[aladdin-add]

Looks great to me! This is an excellent starting point. We can proceed to run it and see how it performs. 🚀🚀🚀

[nzakas]

I just realized we're not really defining what "breakage" is anywhere. Are we just running npm test on each package with the local ESLint changes?

[JoshuaKGoldberg]

I wasn't planning on suggesting including source code from plugins in this proposal. To start I was thinking of just verifying that npm lint doesn't error out - regardless of rule reports. Meanwhile, the instigating issues mentioned in the RFC were all runtime crashes that would be caught by it.

Relying on tests makes me a little nervous. It'd be a lot slower -especially if the plugins have build steps- and we'd need to make sure none of them have tests that rely on specifics of rule reports.

My vote would be to just use the plugins as end-users until we have a breakage that would have been caught by source code level checks.

[nzakas]

To start I was thinking of just verifying that npm lint doesn't error out - regardless of rule reports.

Can you explain how that would work? Are you running npm lint in the ESLint repo? If so, does that mean we can only test plugins that ESLint itself uses?

[JoshuaKGoldberg]

That's what I intend by this:

rfcs/designs/2024-repo-ecosystem-plugin-tests/README.md

Lines 30 to 35 in 8fd5da4

	1. Create a new directory containing:
	- `package.json`
	- `eslint.config.js` with the closest equivalent to an _"enable all rules"_ preset from the plugin
	- A small set of files known to be parsed and not cause lint reports with the plugin
	2. Run a lint command (i.e. `npx eslint .`) in that directory
	3. Assert that the lint command passed with 0 lint reports.

Is that not clear? Is there a different phrasing you'd suggest?

[nzakas]

Suggested change

	Each added plugin brings adds a risk of breakage, so plugins will only be added after filing a new issue and gaining team consensus.
	Each added plugin adds a risk of breakage, so plugins will only be added after filing a new issue and gaining team consensus.

[nzakas]

Suggested change

	A CI job will be added to the `eslint/eslint` issue, but will not immediately be a part of `main` branch or PR branch builds.
	A CI job will be added to the `eslint/eslint` repo, but will not immediately be a part of `main` branch or PR branch builds.

[nzakas]

I'm a bit confused by this process. Is the intent that we end up with all three at the end? Or just number 3?

And I'm unclear on the value of number 2. Presumably, this is meant to run only after a PR is merged, but how will we be notified if the job fails?

[bmish]

Should we utilize eslint-remote-tester for testing against other repositories? If not, can we mention it under "alternatives considered" or at least mention it as prior art?

I know a number of popular plugins use it like:

• https://github.com/jsx-eslint/eslint-plugin-react
• https://github.com/testing-library/eslint-plugin-jest-dom
• https://github.com/eslint-community/eslint-plugin-eslint-plugin
• https://github.com/ember-cli/eslint-plugin-ember

@AriPerkkio

[AriPerkkio]

Thanks for ping @bmish. Here's a quick summary of what eslint-remote-tester does:

• In eslint-remote-tester.config.js/ts you'll configure ESLint config (identical to ESLint's overrideConfig) and repositories that should be cloned on the file system.
• When run, it spins up x-amount of node:worker_threads that each handle a single repository parallel.
• A single thread clones the assigned repository and run ESLint using Node API against it. Results are reported back to main thread.
• Main thread constructs markdown or plain-text files and writes them to file system or outputs to CLI

There are some examples of bugs it can find automatically listed here: AriPerkkio/eslint-remote-tester#3. I used to run it against most popular community plugins for a while couple of years ago.

I know a number of popular plugins use it like:

Also worth to mention: eslint-plugin-unicorn, eslint-plugin-jest, eslint-plugin-testing-library and eslint-plugin-vitest.

For the other ecosystem CI setups I would recommend to check how Vite and Vitest does this. There has also been some thoughts about making a generic ecosystem-ci that all Javascript ecosystem packages could utilize. It would not be strictly tied to Vite-ecosystem like the current ones are.

• https://github.com/vitejs/vite-ecosystem-ci
• https://github.com/vitest-dev/vitest-ecosystem-ci

[bmish]

We could mention that a plugin using an internal/private ESLint API is an example of why the plugin would need to be updated. We don't want to stop ESLint from being able to modify its own internal/private APIs.

Relevant comment about this: eslint/eslint#19139 (comment)