Extension Bisect should not re-enable most extensions throughout the process

[NSExceptional]

This is directly related to #111669 in which the OP describes how bisect is unable to find any "bad" extensions if there is more than one.

According to the docs, bisect re-enables the extensions it considers "fine". This is problematic because if you have more than one "bad" extension, it will never be able to narrow down the problem to a single extension, and it will incorrectly tell you the problem may be with VS Code.

This could be fixed by bisect not re-enabling any extensions it has removed from the problem set until one of the "bad" ones are found. This would allow you to keep finding problematic extensions by repeatedly running the bisect tool until none remain, instead of the tool only working at all when there is only one "bad" extension.

Detailed Design

Here is the example from the docs on how bisect works now:

• Bisect divides 24 extensions into two halves of 12 extensions each, and it disables all 12 extensions of the second half.
• In this sample, the 8th extension is the "bad" one, so it is the first half and not disabled. Things are still not working as we'd expect. Because there is still an issue, extension bisect repeats the process, dividing the first 12 extensions into two parts: 6 are enabled and 6 are disabled. All other extensions are also re-enabled.
  ...

That very last part is what is problematic. If we eliminate that step, the tool would be able to uncover any number of problematic extensions. I will give two examples of my own: one with one bad extension, and one with two. (I am not sure what bisect does when there are an odd number of extensions, so for my examples I will assume it will give the remainder to the upper half when partitioning an odd number)

• An enabled extension: …
• A disabled extension: X
• A "bad" extension: ☢️

Example: 12 extensions with 1 bad extension

1. The bad extension is no. 3; bisect disables the upper half of all 12

—	—	☢️	—	—	—	X	X	X	X	X	X

2. There is still a problem So bisect disables the upper half of the lower 6

—	—	☢️	X	X	X	X	X	X	X	X	X

3. There is still a problem So bisect disables extensions 2 and 3

—	X	X (☢️)	X	X	X	X	X	X	X	X	X

4. Problem gone, so bisect disables the non-problematic extension (no. 1), and then re-enables the lower half of the previously disabled set (so, just the 2nd one)

X	X	☢️	X	X	X	X	X	X	X	X	X

5. Bisect can correctly identify no. 3 as the problematic extension

Example: 12 extensions with 2 bad extensions

1. The bad extensions are no. 4 and 12; bisect disables the upper half of all 12

—	—	—	☢️	—	—	X	X	X	X	X	X (☢️)

2. There is still a problem, so bisect disables the upper half of the lower 6

—	—	—	X (☢️)	X	X	X	X	X	X	X	X (☢️)

2. There is still a problem So bisect disables the lower half of the lower 6 (the first 3). Now we are working with extensions 4-6. Bisect re-enables the lower half of this group, which is just no. 4

X	X	X	☢️	X	X	X	X	X	X	X	X (☢️)

3. A problem is found with only one extension enabled, so this round of bisecting completes with a "bad" extension identified. However the user still experiences a problem so they start a new bisect. Let us assume the user has removed the first bad extension so there are now 11 extensions left, and the 11th one is the problematic one. Bisect disables the upper half of all 11 extensions (6-11)

—	—	—	—	—	X	X	X	X	X	X (☢️)

4. The user no longer experiences the problem, so bisect disables the un-problematic half entirely and re-enables the lower half of the remaining extensions. This repeats until the 10th extension is the only one still enabled, still no problem.

X	X	X	X	X	—	—	—	X	X	X (☢️)

X	X	X	X	X	X	X	X	—	X	X (☢️)

X	X	X	X	X	X	X	X	X	—	X (☢️)

5. I assume it would finally enable the last extension to confirm the user experiences a problem at all. And the user does experience it with only the 11th extension enabled, so bisect identifies it as the problem to the user.

X	X	X	X	X	X	X	X	X	X	☢️

---

Another scenario worth noting is that in the worst case, where the first and second extensions are the bad ones, the user will choose "I can reproduce" every single time. The current implementation takes this opportunity to tell the user that it could not detect a bad extension and that the problem might be with VS Code itself. In cases like this where we cannot be 100% sure that a singular extension is the problem, my new approach should probably include that same disclaimer when it identifies a potentially problematic extension.

While imperfect, this is still more useful than not identifying any extensions at all. I can think of several improvements that can be made as well; for example we could modify the algorithm to test both halves of a set of extensions for problems when a problem is encountered on one half and alert the user. This could save time by letting the user know that both halves of some set of extensions (which may be all of them if it is the first try) reproduce the problem, and let them choose to continue bisecting or not.

[NSExceptional]

An example of a use case for this: when I open VS Code in my home folder, my CPU usage spikes dramatically for several minutes. I quickly found out that something was searching the entire contents of my home folder, which is rather large, so it seemed to go on indefinitely.

Looking at the logs, there were about 3 of my ~30 extensions doing recursive "find in workspace" for some file **/foo.ext. With those extensions disabled, my CPU usage did not spike any longer. (This particular issue isn't a major one per se, but I definitely want those 3 extensions disabled in my home folder workspace, and I wanted bisect to help me find them)

Bisect was totally useless because it couldn't narrow down the problem to a single extension. A version of bisect that gives you a best guess (or allows you to choose whether you want a certain answer or a best guess) would be way more useful

[gjsjohnmurray]

@jrieken I think you created Extension Bisect, so maybe have an opinion about this.