Loop in COMMAND ocrGetText(<object>)

[P-Courteille]

Environment (please complete the following information):

• Node.js version: 22.11.0
• NPM version: 10.9.1
• Platform name and version: Android 11 (testing on native android app)
• WebdriverIO version: 9.4.1
• @wdio/visual-service version: 6.2.2
• @wdio/mocha-framework version: 9.2.8
• @wdio/local-runner version: 9.4.1
• @wdio/globals version: 9.4.1
• @wdio/cli version: 9.4.1
• Appium version: 2.11.5
• tsx version: 4.19.2

Config of WebdriverIO + @wdio/visual-service

services: [
        // your other services
        [
            "ocr",
            {
                contrast: 0.25,
                imagesFolder: "./onboard/output"
            },
        ],
        [
            "visual",
            {
                baselineFolder: path.join(process.cwd(), "onboard", "baseline"),
                formatImageName: "{tag}-{width}x{height}-{dpr}",
                screenshotPath: path.join(process.cwd(), "onboard", "output", "visual"),
                savePerInstance: true,
                clearRuntimeFolder: true,
            },
        ],
    ],

Describe the bug
After my typescript migration, it no longer work, it enter in a COMMAND ocrGetText() loop.

 await browser.ocrGetText({ haystack: await browser.$('myxpath') })

Expected behavior
The text is detected or if not, the following message is return "Error: An error happened when parsing the getNodeOcrData, see: Error: No text was found for the OCR, please verify the stored image."

Additional context
I'm using typescript and my tsconfig contain:

"types": [
            "node",
            "@wdio/globals/types", 
            "@wdio/mocha-framework",
            "@wdio/appium-service",
            "@wdio/ocr-service",
            "@wdio/visual-service",
            "i18next"
]

[wswebcreation]

Hi @P-Courteille

Can you provide a simple, reproducible example?

[P-Courteille]

Hi @wswebcreation,

I recreate an new project to have an example and didn't reproduce the issue.

I found somethins suspicious:
In my original project, in node_modules@wdio\types\package.json the version is 8.40.6 instead of 9.2.2.

I don't know why and even if I delete my node_modules folder and do again an npm install, the result is the same.

Do you have any idea of what can produce the behavior ?

---

I manually did a 'npm install @wdio/types' and now it's up-to-date.
I will try to fix the new issue I've and retest.

[P-Courteille]

@wswebcreation

The update of @wdio/types doesn't fix my issue.

I have a lead in this discord conversation about multiremote capabilties:
https://discord.com/channels/1097401827202445382/1313534623262703667/1313925316397236234

[wswebcreation]

Hi @P-Courteille

Thanks for all info, my time is limited, but I think I can take a look at it this weekend. Can you provide is minimal producible exmaple, with all configs, package.json and a simple test?

[P-Courteille]

Hi @wswebcreation,
I will try to create this example.
The one I did from scratch doesn't reproduce the behavior so I will compare again with my previous MR and if I don't find anything I will try to create an example by removing all the useless things from my project.

Thank you for your help

[P-Courteille]

According to my following analysis, the issue started with @wdio/[email protected].

---

Original dependencies

"devDependencies": {
"@moroo/wdio-slack-reporter": "^8.1.0",
"@types/mocha": "^10.0.10",
"@wdio/allure-reporter": "^9.0.8",
"@wdio/appium-service": "^9.1.0",
"@wdio/cli": "^9.0.9",
"@wdio/local-runner": "^9.1.0",
"@wdio/mocha-framework": "^9.1.0",
"@wdio/ocr-service": "^2.2.1",
"@wdio/visual-service": "^6.2.2",
"canvas": "^3.0.0-rc2",
"i18next": "^23.16.3",
"i18next-sprintf-postprocessor": "^0.2.2",
"mocha-tags-ultra": "^1.0.3",
"tsx": "^4.19.2"
},

Not working target dependencies

"devDependencies": {
"@types/mocha": "^10.0.10",
"@types/node": "^22.10.1",
"@wdio/allure-reporter": "^9.2.14",
"@wdio/cli": "^9.4.1",
"@wdio/globals": "^9.4.1",
"@wdio/local-runner": "^9.4.1",
"@wdio/mocha-framework": "^9.2.8",
"@wdio/ocr-service": "^2.2.1",
"@wdio/visual-service": "^6.2.2",
"canvas": "^3.0.0-rc2",
"i18next": "^24.0.0",
"i18next-sprintf-postprocessor": "^0.2.2",
"mocha-tags-ultra": "^1.0.3",
"tsx": "^4.19.2"
}

Step by step update

➤ FIRST update: $${\color{green}working}$$
npm install @wdio/[email protected] --save-dev
npm install @wdio/allure-reporter --save-dev

➤ SECOND update: $${\color{green}working}$$
npm install @wdio/[email protected] --save-dev
npm install @wdio/[email protected] --save-dev
npm install @wdio/[email protected] --save-dev

➤ THIRD update: $${\color{green}working}$$
npm install @wdio/[email protected] --save-dev
npm install @wdio/[email protected] --save-dev
npm install @wdio/[email protected] --save-dev

➤ FOURTH update: $${\color{red}not~working}$$
npm install @wdio/[email protected] --save-dev
npm install @wdio/[email protected] --save-dev

➤ FIRST downgrade: $${\color{red}not~working}$$
npm install @wdio/[email protected] --save-dev
npm install @wdio/[email protected] --save-dev

➤ SECOND downgrade: $${\color{red}not~working}$$
npm install @wdio/[email protected] --save-dev
npm install @wdio/[email protected] --save-dev

➤ Return to "THIRD update" : $${\color{green}working}$$
npm install @wdio/[email protected] --save-dev
npm install @wdio/[email protected] --save-dev

➤ FIFTH update : $${\color{red}not~working}$$
npm install @wdio/[email protected] --save-dev
npm install @wdio/[email protected] --save-dev

[wswebcreation]

Thanks, I'll take a look this weekend

[P-Courteille]

Hello,

My example are finally ready.

Package

Example_OCR+visual.zip

---

Examples

The package is for both:
#657 (OCR loop)
#656 (Visual comparison no longer working)

---

Scripts

Package scripts are available:

• npm run Snapshot

  • First test with toMatchElementSnapshot $${\color{red}➤\ Fail\ with\ takeBase64ElementScreenshot\ error\ and\ malformed\ elementId\ error}$$
  • Second with toMatchScreenSnapshot $${\color{red}➤\ Fail\ with\ type\ error\ (look\ like\ path\ error)}$$

• npm run OCR

  • First test with OCR directly using browser $${\color{red}➤\ Loop\ on\ INFO\ webdriver:\ COMMAND\ ocrGetText(&lt;object&gt;)}$$
  • Second with OCR using my multiremotebrowser $${\color{red}➤\ Loop\ on\ INFO\ webdriver:\ COMMAND\ ocrGetText(&lt;object&gt;)}$$

• npm run OCR_PO

  • test with OCR with pageObject class $${\color{red}➤\ Loop\ on\ INFO\ webdriver:\ COMMAND\ ocrGetText(&lt;object&gt;)}$$
    (it was in case my implementation was in cause but the issue exist without it so I don't think so)

---

Notes

• In this example I use a WebdriverIO.MultiremoteConfig for only one device with name but on my real device, I've one wdio.conf.ts with multiple sets of capabilities with one or multiple device depending of the parameters used.

• You will maybe need to replace my selector by a selector that exists on your device (if it don't):
  //android.widget.TextView[@content-desc="Contacts"]
  (it was the Contacts icon on the main screen of my phone)

• I didn't reproduce the issue about "Snapshot service is not initialized" but technically my concerne was only about "toMatchElementSnapshot" and the "toMatchSnapshot" is not interesting in my case.

Thank you @wswebcreation and have a nice day,
Paul

[P-Courteille]

The origin of the loop is explain in my other #656 (comment)

[P-Courteille]

Thanks @wswebcreation, have a nice day

[wswebcreation]

@P-Courteille

I found the problem with the loop, I have a fix for it, but now I have an issue with the commands on the browser object like await browser.ocrGetText({ haystack: await browser.$(~Gmail) }). The browser instance works perfectly await multiremotebrowser.androidDevice.ocrGetText({ haystack: await multiremotebrowser.androidDevice.$(~Gmail) })

I need to think about how to solve this, hope to have this fixed by the end of the week

[P-Courteille]

Hi @wswebcreation,

Ok, I don't know how you fixed it but from my understanding:

• In #extendMultiremoteBrowser, you have one loop to add the command to all remote instance and then to the browser object himself

• In the ccharnkij loop, you have only one loop for the instance I think. Nothing for the browser object himself.

• In both solution, it seems to be only for the addCommand and not for overwriteCommand so from my point of view, it's a problem too.

So a mix between the two implementation with an extension to the overwriteCommand is supposed to be, from my point of view, the solution.

Something like that ?
(I haven't tested it)

if (!isStub(automationProtocol)) {
    const origAddCommand = driver.addCommand.bind(driver);
    driver.addCommand = (name, fn, attachToElement) => {
      //Add command to browser object
      driver.origAddCommand(name, fn, attachToElement)
      //Add command to browser instances
      driver.instances.forEach(
        (instance) => driver.getInstance(instance).addCommand(name, fn, attachToElement)
      );
      return origAddCommand(
        name,
        fn,
        attachToElement,
        Object.getPrototypeOf(multibrowser.baseInstance),
        multibrowser.instances
      );
    };
    const origOverwriteCommand = driver.overwriteCommand.bind(driver);
    driver.overwriteCommand = (name, fn, attachToElement) => {
      //Overwrite command to browser object
      driver.origOverwriteCommand(name, fn, attachToElement)
      //Overwrite command to browser instances
      driver.instances.forEach(
        (instance) => driver.getInstance(instance).overwriteCommand(name, fn, attachToElement)
      );
      return origOverwriteCommand(
        name,
        fn,
        attachToElement,
        Object.getPrototypeOf(multibrowser.baseInstance),
        multibrowser.instances
      );
    };
  }

Tell me if I can help and have a nice day.
Paul

[wswebcreation]

Hi @P-Courteille

The fix in the WebdriverIO package seems to be breaking stuff for multi-remote implementations. We (mainly @christian-bromann 😅 ) are looking into fixing/reverting it. The issue is that you can now not handle/manipulate the execution order when the commands are added to the browser-object. Modules that use the multiremote implementation will now only get the multiremote contexts instead of having the option to adjust how the browser instances are executed.
The implementation in the OCR service now (based on the old way) adds the commands on the browserinstance contexts, and also manipulates the execution of each browserinstance on the browser-object. With the latest fix we get into a infinite loop as you are experiencing.