Consistent Page Close Issue in Playwright Crawler (Crawlee) on Async Locator Operations

[wojtekKrol]

Hello everyone,

I'm encountering a consistent issue with Playwright in the context of the Crawlee library. Each time I perform an async operation on a locator instance, specifically using await, the page unexpectedly closes. This issue happens every single time without fail.

The following simplified code snippet showcases where the problem occurs:

const doesContainAllParts: AlertsProximityAnalyzerComparator<
  Frame | Page
> = async (element) => {
  try {
    const test = element.locator('body');
    const result = await test.count();  // Page closure issue occurs here

    return result > 0;
  } catch (error) {
    console.error('Error in doesContainAllParts:', error);
    throw error;
  }
};

Key details:

• The problem is consistently reproducible every time this code is executed.
• I am using the latest versions of both Playwright and Crawlee.
• The issue appears to be linked to the use of await on the locator instance.

I am at a loss as to why this is happening. Is this a known issue with Playwright or Crawlee, or is there a mistake in my implementation? Any insights, advice, or shared experiences would be immensely appreciated.

Thank you in advance for any help you can provide!

Those are settings of my my Playwright instance:

  private initializeCrawler = (): PlaywrightCrawler => {
    return new PlaywrightCrawler({
      autoscaledPoolOptions: {
        loggingIntervalSecs: null,
      },
      browserPoolOptions: {
        operationTimeoutSecs: 30,
        retireBrowserAfterPageCount: 2,
        maxOpenPagesPerBrowser: 10,
        closeInactiveBrowserAfterSecs: 200,
      },
      launchContext: {
        launchOptions: { headless: false },
      },
      requestHandlerTimeoutSecs: 10,
      maxRequestRetries: 3,
      keepAlive: true,
      requestHandler: this.requestHandler.bind(this),
      failedRequestHandler: this.failedRequestHandler.bind(this),
    })
  }

private requestHandler: PlaywrightRequestHandler = async ({
    page,
    request,
    log,
  }) => {
    log.info(`Request to: ${request.url} ...`)
    try {
      await playwrightUtils.infiniteScroll(page, {
        scrollDownAndUp: true, // some page requires to scroll up and down to load new content
        waitForSecs: 2, // how long wait until new content is loaded
        timeoutSecs: 5, // scroll max for N seconds
      })
      this.resolvePromise(request.url, page)
    } catch (error) {
      this.rejectPromise(request.url, error)
    }
  }

  private failedRequestHandler: BrowserErrorHandler<
    PlaywrightCrawlingContext<Dictionary>
  > = ({ request, log }, error) => {
    log.info(`Request to: ${request.url} failed...`)

    this.rejectPromise(request.url, error)
  }

Here is a video presenting how it works in my case:

2023-11-17.09-36-57.mp4

If you need any additional information/data just let me know,

Thanks in advance