[BUG] Test not found error when using async/await calls to fetch data

[mailtogagan]

Context:

• Playwright Version: 15.2
• Operating System: Windows
• Node.js version: 14.17.3
• Browser: Chromium

Describe the bug

My use case is a simple data driven scenario. Where I am using async/await calls when reading data sets under test.describe its not able to identify the test available. Maybe because test.describe can not be async. Getting no tests found response when executing my test file.

// MY TEST  FILE -- 1
import { test, expect } from '@playwright/test';
import { TestData } from '../../data/testData';
test.describe('Test suite 1', async () => {
  const sample = new TestData();
  await console.log('Testing starts');
  const urlList = await sample.getUrlList();
  urlList.forEach((url) => {
    test(`Testing URL:  ${url}`, async ({ page }) => {
      await page.goto(url);
    });
  });
});

============================================

// TEST DATA FILE
export class TestData {
  async getUrlList() {
    const urlList = ['https://playwright.dev', 'https://google.com', 'https://github.com/'];
    return urlList;
  }
}

Even if I don't use test.describe as async and move my fetching dataset logic to test.beforeAll hook then also its not working.. Maybe the reason is that its not looking for the test inside the loop i.e. urlList.forEach. Getting no tests found response when executing my test file.

// MY TEST  FILE -- 2
import { test, expect } from '@playwright/test';
import { TestData } from '../../data/testData';
test.describe('Test suite 1', () => {
  let urlList = [];
  test.beforeAll(async () => {
    const sample = new TestData();
    await console.log('Testing starts');
    urlList = await sample.getUrlList();
  });
  urlList.forEach((url) => 
    test(`Testing URL:  ${url}`, async ({ page }) => {
      await page.goto(url);
    });
  });
});

[mailtogagan]

@pavelfeldman As discussed I have raise this issue to track the feasibility with respect to playwright-test.

[dgozman]

We've discussed async describes and decided to not support them for now. test.describe body first runs when we collect tests, and then in every worker that actually runs the tests (usually just one for a particular file). Therefore, doing work in describe body does not look like ideal.

---

Workaround #1. Instead of multiple tests, use multiple steps.

test(`Testing multiple urls`, async ({ page }) => {
  const urlList = await sample.getUrlList();
  for (const url of urlList) {
    await test.step(`testing url: ${url}`, async () => {
      await page.goto(url);
      // ...
    });
  }
});

---

Workaround #2. Prepare data in global setup and store it in the file. Read from the file (synchronously) in describe body.

// global-setup.js
export default async function() {
  const urlList = await sample.getUrlList();
  fs.writeFileSync('./urls.json', JSON.stringify(urlList), 'utf-8');
};

//example.spec.js
test.describe('Test suite 1', async () => {
  const urlList = require('./urls.json');
  urlList.forEach((url) => {
    test(`Testing URL:  ${url}`, async ({ page }) => {
      await page.goto(url);
    });
  });
});

[mailtogagan]

@dgozman thanks for the detailed response.
I think Workaround #1 wont work exactly because test case execution will stop as soon as any test.step fails inside for loop.

[imartinflores]

Hey @dgozman is this still the behaviour of describe? Does not work with async? Sorry to bother here, didnt want to file a new issue just for this! Thank you!

[dgozman]

@imartinflores Yes, this has not changed, describe should be synchronous.

[imartinflores]

Thanks a lot!