How To Get All Links From The Dom?

According to https://github.com/GoogleChrome/puppeteer/issues/628, I should be able to get all links from < a href='xyz' > with this single line: const hrefs = await page.$$e

Solution 1:

In your example code you're using page.$eval, not page.$$eval. Since the former uses document.querySelector instead of document.querySelectorAll, the behaviour you describe is the expected one.

Also, you should change your pageFunctionin the $$eval arguments:

const hrefs = await page.$$eval('a', as =>as.map(a => a.href));

Solution 2:

The page.$$eval() method runs Array.from(document.querySelectorAll(selector)) within the page and passes it as the first argument to the page function.

Since a in your example represents an array, you will either need to specify which element of the array you want to obtain the href from, or you will need to map all of the href attributes to an array.

page.$$eval()

const hrefs = await page.$$eval('a', links => links.map(a => a.href));

---

Alternatively, you can also use page.evaluate() or a combination of page.$$(), elementHandle.getProperty(), or jsHandle.jsonValue() to achieve an array of all links from the page.

page.evaluate()

const hrefs = await page.evaluate(() => {
  returnArray.from(document.getElementsByTagName('a'), a => a.href);
});

page.$$() / elementHandle.getProperty() / jsHandle.jsonValue()

const hrefs = awaitPromise.all((await page.$$('a')).map(async a => {
  returnawait (await a.getProperty('href')).jsonValue();
}));