How to get page source as it is in browser using selenium

Question

I know that driver.getPageSource() gets you the web page as Html but the issue I am having is that the site I am testing puts data in brackets like {{{DepartDayShort}}}, maybe using some js framework to get the data? I can view the data when I inspect the element in browser but viewing the page source in browser also show only the brackets and not the data.

Below is the website I am testing to be exact:

driver.get("https://www.united.com/ual/en/US/flight-search/book-a-flight/results/rev?f=Miami,+FL,+US+(MIA+-+All+Airports)&t=IAH&d=2020-05-02&r=2020-05-16&sc=7,7&px=1&taxng=1&newHP=True&idx=1");
String pageSource = driver.getPageSource();

Have you tried using

String pageSource=driver.findElement(By.tagName("body")).getText() — Apr 17, 2020
Hi, I tried that but it only gives me stripped-down information like page title and headers. It does not contain much of the pages info. — laiman, Apr 17, 2020

laiman · Answer 1 · Apr 18, 2020

Best answer

I needed to wait for an element to load first before getting the page source:

WebDriverWait wait = new WebDriverWait(driver, SECONDS);
wait.until(ExpectedConditions.elementToBeClickable(By.className("ClassName")));

String pageSource = driver.getPageSource();

For further understanding, you can refer to the Selenium Course.

answered Apr 18, 2020 by laiman
• 330 points
selected Apr 20, 2020 by Sirajul

laiman · Answer 2 · Apr 17, 2020

I too had faced a problem while viewing the source code in the browser. There were many extra characters that got appended apparently. By using JavaScript code to get the innerHTML property, it finally worked for me. You could probably try executing the following code:

String javascript = "return arguments[0].innerHTML";
String pageSource=(String)(JavascriptExecutor)driver)
    .executeScript(javascript, driver.findElement(By.tagName("html")));
pageSource = "<html>"+pageSource +"</html>";
System.out.println(pageSource);
//FileUtils.write(new File("e:\\test.html"), pageSource,);

answered Apr 17, 2020 by Liana

Hi, this is an interesting approach. However, it was still not getting the data I was looking for. It did head me in the right direction as I needed to use WebDriverWait wait and ExpectedConditions.elementToBeClickable to get the data.

commented Apr 18, 2020 by laiman
• 330 points

I had this in a different situation where the page is different from the save page data, and it worked perfect.

String javascript = "return arguments[0].innerHTML";
String pageSource =(String)((JavascriptExecutor)driver).executeScript(javascript, driver.findElement(By.tagName("html")));
pageSource = "<html>"+pageSource +"</html>";