How to web scrape using python without using a browser

Question

I am trying to extract some data from a website. I am using selenium for web scraping. In certain conditions, the code will have to open many tabs which become a mess. Is there any way I can Web scrape without opening a browser?

Answer 1

Yes, you can use the headless mode. When you use headless mode, you do use a browser but there is no UI for it. To initialize it, you will need to change some options. Refer to the below command:

from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')  # Last I checked this was necessary.
driver = webdriver.Chrome(CHROMEDRIVER_PATH, chrome_options=options)

This should work.

To know more, It's recommended to join our Python Training in Chennai today.

Comments

it is not working.every time i run code browser opens.

---

Hi, @Jay,

Could you please elaborate on what you tried to do and what kind of problem you are facing?