scrape download file having customize selection using python selenium - stock screener site

Question

Hi,

I couldn't figure out how to paste clipboard image of the screenshot of the page nor upload the image in this query.

I would like to use Selenium and Python to download file. The thing is that there are selections that can be checked for the query before clicking on the "Download" button.

1) a set of checkbox with range bar of selection criterion e.g. Market Cap, 52 weeks % change.

2) a list of customize options (a dropdown list to display which column fields you want to see) that can be checked.

I have no idea how to code this. The code examples I managed to find teaches how to download a file but I couldn't find examples on how to download a file that has customize options.

The website is https://www2.sgx.com/securities/stock-screener

I'm grateful for your time in reading my query and appreciate if you could help me out or point me to some similar examples that I can follow.

Thank you.

Answer 1

Hi YI Wen Edwin, I am glad that I could help you with your issue. I am sharing the code snippet which would resolve your query on how to customize the filters and then download the stocks file:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

driver = webdriver.Chrome("C:\\\\Users\\\\Abha_Rathour\\\\Downloads\\\\ExtractedFiles\\\\chromedriver_win32\\\\chromedriver.exe")
driver.maximize_window()
driver.implicitly_wait(2)
driver.get("https://www2.sgx.com/securities/stock-screener")

driver.find_element_by_xpath('//*[@id="beta-warning-dialog"]/div[2]/div/div/div/sgx-input-checkbox/label/span').click()
driver.find_element_by_xpath('//*[@id="beta-warning-dialog"]/div[2]/div/div/div/div[3]/button').click()
driver.find_element_by_xpath('//*[@id="gdpr-banner"]/div/button').click()

time.sleep(5)

driver.find_element_by_xpath("//button[@class='widget-stock-screener-button widget-stock-screener-filters-toggle']").click()
driver.find_element_by_xpath("//input[@name='sector']").click()
driver.find_element_by_xpath("//span[contains(text(),'Energy')]").click()

driver.find_element_by_xpath("//span[contains(text(),'Tot. Rev ($M)')]").click()
driver.find_element_by_xpath("//span[contains(text(),'Mkt Cap ($M)')]").click()
driver.find_element_by_xpath("//span[contains(text(),'52-wk %Pr. Chg.')]").click()
driver.find_element_by_xpath("//span[contains(text(),'Net Profit %')]").click()

driver.find_element_by_xpath("//span[contains(text(),'Download')]").click()

time.sleep(3)

driver.close()

Now, read the following steps to understand the code better:

1. First import the ChromeDriver and use some of its basic methods like maximize_window() or implicitly_wait(time_in_seconds) to setup the driver properties.

2. Then to close the pop-up which shows every-time you open the page, write first 3 lines of driver.find_element... code.

3. Now, use driver.find_element_by_xpath to click on Select Sector dropdown and Show More Filters. You can use this method also for selecting which Sector you want to choose. Eg. I chose Energy sector in my code.

4. Next, use same driver.find_element_by_xpath method individually to select all the Selection Criteria you wish to choose.

5. Finally write the driver method for downloading the file.

Note: I used time.sleep(time) at 2 places so that page could all the elements properly and you could see the automation process.