[FIXED] Is there a way for a scraping script to pause for the user to manually log into a website and then resume with scraping the site?

Issue

Here is what I have so far:

from time import sleep
from selenium import webdriver

driver = webdriver.Chrome('/Users/uglyr/chromedriver')  
driver.get('https://www.linkedin.com')

#now the script must pause until I manually login.

...

#after successful login the script must resume taking me from my feed page to my profile page

profile = driver.find_element_by_xpath("//div[@data-control-name='identity_profile_photo']/parent::a")

profile.click()

sleep(4)
# the code to scrape my own profile

I am building a webapp and I would like to give the users the ability to import their profile in the webapp after they logged in their LinkedIn account.

After scraping the profile information I would display it for the user to confirm and import this info to the app's database. Most probably I will need to run the webdriver remotely, but will cross that bridge when I get to it.

I would appreciate any ideas you may have.

Solution

You have to use the wait.until function. You can read more about it here: https://selenium-python.readthedocs.io/waits.html

from time import sleep
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 

driver = webdriver.Chrome('C:/chromedriver')  
driver.get('https://www.linkedin.com/login?')

wait = WebDriverWait(driver, 10)

wait.until(EC.visibility_of_element_located((By.XPATH, '//*[@id="ember227"]')))

profile = driver.find_element_by_xpath('/html/body/div[8]/div[3]/div/div/div/aside[1]/div[1]/div[1]/a')

profile.click()

sleep(4)

Answered By - Alexander Riedel