[FIXED] The website seems to detect selenium, blocks jquery and gives PR_CONNECT_RESET_ERROR

Issue

I'm new to Selenium, so I'm sorry if that's a noob issue.

I'm trying to automate the generation of documents through this with Python and Selenium.

When I use the automated firefox or chrome driver, after the final step, which is clicking the button "gerar 2a via" (xpath = '//* [@id="btnGerarSegundaVia"]'), the page freezes and gives nothing besides an icon which doesn't even load properly. There are also some jquery errors in the network tab in the developer's tool.

Another thing that's interesting is the fact that when you try to reload the page through the automated browser, the server returns PR_CONNECT_RESET_ERROR.

To emulate the problem on your end:

1. Open one normal browser and another automated driver browser.
2. Input the following data:

Normal Browser:

Automated Browser:

3. Click the button
4. See the different outputs for the normal and automated browsers:

5. And when you refresh the page on the automated browser:

I coded only the opening of the automated browser to test some options and none of the commented out code resolved my issue.

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.firefox.options import Options


# profile_path = r'whatever'
# options = Options()
# PROXY_HOST = "12.12.12.123"
# PROXY_PORT = "1234"
# options.set_preference('profile', profile_path)
# options.set_preference("network.proxy.type", 1)
# options.set_preference("network.proxy.http", PROXY_HOST)
# options.set_preference("network.proxy.http_port", int(PROXY_PORT))
# options.set_preference("dom.webdriver.enabled", False)
# options.set_preference('useAutomationExtension', False)
# options.add_argument('--ignore-certificate-errors')

# desired_caps = DesiredCapabilities.FIREFOX.copy()
# desired_caps.update({'acceptInsecureCerts': True, 'acceptSslCerts': True})

driver = webdriver.Firefox()


driver.get('https://iptu.prefeitura.sp.gov.br/')

If I had to guess, the problem is that the site detects that the 'connection is not secure' because Selenium is controlling the browser and then blocks the response from the server, but you guys are smarter than me, so please HEEEELP

Thanks

Solution

The following will bring you to the point of inserting captcha text, then when you click the button, it will take you to the pdf page:

import undetected_chromedriver as uc

from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

browser = uc.Chrome()

url = 'https://iptu.prefeitura.sp.gov.br/'

browser.get(url) 

contrib_field = WebDriverWait(browser,10).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='txtNumeroContribuinte']")))
contrib_field.send_keys('019.045.0170-2')

parcel_field = Select(WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//select[@id='comboBoxParcela']"))))
parcel_field.select_by_index(7)

ex_field = WebDriverWait(browser,10).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='txtExercicio']")))
ex_field.send_keys('2022')

About undetected_chromedriver package: https://pypi.org/project/undetected-chromedriver/

Answered By - platipus_on_fire