[FIXED] How to access a website with a non-inspectable login pop-up

Issue

I am trying to write a program to scrape data from this website: https://www.dmr.nd.gov/oilgas/feeservices/getscoutticket.asp

The issue is when you go to that link, a login pop-up appears that is non-inspectable. There is only a blank html page behind it, and I cant figure out how to get the username and password(which I have) into the login box. I have tried using various post requests, but as the pop up doesn't have a separate URL and doesn't seem to be accessible from my current code, none of them are working

from bs4 import BeautifulSoup
import requests

url = 'https://www.dmr.nd.gov/oilgas/feeservices/getscoutticket.asp'
s = requests.Session()
payload = {
    'dbconnect': 'y',
    'entryPoint': 1001,
    'nomblogon': 0,
    'Username': 'username',
    'Password': 'password',
    }
s.post(url, data = payload)

site = s.get(url)
soup = BeautifulSoup(site.content, "html.parser")

Has anyone dealt with this type of login pop-up before? Is there any way to navigate past this using code?

I am currently writing in python, but java answers would also be appreciated.

Solution

That non-inspectable pop-up is Basic HTTP Authentication, and you handle it with something like this, in requests:

from requests.auth import HTTPBasicAuth

basic = HTTPBasicAuth('user', 'pass')
requests.get('https://www.dmr.nd.gov/oilgas/feeservices/getscoutticket.asp', auth=basic)

I suggest reading the requests documentation, which can be found at https://requests.readthedocs.io/en/latest/user/quickstart/

Answered By - platipus_on_fire