[FIXED] 500 Error When Trying to Scrape ASP.NET Page in Python

Issue

I'm writing a python script to scrape my employer's online electronic schedule portal written in ASP.NET and present the data as a proper calendar feed. My progress so far can be seen at https://github.com/Renkin42/sws-scraper/blob/main/scraper.py . I have successfully logged in and retrieved the event data for the current week. However to get next week's data (arguably more useful) requires a second POST request to change from "Current Week" to "Specific Date" and set that date to Sunday of next week. Viewing this POST in my browser the response is a partial html tree of type "XHR", which seems to contain the data I'm after. However replicating the POST in my script I get the following response: 179|error|500|The page is performing an async postback but the ScriptManager.SupportsPartialRendering property is set to false. Ensure that the property is set to true during an async postback.| . I can only imagine this is something being handled on the javascript side, as I don't see anything about this in the POST payload. Any suggestions on how I might proceed from here? I've seen mention of Selenium, but since it basically runs a browser in the background, I'm not sure if it can be used on a headless server without X.

I've checked that my script matches the POST as seen in my browser dev tools, including empty parameters. Program currently returns 500 error instead of expected partial html tree. Unsure how to proceed.

Solution

I found the issue. The final argument of my POST request was ”__ASYNCPOST”:”true”, which I copied from the request in my browser. Changing it to “false” got rid of the error and returned the full html response rather than a partial one.

Answered By - Austin Leydecker