[FIXED] How do I get Python to send as many concurrent HTTP requests as possible?

Issue

I'm trying to send HTTPS requests as quickly as possible. I know this would have to be concurrent requests due to my goal being 150 to 500+ requests a second. I've searched everywhere, but get no Python 3.11+ answer or one that doesn't give me errors. I'm trying to avoid AIOHTTP as the rigmarole of setting it up was a pain, which didn't even work.

The input should be an array or URLs and the output an array of the html string.

Solution

This works, getting around 250+ requests a second. This solution does work on Windows 10. You may have to pip install for concurrent and requests.

import time
import requests
import concurrent.futures

start = int(time.time())

urls = [] #input URLs/IPs array
for y in range(5000):urls.append("https://example.com?n="+str(y))
responses = [] #output content of each request as string in an array

def send(pageUrl):
    response = requests.get(url=pageUrl)
    return pageUrl + " ~ " + str(response.content)

with concurrent.futures.ThreadPoolExecutor(max_workers=10000) as executor:
    futures = []
    for url in urls:
        futures.append(executor.submit(send, pageUrl=url))
    for future in concurrent.futures.as_completed(futures):
        responses.append(future.result())
        
end = int(time.time())
print(str(round(len(urls)/(end - start),0))+"/sec")

Output: 278.0/sec

This is a modified version of what this site showed in an example.

The secret sauce is the max_workers=10000. Otherwise, it would average about 80/sec. Although, when setting it to beyond 1000, there wasn't any boost in speed.

Answered By - Surgemus