[FIXED] Some websites dont fully load/render in selenium headless mode

Issue

So I have a problem that I have been noticing with selenium when I run it headless where some pages don't totally load/render some elements. I don't exactly know what's happening not to load 100%; maybe JS not running?

My code:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from decouple import config
from time import sleep

DEBUG = config('DEBUG')

class DiscordME(object):
    def __init__(self):
        self.LINUX = config('LINUX', cast=bool)
        self.DRIVER_VERSION = config('DRIVER_VERSION')
        self.HEADLESS = True

        options = Options()
        options.add_argument('--no-sandbox')
        options.add_argument('--disable-gpu')
        options.add_argument('--ignore-certificate-errors')
        options.add_argument('--disable-extensions')
        options.add_argument('--disable-dev-shm-usage')
        if self.HEADLESS:
            options.add_argument('--headless')
            options.add_argument('--window-size=1920,1200')

        if self.LINUX:
            self.browser = webdriver.Chrome(executable_path=f'./drivers/chromedriver-{self.DRIVER_VERSION}', options=options)
        else:
            self.browser = webdriver.Chrome(executable_path=f'.\drivers\chromedriver-{self.DRIVER_VERSION}.exe', options=options)

    def get_website(self):
        self.browser.get('https://discord.me/login')
        WebDriverWait(self.browser, 10).until(
            EC.url_changes('https://discord.me/login')
        )
        print(self.browser.current_url)
        print(self.browser.page_source)
        #print(self.browser.find_element_by_xpath('//*[@id="app-mount"]/div[2]/div/div[2]/div/div/form/div/div/div[1]/div[3]/div[1]/div/div[2]/input'))

DiscordME().get_website()

In this script, it doesn't load the login inputs when it accesses the discord API login page. As I can see in the page_source I noticed that the page is not being mounted so that could be the problem.

Solution

from selenium import webdriver
from time import sleep

options = webdriver.ChromeOptions()
options.add_argument("--window-size=1920,1080")
options.add_argument("--headless")
options.add_argument("--disable-gpu")
options.add_argument(
    "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36")
browser = webdriver.Chrome(options=options)

some websites uses user-agent to detect whether the browser is in headless mode or not as headless browser uses a different user-agent than normal browser. So explicitly set user agent.

Headless browser detection

Answered By - PDHide