[FIXED] How can I change "display: flex" to "display: none" when parsing (scraping) a web-page? | Python (telegram-bot) | Selenium

Issue

Good afternoon. I'm working on a telegram bot that parses (scrapes) information from public sources. Language: Python

In this particular case you need to make a simple screenshot of the site and send it to the user. Link to the site: https://deepstatemap.live/

Procedurally everything is done, but there is a problem, when visiting the site, a disclaimer appears before the first confirmation, namely:

<div class="disclaimer" style="display: flex;">...</div>

Accordingly, a screenshot of the disclaimer appears.

Can you please tell me how I can replace the html layout with:

<div class="disclaimer" style="display: none;">...</div>

To get access to the exact material I need.

Code snippet:

import telebot
from telebot import types
import requests
from bs4 import BeautifulSoup
from config import TOKEN, URL6
import time
from selenium import webdriver
from PIL import Image
import os

options = webdriver.ChromeOptions()
options.add_argument('headless')
options.add_argument('--disable-extensions')
options.add_argument('--disable-gpu')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--no-sandbox')

@bot.message_handler(content_types=['text'])
def text(message):
    if message.chat.type == 'private':
        if message.text == 'MAP':
        url6 = URL6
        photo_path6 = str(message.chat.id) + '.png'
        driver = webdriver.Chrome(options=options)
        driver.set_window_size(1140, 1140)
        driver.get(url6)
        time.sleep(3)
        driver.save_screenshot(photo_path6)
                
        bot.send_photo(chat_id=message.chat.id,
        photo=open(photo_path6, 'rb'),
        reply_markup=markup10, parse_mode='html')
                
        driver.quit()
        os.remove(photo_path6)

The web-page fragment that is responsible for the disclaimer:

<div class="disclaimer" style="display: flex;">
<div class="inner">
<div class="inner-text">
<h2>УВАГА!</h2>
<p>
Мапу <strong>небезпечно та заборонено</strong> використовувати для прокладання маршрутів евакуації, вона має неточності та оновлюється з затримкою
</p>
</div>
<div class="inner-buttons">
<button class="disagree">НЕ<br>ПОГОДЖУЮСЬ</button>
<button class="agree">ПОГОДЖУЮСЬ</button>
</div>
<center><img alt="DeepStateMap Logo" src="images/meta_og.jpg"></center>
</div>
</div>

Result so far: enter image description here

Solution

You can capture the div as a WebElement and then pass it to driver.execute_script() to set the style attribute to display: none;

This code should work for you:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By


driver = webdriver.Chrome(executable_path='D://chromedriver/100/chromedriver.exe')
wait = WebDriverWait(driver, 20)

url = "https://deepstatemap.live/"

driver.get(url)

# capturing the disclaimer div
disclaimer = wait.until(EC.presence_of_element_located((By.XPATH, '//div[@class="disclaimer"]')))

# setting the style attribute to display: none;
driver.execute_script("arguments[0].setAttribute('style', 'display: none;')", disclaimer)

Answered By - Shine J