[FIXED] Python-telegram-bot bypass flood and 429 error using Scrapy

Issue

I follow the price drops on the target site. If there is a price decrease in accordance with the rules I have set, it is recorded in the notificate table. From there, a telegram notification is sent through the code I created in the pipelines.py file. Sometimes the target site discounts too many products and 200 products can come from the notificate table. I'm stuck with telegram's flood protection while sending these.

Things I've tried:

1- Rotating multiple tokens

2- add sleep time

However, I cannot say that I was successful. I'm still stuck with the flood barrier.

What I want to do here is; No matter how many notifications come from the Notificate table, queue them and send these notifications in a way that does not exceed 20 messages per second.

How can I do that.

My pipeline.py code:

def sendnotifications(self, token):
        cursor = self.cnx.cursor()
        req = requests
        cursor.execute("SELECT * FROM notificate WHERE token= '"+token+"'")
        notifications = cursor.fetchall()
        for notification in notifications:
            print(notification)
            productid = notification[1]
            url = notification[3]
            name = notification[2]
            old = notification[4]
            new = notification[5]
            price_difference = old - new
            percentage = price_difference / old
            percentage_str = str("%.2f" % (percentage * 100))
            

            message = "<b>" + name + "</b>" + "\n\n" + \
                str(old) + " TL >>>> " + \
                str(new) + f" TL - {percentage_str}%" + "\n\n" + \
                url + "\n\n" + \
                

            if str(old) == "1.00" or str(old) == "2.00":
                message = "<b>" + name + "</b>" + "\n\n" + \
                    "<b>" + str(new) + " TL\n\n" + "</b>" + \
                    url + "\n\n" + \
                    
                

            token_list = [
                "xxxxxxxxxxxxxxxxxxxxxxx",
                "yyyyyyyyyyyyyyyyyyyyyyyy",
                "zzzzzzzzzzzzzzzzzzzzzzzzzzz",
                "aaaaaaaaaaaaaaaaaaaaaaaaaaaa",
                "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
                "ccccccccccccccccccccccccccccccccc",
          

            ]

            TOKEN = token_list[random.randint(0, len(token_list)-1)]


            
            chat_id = "-100xxxxxxxxxxxxx"
            bot = telegram.Bot(token=TOKEN)
            # tel_url = bot.sendMessage(chat_id = chat_id, text = message, parse_mode=ParseMode.HTML)
            
            try:
                bot.sendMessage(chat_id = chat_id, text = message, parse_mode=ParseMode.HTML)
                sleep(0.05)

            except Exception:

                return False
        cursor.close()
        return True

Solution

The obvious way seems to be to chunk the updates into batches of 20 and sleep for more than 1 second between them:

# ...
notifications = cursor.fetchall()
cursor.close()

for i in range(0, len(notifications), 20):
    chunk = notifications[i:i+20]
    for notification in chunk:
        print(notification)
        # ...
    time.sleep(1.5)

Answered By - thebjorn