[FIXED] Make API request every x seconds in Python 3

Issue

I am trying to do stress test on a server using Python 3. The idea is to send an HTTP request to the API server every 1 second for 30 minutes. I tried using requests and apscheduler to do this but I kept getting

Execution of job "send_request (trigger: interval[0:00:01], next run at: 2017-05-23 11:05:46 EDT)" skipped: maximum number of running instances reached (1)

How can I make this work? Below is my code so far:

import requests, json, time, ipdb
from apscheduler.schedulers.blocking import BlockingScheduler as scheduler

def send_request():
    url = 'http://api/url/'

    # Username and password
    credentials = { 'username': 'username', 'password': 'password'}

    # Header
    headers = { 'Content-Type': 'application/json', 'Client-Id': 'some string'}

    # Defining payloads
    payload = dict()

    payload['item1']    = 1234
    payload['item2'] = 'some string'
    data_array = [{"id": "id1", "data": "some value"}]
    payload['json_data_array'] = [{ "time": int(time.time()), "data": data_array]

    # Posting data
    try:
        request = requests.post(url, headers = headers, data =  json.dumps(payload))
    except (requests.Timeout, requests.ConnectionError, requests.HTTPError) as err:
        print("Error while trying to POST pid data")
        print(err)
    finally:
        request.close()

    print(request.content)

    return request.content

if __name__ == '__main__':
    sched = scheduler()
    print(time.time())
    sched.add_job(send_request, 'interval', seconds=1)
    sched.start()
    print('Press Ctrl+{0} to exit'.format('Break' if os.name == 'nt' else 'C'))

    try:
        # This is here to simulate application activity (which keeps the main thread alive).
        while true:
            pass
    except (KeyboardInterrupt, SystemExit):
        # Not strictly necessary if daemonic mode is enabled but should be done if possible
        scheduler.shutdown()

I tried searching on stack overflow but none of the other questions does what I want so far, or maybe I missed something. I would appreciate someone to point me to the correct thread if that is the case. Thank you very much!

Solution

I have never used the scheduler in python before, however this other stackOverflow question seems to deal with that.

It means that the task is taking longer than one second and by default only one concurrent execution is allowed for a given job... -Alex Grönholm

In your case I imagine using threading would meet your needs. If you created a class that inherited threads in python, something like:

class Requester(threading.Thread):
  def __init__(self, url, credentials, payload):
    threading.Thread._init__(self)
    self.url = url
    self.credentials = credentials
    self.payload = payload        
  def run(self):
    # do the post request here
    # you may want to write output (errors and content) to a file
    # rather then just printing it out sometimes when using threads 
    # it gets really messing if you just print everything out

Then just like how you handle with a slight change.

if __name__ == '__main__':
  url = 'http://api/url/'
# Username and password
  credentials = { 'username': 'username', 'password': 'password'}
# Defining payloads
  payload = dict()
  payload['item1']    = 1234
  payload['item2'] = 'some string'
  data_array = [{"id": "id1", "data": "some value"}]
  payload['json_data_array'] = [{ "time": int(time.time()), "data": data_array]
  counter = 0
  while counter < 1800:
    req = Requester(url, credentials, payload)
    req.start()
    counter++
    time.sleep(1)

And of course finish the rest of it however you would like to, if you want to you could make it so that the KeyboardInterrupt is what actually finishes the script.

This of course is a way to get around the scheduler, if that is what the issue is.

Answered By - Jeff