[FIXED] Azure Functions Python Vx - ModuleNotFound Exception with ORM

Issue

Context
I am using Azure Functions (I have tried both v1, and v2) to coordinate a set of scripts that trigger on HTTP. These scripts will pull data from an API, transform, then upload to a PostgreSQL database using the Pony ORM. The script works locally using Python 3.8, and the only reason I am tied to using Az Functions is for the below reasons:

1. Functions natively support HTTP triggers for these scripts without any overhead manipulation on my end. This can be integrated into the downstream solution to allow the client to trigger a refresh of the data.

2. Environment variables can be adjusted using the Azure portal, making it simpler for the non-tech clients to adjust the transformations without having to get into the system.

Problem
When I attempt to move my scripts into an Azure Function, I am unable run this locally, and am faced with a ModuleNotFound Exception with Pony. I have attempted to use other ORMs such as PeeWee, but run into the same issue.

My Attempted Solutions
I have verified that the requirements.txt file is up to date with all necessary packages. I have also manually installed these packages and verify that they appear in the venv folder. I have tried moving the database code into helper files (outside of the AzFunction) because for some reason I thought that might work, but to no avail. I will be attempting Django next, but am hesitant because I am unsure how it might affect performance (and it will require quite a refactor, it would be easier if I could just get Pony to work). From Microsoft Docs, I know that this may work better, as it relates to the linux-wheels and the respective packages having support for linux (as this is what Azure Functions run on). !

Output

When I run my functions locally, I receive this error:

 *  Executing task: .venv\Scripts\python -m pip install -r requirements.txt 

Requirement already satisfied: azure-functions in c:\users\...
Requirement already satisfied: pony in c:\{pwd}\.venv\lib\site-packages (from -r requirements.txt (line 6)) (0.7.17)
Requirement already satisfied: ratelimit in c:\{pwd}\.venv\lib\site-packages (from -r requirements.txt (line 7)) (2.2.1)
Requirement already satisfied: requests in c:\{pwd}\.venv\lib\site-packages (from -r requirements.txt (line 8)) (2.31.0)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\{pwd}\.venv\lib\site-packages (from requests->-r requirements.txt (line 8)) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in c:\{pwd}\.venv\lib\site-packages (from requests->-r requirements.txt (line 8)) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\{pwd}\.venv\lib\site-packages (from requests->-r requirements.txt (line 8)) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in c:\{pwd}\.venv\lib\site-packages (from requests->-r requirements.txt (line 8)) (2023.7.22)

[notice] A new release of pip available: 22.3 -> 23.3.1
[notice] To update, run: C:\{pwd}\.venv\Scripts\python.exe -m pip install --upgrade pip
 *  Terminal will be reused by tasks, press any key to close it. 

 *  Executing task: .venv\Scripts\activate ; func host start 

Found Python version 3.8.10 (python3).

Azure Functions Core Tools
Core Tools Version:       4.0.5455 Commit hash: N/A  (64-bit)
Function Runtime Version: 4.27.5.21554

[2023-11-12T02:04:03.820Z] Customer packages not in sys path. This should never happen! 
[2023-11-12T02:04:06.309Z] Error in index_function_app. Sys Path: {$PATH}
[2023-11-12T02:04:06.334Z] Worker failed to index functions
[2023-11-12T02:04:06.344Z] Result: Failure
Exception: ModuleNotFoundError: No module named 'ratelimit'. Troubleshooting Guide: https://aka.ms/functions-modulenotfound
Stack:   File "C:\Users\Nathan Verghis\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\python\3.8/WINDOWS/X64\azure_functions_worker\dispatcher.py", line 345, in _handle__functions_metadata_request
    fx_metadata_results = self.index_functions(function_path)
  File "C:\Users\Nathan Verghis\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\python\3.8/WINDOWS/X64\azure_functions_worker\dispatcher.py", line 617, in index_functions
    indexed_functions = loader.index_function_app(function_path)
  File "C:\Users\Nathan Verghis\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\python\3.8/WINDOWS/X64\azure_functions_worker\utils\wrappers.py", line 48, in call
    raise extend_exception_message(e, message)
  File "C:\Users\Nathan Verghis\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\python\3.8/WINDOWS/X64\azure_functions_worker\utils\wrappers.py", line 44, in call
    return func(*args, **kwargs)
  File "C:\Users\Nathan Verghis\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\python\3.8/WINDOWS/X64\azure_functions_worker\loader.py", line 214, in index_function_app
    imported_module = importlib.import_module(module_name)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "C:\{pwd}\function_app.py", line 4, in <module>
    from ratelimit import limits, sleep_and_retry
.
[2023-11-12T02:04:07.068Z] No job functions found. Try making your job classes and methods public. If you're using binding extensions (e.g. Azure Storage, ServiceBus, Timers, etc.) make sure you've called the registration method for the extension(s) in your startup code (e.g. builder.AddAzureStorage(), builder.AddServiceBus(), builder.AddTimers(), etc.).
For detailed output, run func with --verbose flag.

I will keep this updated if it works.If anyone knows a way to implement this functionality without Functions (while retaining the above benefits), or how to get the ORMs to correctly install on the Functions, do let me know. Thanks

-- UPDATE --

I was originally developing on Windows. Switching to Mac seemed to avoid the ModuleNotFoundError entirely. I discovered some bugs in the code I was deploying initially, and after fixing them was able to get the code working on Azure as well. Bringing the debugged code back to Windows and still no luck with local. At this point I'm just curious as to what caused the issue.

Solution

I tried the below Function code with pony-orm package connecting to PostgreSQL and it worked successfully like below:-

Function Python - v1, Python version 3.10

init.py code:-

import os
import json
import azure.functions as func
from pony.orm import Database, Required, db_session, select

# Define your Pony ORM entities here
db = Database()

# Define your entity model (example)
class Product(db.Entity):
    name = Required(str)
    price = Required(float)

# Initialize Pony ORM database connection
def initialize_database():
    db.bind(provider='postgres', user='siliconuser', password='xxxx',
            host='xxxxnpostgresql.postgres.database.azure.com', database='postgres')
    db.generate_mapping(create_tables=True)

# Call the initialize function outside of a db_session
initialize_database()

# HTTP trigger function
def main(req: func.HttpRequest) -> func.HttpResponse:
    with db_session:
        # Example: Inserting data into the database
        new_product = Product(name='Sample Product', price=20.0)

    with db_session:
        products = select(p for p in Product)[:]
        products_list = [{'name': p.name, 'price': p.price} for p in products]

    return func.HttpResponse(json.dumps(products_list), mimetype="application/json")

My requirements.txt :-

For pony package to work with postgresql psycopg2-binary package is also required.

azure-functions

pony

psycopg2-binary

Output:-

While loading the Azure Functions locally and to install the package via virtual environment, Click on fn + f5 or Just f5 this will initialize the Function to run by itself along with venv package. Make sure you have Azure Functions Core tools installed in your system.

And while deploying the Function instead of using, Azure Functions Extension, I have used Azure Functions Core tools command like below:-

az login
az account set --subscription "SID Subscription"
func azure functionapp publish siliconfunc89

Output:-

The HttpTrigger got deployed and running successfully refer below:-

Azure Function app Python version- 3.10 with Premium Function plan:-

While importing large packages using Premium or Dedicated Function plan is more efficient.

Note- I am using Azure PostgreSQL.

Also, In your error code- python Customer packages not in sys path. This should never happen! [2023-11-12T02:04:06.309Z] Error in index_function_app. Sys Path: {$PATH} [2023-11-12T02:04:06.334Z] Worker failed to index functions Error occurs when the python packages are not added in the correct folder while deployment. In order to resolve it and get more insights on the same refer this SO thread answer by Abdul Niyas P M and Pravallika. Make sure you downgrade your Azure Functions Core tools to python 4.0.534

My Azure functions core tools version:-

4.0.5148

You can also initialize the virtual environment manually and then install package and run your function with func start command:-

py -m venv .venv
.venv\scripts\activate

Answered By - SiddheshDesai