[FIXED] Download file using boto3 within Docker container deployed on Sagemaker Endpoint

Issue

I have built my own Docker container that provides inference code to be deployed as endpoint on Amazon Sagemaker. However, this container needs to have access to some files from s3. The used IAM role has access to all s3 buckets that I am trying to reach.

Code to download files using a boto3 client:

import boto3

model_bucket = 'my-bucket'

def download_file_from_s3(s3_path, local_path):
    client = boto3.client('s3')
    client.download_file(model_bucket, s3_path, local_path)

The IAM role's policies:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::my-bucket/*"
            ]
        }
    ]
}

Starting the docker container locally allows me to download files from s3 just like expected.

Deploying as an endpoint on Sagemaker, however, the request times out:

botocore.vendored.requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='my-bucket.s3.eu-central-1.amazonaws.com', port=443): Max retries exceeded with url: /path/to/my-file (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x7f66244e69b0>, 'Connection to my-bucket.s3.eu-central-1.amazonaws.com timed out. (connect timeout=60)'))

Any help is appreciated!

Solution

For anyone coming across this question, when creating a model, the 'Enable Network Isolation' property defaults to True. From AWS docs:

If you enable network isolation, the containers are not able to make any outbound network calls, even to other AWS services such as Amazon S3. Additionally, no AWS credentials are made available to the container runtime environment.

So this property needs to be set to False in order to connect to any other AWS service.

Answered By - Jannik Bertram