Issue
I have one airflow connection that looks like this:
- Conn id :
my_conn_id
- Conn type:
s3
- Host:
my_host
- Login:
abcd
I tried to connect to my s3 using boto3 with the following code without luck
s3client = boto3.client(
service_name='s3',
region_name='us-east-1',
aws_access_key_id="{{ conn.conn_id.host }}",
aws_secret_access_key="{{ conn.conn_id.host }}"
)
Any help would be appreciated
Solution
Your connection is misconfigured. Enter your aws_access_key_id as login and aws_secret_key_id as password. When you generated your AWS keys, you got both the access key and secret key - you will need them both. If you don't have the secret key, regenerate the keys in AWS. See connection details here - Airflow AWS connection
In Airflow, you should use the S3Hook to generate a boto3 S3 client if you need to, but check out the functionality of the S3Hook first to see if you can use it to do your task. See functions here - S3Hook source code
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
s3client = S3Hook(aws_conn_id=my_conn_id).get_conn()
Answered By - holly.evans
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.