2 réponses
- Le plus récent
- Le plus de votes
- La plupart des commentaires
0
Referencing the docs page https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/redshift.html , I see it says there "If you want to use IAM with Amazon Redshift Serverless, you need to set is_serverless
to true and provide serverless_work_group
."
Have you setup serverless_work_group
parameter ?
répondu il y a 4 mois
0
Thanks, the error was an airflow problem answered in awsdevelopers.slack.com, I added the serverless_work_group
parameter.
After upgrading to Airflow 2.8.1, had time out error, what worked attaching s3FullAccess
policy to the IAM role in default-namespace security and encryption
tab then
switched to recommended Data API Operator used dataset from this blog .
with DAG(dag_id="redshift", start_date=datetime(2021, 1, 1), schedule_interval=None, tags=['example']) as dag:
setup__task_create_table2 = RedshiftDataOperator(
task_id='setup__create_table2', region='us-east-1', workgroup_name='default-workgroup',
database='dev',
sql="""
CREATE TABLE public.yellow2_201601(vendorid bigint, tpep_pickup_datetime timestamp,
tpep_dropoff_datetime timestamp, passenger_count double precision, trip_distance double precision,
ratecodeid double precision, store_and_fwd_flag varchar(12), Pulocationid bigint, dolocationid bigint,
payment_type bigint, fare_amount double precision, extra double precision, mta_tax double precision,
tip_amount double precision, tolls_amount double precision, improvement_surcharge double precision,
total_amount double precision, congestion_surcharge double precision, airport_fee double precision);
""")
task_get_all_table_data2 = RedshiftDataOperator(
task_id='task_get_all_table_data2', database='dev', region='us-east-1',
workgroup_name='default-workgroup',
sql="COPY public.yellow2_201601 FROM 's3://yellow2-taxi/Parquet/' REGION 'us-east-1' IAM_ROLE 'arn:aws:iam::123456789012:role/service-role/AmazonRedshift-CommandsAccessRole-01234567T0123567' FORMAT AS PARQUET;"
)
setup__task_create_table2 >> task_get_all_table_data2
répondu il y a 3 mois
Contenus pertinents
- demandé il y a un an
- demandé il y a 2 mois
- demandé il y a 20 jours
- AWS OFFICIELA mis à jour il y a 2 ans
- AWS OFFICIELA mis à jour il y a 2 ans
- AWS OFFICIELA mis à jour il y a 2 ans