2 Antworten
- Neueste
- Die meisten Stimmen
- Die meisten Kommentare
0
Referencing the docs page https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/redshift.html , I see it says there "If you want to use IAM with Amazon Redshift Serverless, you need to set is_serverless
to true and provide serverless_work_group
."
Have you setup serverless_work_group
parameter ?
beantwortet vor 4 Monaten
0
Thanks, the error was an airflow problem answered in awsdevelopers.slack.com, I added the serverless_work_group
parameter.
After upgrading to Airflow 2.8.1, had time out error, what worked attaching s3FullAccess
policy to the IAM role in default-namespace security and encryption
tab then
switched to recommended Data API Operator used dataset from this blog .
with DAG(dag_id="redshift", start_date=datetime(2021, 1, 1), schedule_interval=None, tags=['example']) as dag:
setup__task_create_table2 = RedshiftDataOperator(
task_id='setup__create_table2', region='us-east-1', workgroup_name='default-workgroup',
database='dev',
sql="""
CREATE TABLE public.yellow2_201601(vendorid bigint, tpep_pickup_datetime timestamp,
tpep_dropoff_datetime timestamp, passenger_count double precision, trip_distance double precision,
ratecodeid double precision, store_and_fwd_flag varchar(12), Pulocationid bigint, dolocationid bigint,
payment_type bigint, fare_amount double precision, extra double precision, mta_tax double precision,
tip_amount double precision, tolls_amount double precision, improvement_surcharge double precision,
total_amount double precision, congestion_surcharge double precision, airport_fee double precision);
""")
task_get_all_table_data2 = RedshiftDataOperator(
task_id='task_get_all_table_data2', database='dev', region='us-east-1',
workgroup_name='default-workgroup',
sql="COPY public.yellow2_201601 FROM 's3://yellow2-taxi/Parquet/' REGION 'us-east-1' IAM_ROLE 'arn:aws:iam::123456789012:role/service-role/AmazonRedshift-CommandsAccessRole-01234567T0123567' FORMAT AS PARQUET;"
)
setup__task_create_table2 >> task_get_all_table_data2
beantwortet vor 3 Monaten
Relevanter Inhalt
- AWS OFFICIALAktualisiert vor 2 Jahren
- AWS OFFICIALAktualisiert vor 2 Jahren
- AWS OFFICIALAktualisiert vor einem Jahr
- Wie verbinde ich verschiedene Zweigstellen mithilfe von AWS Site-to-Site VPN und AWS Direct Connect?AWS OFFICIALAktualisiert vor einem Jahr