I'm trying to use a python script using the redshift_connector library to perform multiple COPY commands from S3 and possibly DELETE commands - all on the same redshift table. I'm using serverless redshift.
The commands on redshift gets stuck because of locks - they work from time to time but mostly I find the following locks in redshift:
- ShareRowExclusiveLock
- AccessShareLock
Both are locks on the table I work with, and the only way I found to end the process is by calling the locks_to_release stored procedure
Here is how my code looks like:
with redshift_connector.connect(...) as cnn:
with cnn.cursor() as cursor:
cursor.execute("DELETE FROM my_table WHERE ...'")
cursor.execute("COPY my_table FROM 's3://my_bucket/my_folder/' IAM_ROLE 'my-role' FORMAT AS PARQUET")