How to use a function from one glue script to another in AWS glue

Question

I have one AWS glue pyspark script for example scriptA.py. In this script I have defined few generic functions like readSourceData()

def readSourceData(parameter1, parameter2):

//logic of function

Now I want to use this generic function in my secong glue pyspark script scriptB.py.

I have many such generic functions. How can I import these functions and use them in other scripts?

Answer 1

You can create modules with your generic functions and attach those external python modules to your glue jobs. More on this you can read up here.

Extensive answer:

1. You bundle your generic functions in a python module.
2. Then you .zip the module and upload it to S3.
3. You add the path of your module in S3 to your Glue job in the Python library path field

Make sure that your Job Role has access to the location in S3.