How to configure optional component with TFX?

24 views Asked by At

I would like to run one optional custom component with TFX and Airflow. Is it possible?

This optional component is used to send updates to another system, if this optional component fails, it will not be able to interrupt the execution of the main tasks.

I would like something like this:

# fictitious example

from tfx.orchestration import pipeline
from my_custom_component_1 import MyCustomComponent1
from my_custom_component_optional import MyCustomComponentOptional
from my_custom_component_2 import MyCustomComponent2
from utils import PIPELINE_NAME, PIPELINE_ROOT, get_ml_metadata_conn_config

pipeline_name = PIPELINE_NAME
pipeline_root = PIPELINE_ROOT

components = [
    MyCustomComponent1(),
    MyCustomComponentOptional(),
    MyCustomComponent2()
]

pipeline.Pipeline(
    pipeline_name=pipeline_name,
    pipeline_root=pipeline_root,
    components=components,
    enable_cache=True,
    metadata_connection_config=get_ml_metadata_conn_config(),
)

I'm using these versions:

apache-airflow = 1.10.15
tfx = 0.22.0
tensorflow = "2.2.0"

Can someone help me?

0

There are 0 answers