I would like to run one optional custom component with TFX and Airflow. Is it possible?
This optional component is used to send updates to another system, if this optional component fails, it will not be able to interrupt the execution of the main tasks.
I would like something like this:
# fictitious example
from tfx.orchestration import pipeline
from my_custom_component_1 import MyCustomComponent1
from my_custom_component_optional import MyCustomComponentOptional
from my_custom_component_2 import MyCustomComponent2
from utils import PIPELINE_NAME, PIPELINE_ROOT, get_ml_metadata_conn_config
pipeline_name = PIPELINE_NAME
pipeline_root = PIPELINE_ROOT
components = [
MyCustomComponent1(),
MyCustomComponentOptional(),
MyCustomComponent2()
]
pipeline.Pipeline(
pipeline_name=pipeline_name,
pipeline_root=pipeline_root,
components=components,
enable_cache=True,
metadata_connection_config=get_ml_metadata_conn_config(),
)
I'm using these versions:
apache-airflow = 1.10.15
tfx = 0.22.0
tensorflow = "2.2.0"
Can someone help me?