We are developing a ML pipeline with TFX, the model is a Keras sequential DNN with mostly dense layers. We would like to customize the Evaluator component to compare the blessed and the candidate model by comparing their predictions on a test set of our choice. Therefore given the test set we want to ask both models for predictions and decide which model is the best by using a custom metric. Based on the official documentation, it seems that we should create a custom python file and provide it to the Evaluator component via the module_file parameter, however there are no information about how this file has to be. For example, which are the functions that the Evaluator component calls? (for example the trainer component calls run_fn). What output should this function return?
We are using:
- TF Version: 2.6.3
- TFX Version: 1.4.0
Thank you!