Parsing Deequ Rules from a csv/table dynamically

743 views Asked by At

I'm using amazon deequ library and trying to pass the rules from a csv or a mysql table. My csv file will have the column with values like below

rules_column
-------------------------------------------------------------------------
.hasCompleteness("column1", _ >= 0.86, Some("It should be above 0.86!"))                        
.hasDataType("colm2", ConstrainableDataTypesIntegral)                                            
.isComplete("column3")                                                                          
.isContainedIn("column1", Array("R", "M", "A", "L"))                                            
.isNonNegative("col7") 

I'm trying to find if there's a way to translate this string column directly to contraints. Here's a psudo code of what I'm trying to do.

 var check = Check(CheckLevel.Error, "Data Validation Check")
 for (constrain <- rules_column){ 
   check = check + constrain } // constrain is a string here 

This way...if there's a new constrain say containsEmail in the table, i don't have to come back to my code and separately write parsing logic to translate containsEmail string to containsEmail constraint.

0

There are 0 answers