Set flink nested tuple value at runtime?

452 views Asked by At

I have app that allows the TupleI<x,y,z> data type to be created at runtime via a maven archetype. Via java reflection, I know which primitive data types x,y,z etc end up being. I also know if x,y or z etc end up being defined as tuples as well. Problem comes when I try to map the values from the source input under a class extending RichMapFunction<Tuple<x,y,z>> - i.e. I can use tuple.setfield(aValue, position) to map source to the main tuple definition, but not if the main tuple definitions contains a nested tuple, ie there is no tuple.setField(aValue, 1).setfield("",0). In this case we are trying to set the value of the first piece of data in a tuple defined as having a tuple at field 1, so a bit like :-Tuple2<String,Tuple1<Integer>>. Any way possible. (Remember, cannot refer to the fields as f0, f1, f2 etc as sthey might not exist at runtime so the application could never compile. I could write something that creates the methods that could refer to f0,f1,f2 , ie as many a are needed, before injecting that into the archetype before the application is created - but that is a fair bit of work and sounds risky when all I am doing is setting a value that is very well known to the class anyway.

1

There are 1 answers

0
david Mclaughlin On

If it helps anybody down the line, or in the my own personal interest in garnering a possibly better response, this kind of code, (not pretty) did the job in the end:-

public Tuple1<?> getTuple1 (Object f1) {
           return Tuple1.of(f1);
    }

    public  Tuple2<?,?> getTuple2 (Object f1, Object f2) {
           return Tuple2.of(f1,f2);
    }

    public  Tuple3<?,?,?> getTuple3 (Object f1, Object f2, Object f3) {
           return Tuple3.of(f1,f2,f3);
    }

just keep going for as many as you need. Of course, when you come to populate the real values your setField() calls must match the type of the inner field with the type of the data it expects - other wise a class cast is waiting for you. But this is not a problem as the Flink tuple your are trying to populate dynamically knows what it was declared as anyway, so ignores the fact that you created fields of type Object and defaults strings, and quite happily accepts your parsed values of the appropriate data type - so If you declare a Tuple1 it will gladly accept a field value of Interger atop your Object of string, without question, which is just as well as it throws said class cast otherwise. Its just strange it works like that, i.e. you cannot your reach in and set your nested value to the known type, you need this Object to String To ActualDataType kludge. (or at least I did)