Why do I sometimes get a malformed data error for some Contour boards but not others?

Question

I have a dataset that when being used as an input to build another dataset results in a 'malformed record' error indicating that something is wrong with the raw data file (some malformed values). I would therefore expect not to be able to use that dataset in Contour. So, why do I sometimes get a malformed record error for some boards in Contour but not others?

Answer 1

So the answer here is that you will be able to perform some operations in Contour, and others you won't. This depends on whether or not the spark job that is being executed under the hood in Contour actually encounters the malformed records. Basically spark is lazy so won’t actually perform all operations over all of the data - only what it needs to to show you the results. So if the function performed in Contour doesn't include the particular column / rows where the malformed records exist, you'll be able to utilize the dataset.