How to escape delimiter found in value - pig script?

Question

In pig script, I would like to find a way to escape the delimiter character in my data so that it doesn't get interpreted as extra columns. For example, if I'm using colon as a delimiter, and I have a column with value "foo:bar" I want that string interpreted as a single column without having the loader pick up the comma in the middle.

Answer 1

you can use STRSPLIT(string, regex, limit); for the column split based on the delimiter.

Answer 2

It seems Pig takes the Input as the string its not so intelligent to identify how what is data or what is not.

The pig Storage works on the Strong Tokenizer. So if u want to do something like

a = LOAD '/abc/def/file.txt' USING PigStorage(':');

It doesn't seems to be solving your problem. But if we can write our own PigStorage() Method possibly we could come across some solution.

I will try posting the Code to resolve this.

Answer 3

You can try http://pig.apache.org/docs/r0.12.0/func.html#regex-extract-all

A = LOAD 'somefile' AS (s:chararray);
B = FOREACH A GENERATE FLATTEN(REGEX_EXTRACT_ALL(s, '(.*) : (.*)'));

The regex might have to be adapted.