I am writting a programme in mapreduce. I need to save a big value for each key. In detail for each id(key), I want to save a value that consists of large numbers. I used numbers from 1 to 100000000. for example:
id value
1 1,3,9,23,56,345,.......,10000000000
2 6,8,45,321,876,.........,98760000876
.
.
.
100000000 1,2,6.83,90,126,567,.......,7632786765643
In each iteration the amount of numbers in each value increases. Firstly, I choose Text type for value, but in the results I saw that shuffle size became very big and I couldn't get answer. Then i choosed BitSet
Type but the process of BitSet was very slow.I don't know which data structure, I can use that can provide me with size and process speed.Can anyone help?
Thanks.
I think that you can associate a
List
for each key. So you can use aMap
wich associates an ID to a List of numbers :Map<Integer, List<Long>>