Gulp. How to merge files without lines duplication

Question

I need to merge 2 files with gulp. Like

concat = require('gulp-concat')

gulp.src(['file1', 'file2'])
.pipe(concat('merge'))
.pipe(gulp.dest('dest/'));

But without lines duplication. So if two files contains same line, merge file needs to contain this line only once.

file1

lineA
lineB
lineC

file2

lineD
lineA
lineF

merge

lineA
lineB
lineC
lineD
lineF

Is there some plugin for this? Help please)

Answer 1

You need to use a buffer mapper, here is an example with map-stream:

var concat = require('gulp-concat');
var map    = require('map-stream');

var uniqueFilterFn = function(item, idx, all) {
  return idx === all.indexOf(item);
};

gulp.src(['file1', 'file2'])
.pipe(concat('merge'))
.pipe(map(function(file, cb) {

  // convert file buffer into a string
  var contents = file.contents.toString();

  // split it by lines
  var lines = contents.split(/[\r\n]/);

  // apply the unique filter
  var uniqueLines = lines.filter(uniqueFilterFn);

  // join unique list into lines
  var output = uniqueLines.join('\n');

  // convert string back into buffer
  var buffer = new Buffer(output, 'binary');

  // replace the file contents
  file.contents = buffer;

  // continue
  return cb(null, file);
}))
.pipe(gulp.dest('dest/'));