Confused by Node's filesystem parsing. Here's my code:
var fs = require('fs'),
xml2js = require('xml2js');
var parser = new xml2js.Parser();
var stream = fs.createReadStream('xml/bigXML.xml');
stream.setEncoding('utf8');
stream.on('data', function(chunk){
parser.parseString(chunk, function (err, result) {
console.dir(result);
console.log('Done');
});
});
stream.on('end', function(chunk){
// file have been read over,do something...
console.log("IT'S OVER")
});
This causes...nothing to happen. No output from XML2JS/the parser at all. When I try to console.log(chunk)
it seems that the chunks
aren't being output in any sort of meaningful chunks based on anything other than perhaps byte size. The output for one 'chunk' is:
<?xml version="1.0" encoding="UTF-8"?>
<merchandiser xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="merchandiser.xsd">
<header><merchantId>1237</merchantId><merchantName>NORDSTROM.com</merchantName><createdOn>12/13/2013 23:50:57</createdOn></header>
<product product_id="52863929">// product info</product>
<product product_id="26537849">// product info</product>
<product product_id="25535647">// product info</product>
This chunk has lots and lots of <product>
entries from the XML inside of it. The chunk will end somewhere in the middle of a <product>
entry and the next chunk will begin from where this left off.
The main question is How do I get the createReadStream
to output chunks starting at <product
and ending at </product>
?
EDIT: for the purposes of getting the proper output, here's what the XML from the beginning to the end of the first <product>
looks like:
<?xml version="1.0" encoding="UTF-8" ?>
<merchandiser xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="merchandiser.xsd">
<header>
<merchantId>1237</merchantId>
<merchantName>NORDSTROM.com</merchantName>
<createdOn>12/13/2013 23:50:57</createdOn>
</header>
<product product_id="52863929" name="Teva 'Psyclone' Print Sandal (Baby, Walker & Toddler) Camo/ Dark Olive 6 M" sku_number="52863929" manufacturer_name="Teva" part_number="1001701">
<category>
<primary>Toddler Unisex</primary>
<secondary>Shoes~~Sandals/Slides</secondary>
</category>
<URL>
<product>http://click.linksynergy.com/link?id=LUyP0GcLCGc&offerid=276223.52863929&type=15&murl=http%3A%2F%2Fshop.nordstrom.com%2FS%2F3297406%3Fcm_cat%3Ddatafeed%26cm_pla%3Dshoes%3Asandals%252fslides%26cm_ite%3Dteva_%2527psyclone%2527_print_sandal_%2528baby%252c_walker_%2526_toddler%2529%3A503158_1%26cm_ven%3DLinkshare</product>
<productImage>http://content.nordstrom.com/imagegallery/store/product/large/0/_6880020.jpg</productImage>
<buy></buy>
</URL>
<description>
<short>Rugged construction and stylish good looks define a sporty sandal, with the added convenience and security of hook-and-loop closures across the toe and at the instep.Rugged construction and stylish good looks define a sporty sandal, with the added
convenience and security of h...</short>
<long>Rugged construction and stylish good looks define a sporty sandal, with the added convenience and security of hook-and-loop closures across the toe and at the instep.Rugged construction and stylish good looks define a sporty sandal, with the added
convenience and security of hook-and-loop closures across the toe and at the instep. Color(s): camo/ dark olive, daisy blue. Brand: Teva. Style Name: Teva 'Psyclone' Print Sandal (Baby, Walker & Toddler). Style Number: 503158_1.</long>
</description>
<discount currency="USD">
<amount></amount>
<type>amount</type>
</discount>
<price currency="USD">
<sale begin_date="" end_date="">24.95</sale>
<retail>24.95</retail>
</price>
<brand>Teva</brand>
<shipping>
<cost currency="USD">
<amount>0.00</amount>
<currency>USD</currency>
</cost>
<information></information>
<availability>Y</availability>
</shipping>
<keywords></keywords>
<upc>737872649135</upc>
<m1>503158_1.</m1>
<pixel>http://ad.linksynergy.com/fs-bin/show?id=LUyP0GcLCGc&bids=276223.52863929&type=15&subid=0</pixel>
<attributeClass class_id="60">
<Misc></Misc>
<Product_Type>Shoes</Product_Type>
<Size>6 M</Size>
<Material></Material>
<Color>CAMO/ DARK OLIVE</Color>
<Gender>Unisex</Gender>
<Style></Style>
<Age></Age>
</attributeClass>
</product>
You have two possibilities to tackle your issue.
As stated by damphat, XML2JS needs the full XML content before it can parse the data. But you have a file stream, which, well, streams data chunk by chunks. The first solution is to convert this stream of data into a nice big Buffer, and then send it to XML2JS. For this purpose, you can use the
stream-to
package (npm i stream-to
) which will convert the file stream into an array of buffers, which we'll then concatenate into one single buffer usingBuffer.concat
, like this:This works quite well, but since it needs to hold the full file into memory, it won't work if your input files are really big. To handle really big files, you need to use a streaming XML parser, such as
sax
. Howeversax
doesn't create Javascript objects, but is an EventEmitter, and is a bit harder to use since you have to handle all relevant events to build your object on the fly.You can use for instance the SaXPath library, which supports a small subset of the XPath syntax. This library emits a
match
event every time it matches the XPath pattern. Here's an example:You then have two options:
xml2js
to parse a single product at a time