Running a simple http request in nodejs for web crawling purposes.
Unfortunately, one website we're trying to crawl is configured such that it has the Content-Length header twice.
This is throwing an error on nodejs:
Error: Parse Error
at Error (native)
at Socket.socketOnData (_http_client.js:363:20)
at emitOne (events.js:96:13)
at Socket.emit (events.js:188:7)
at readableAddChunk (_stream_readable.js:176:18)
at Socket.Readable.push (_stream_readable.js:134:10)
at TCP.onread (net.js:548:20) bytesParsed: 239, code:'HPE_UNEXPECTED_CONTENT_LENGTH' },
isOperational: true,
bytesParsed: 239,
code: 'HPE_UNEXPECTED_CONTENT_LENGTH'
I'm trying to find a workaround that doesn't involve downgrading my nodejs server... Any ideas?