Unexpected conduit behaviour with wai

Question

I'm trying to write a really trivial "echo" webapp using wai; all I want it to do is reply with the data that is POSTed to it (I really don't care about the method, but I'm using curl and curl is using POST, so that's what I'm going for). My trivial web server is this:

import Network.Wai
import Network.HTTP.Types (status200)
import Network.Wai.Handler.Warp (run)
import Control.Monad.Trans ( liftIO )
import Blaze.ByteString.Builder.ByteString (fromByteString)
import qualified Data.Conduit.List as CondList
import Data.Conduit ( ($$), ($=), Flush(Chunk) )

application req = do
  let src = requestBody req $= CondList.map (Chunk ∘ fromByteString)
  return $ ResponseSource status200 [("Content-Type", "text/plain")] src

main = run 3000 application

What I expected this to do is basically tie the request body to the response body, so that when I run curl --data @/usr/share/dict/words localhost:3000; it would spit my words file back at me. Instead, it gives an empty body. Running curl with "-v" shows that my app is replying with a "200 OK" and no data. I'm not sure what I'm doing wrong here.

If I replace the application function with this:

_ ← requestBody req $$ CondList.mapM_ (liftIO ∘ print)
return $ responseLBS status200 [("Content-Type", "text/plain")] "Hello world\n"

and add an OverloadedStrings pragma to allow the "Hello World" part to work, then I do see my app printing the entire request body to stdout, so I know that curl is submitting the data properly. I also get "Hello World" printed to the curl stdout, so I know that curl works as I expect it to. I must be doing something wrong where I'm tying my requestBody to my ResponseSource, but I don't see it.

Answer 1

You're correctly using conduit, the problem is that the streaming behavior you're trying to get cannot reliably work in the context of HTTP. Essentially, you want to start sending the response body while the client is sending the request body. This could lead to a deadlock, as both the client and server could be stuck in send mode. To avoid this, Warp flushes the request body before sending the response, which is why the request body appears empty at the time the response body is sent.

In order to get the echo behavior correct, you would need to strictly consume the request body and then send it back. Obviously this could be problematic from a memory usage standpoint if there's a large request body, but this is an inherent aspect of HTTP. If you want constant memory echo, my recommendation would be to stream the request body to a file, and then use ResponseFile for the response body.