Streaming video with ASP.NET WebAPI - Video seeking issue

Question

I am using PushStreamContent to stream the video hosted as a BLOB asynchronously. Here's what I've done so far

    public HttpResponseMessage Get(string filename, string extension)
    {
        var video = new VideoStream(filename, extension);
        var response = Request.CreateResponse();
        response.Content =
            new PushStreamContent(
                (Action<Stream, HttpContent, TransportContext>) video.WriteToStream,
                new MediaTypeHeaderValue("video/" + extension));
        return response;
    }

Where video.WriteToStream just reads the file and writes to output stream. Everything works great as I followed this article here.

This is how I'm using video.js to stream the video

<video id="really-cool-video" class="video-js vjs-default-skin" controls poster="@Model.MediaThumbnailUrl"
                       preload="auto" style="width: 100%; min-height: 380px; height: 100%;" autoplay
                       data-setup='{}'>
                    <source src="/api/Stream/myfilenamehere/mp4" type="video/mp4">
                    <p class="vjs-no-js">
                        To view this video please enable JavaScript, and consider upgrading to a web browser
                        that supports HTML5 video.
                    </p>
                </video>

I am having an issue when I try to drag the video forward, it restarts the video. Earlier I had a similar issue that was due to I missed to set content-type for a block blob while uploading. Am I missing something here again?