C# HTTPWebRequest against Negotiate/Basic with Realm Site

Question

sorry for doing another question about this topic but i've been reading and can't really make it work...

Getting to the point... I've a link that will download a zip file. I've been given access to this website (my user) and now i'm trying to code a program to fetch the zip.file every once in a while so I can do things with its data.

When I just tryied the code I've to download stuff that need authentication (see code below), I get an HTTP 401. I wondered why since my credentials were ok and I could get the file using a browser.

HttpWebRequest request;
HttpWebResponse response;

request = (HttpWebRequest)WebRequest.Create(strURL);

request.Credentials = ((domain == null || domain == "") ?
                new System.Net.NetworkCredential(user, pass) :
                new System.Net.NetworkCredential(user, pass, domain)));

response = (HttpWebResponse)request.GetResponse();
//...do somthing

After reading and searching I got to the conclusion I had to explicitly tell the auth. method and found the CredentialCache example. What I did after was opening fiddler and see what the website was responding and found the following headers:

WWW-Authenticate: Negotiate
WWW-Authenticate: Basic realm="BACKLOG ::::::: AUTENTICACAO NECESSARIA - USAR APENAS USERNAME E PASSWORD DE DOMINIO"

With this I could know what auth. type I needed to, so I changed my code to the following:

HttpWebRequest request;
HttpWebResponse response;

request = (HttpWebRequest)WebRequest.Create(strURL);

CredentialCache cc = new CredentialCache();
cc.Add(new Uri(strURL), "Negotiate", (
                (domain == null || domain == "") ?
                new System.Net.NetworkCredential(user, pass) :
                new System.Net.NetworkCredential(user, pass, domain)));

request.Credentials = cc;

response = (HttpWebResponse)request.GetResponse();
//...do somthing

Still, I was getting a 401 HTTP error. After reading a bit more I found I probably needed to add request.PreAuthenticate = true; but the result was the same. I also read that, I needed to use request.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials; wich stopped the 401 HTTP error but started to give me "Too many automatic redirections were attempted" error wich made me give up of this ideia because Default Credentials is not what I want, I really want to be able to change the user credentials.

After a few more tries I gave up and tried the Basic auth. method but with no success (I think I'm missing the "realm" part somehow).

HttpWebRequest request;
HttpWebResponse response;

request = (HttpWebRequest)WebRequest.Create(strURL);

CredentialCache cc = new CredentialCache();
cc.Add(new Uri(strURL), "Basic", (
                (domain == null || domain == "") ?
                new System.Net.NetworkCredential(user, pass) :
                new System.Net.NetworkCredential(user, pass, domain)));

request.Credentials = cc;

response = (HttpWebResponse)request.GetResponse();
//...do somthing

Anyway, I lost a bit of hope on this and got confused with all the search I've done. I really feel a bit dumb about requests and responses...

Can anyone shed some ligth on the subject and help me? I would prefer the Negotiate method since it's more safe. Do I have to code somehow a second request to simulate the handshake??

======================================================================================================== UPDATE 1

After comparing requests browser vs app I can see some differences in the first request made.

Browser

GET http://example.host/maps/selector/download?map_name=workline_fornecedores&organization_id=1 HTTP/1.1 Accept: application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/x-shockwave-flash, / Accept-Language: PTSI User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; MS-RTC LM 8; .NET4.0C; .NET4.0E; .NET CLR 1.1.4322) Accept-Encoding: gzip, deflate Connection: Keep-Alive Host: example.host Cookie: WT_FPC=id=2edd0fbf206582555011375854627180:lv=1375854634144:ss=1375854627180; _swa_uv=527730711375858227; __utma=163226403.1634134224.1376900542.1376900542.1376900542.1

....

App

GET http://example.host/maps/selector/download?map_name=workline_fornecedores&organization_id=1 HTTP/1.1 Host: example.host Cache-Control: no-store,no-cache Pragma: no-cache Connection: Keep-Alive

In both samples the server responses 401 HTTP but with differences

...

Browser

GET http://example.host/authorizer/check_remote_user?url=%2Fmaps%2Fselector%2Fdownload%3Fmap_name%3Dworkline_fornecedores%26organization_id%3D1 HTTP/1.1 Accept: application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/x-shockwave-flash, / Accept-Language: PTSI User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; MS-RTC LM 8; .NET4.0C; .NET4.0E; .NET CLR 1.1.4322) Accept-Encoding: gzip, deflate Connection: Keep-Alive Host: example.host Cookie: WT_FPC=id=2edd0fbf206582555011375854627180:lv=1375854634144:ss=1375854627180; _swa_uv=527730711375858227; __utma=163226403.1634134224.1376900542.1376900542.1376900542.1; Apache=10.188.178.249.1416485792985658

...

App

GET http://example.host/authorizer/check_remote_user?url=%2Fmaps%2Fselector%2Fdownload%3Fmap_name%3Dworkline_fornecedores%26organization_id%3D1 HTTP/1.1 Host: example.host Cookie: Apache=10.188.178.249.1416491766480034 Cache-Control: no-store,no-cache Pragma: no-cache

After this, the browser request keeps sends the credentials and gets the file, while my app does nothing else.

Apart from the obvious differences on the present headers I also noted the cookie header difference can it be the cause? I'm going to make more "browse" requests in roder to check if the cookie value changes..

Answer 1

I finnaly managed to do it!

I found that, when making the request with my code I was being redirected to a second url. This url is fixed and if I try to make the request to this second link I'm able download the file.

Still can't understand why...

All I had to be sure is to have a CookieContainer in my WebRequest, otherwise a "Too many automatic redirections were attempted" exception would be thrown ( I believe this is because the destination tries to set cookies).

FINAL CODE - where strURL is the second link in text (redirected link)

HttpWebRequest request;
HttpWebResponse response;

request = (HttpWebRequest)WebRequest.Create(strURL);

CredentialCache cc = new CredentialCache();
cc.Add(new Uri(strURL), "Negotiate", (
            (domain == null || domain == "") ?
            new System.Net.NetworkCredential(user, pass) :
            new System.Net.NetworkCredential(user, pass, domain)));

request.Credentials = cc;
request.CookieContainer = new CookieContainer();

response = (HttpWebResponse)request.GetResponse();