ServerXMLHTTP request returing data but not returning url of final page after 301 redirection

Question

I am trying to make LINK FINDER app in ASP
It working is divided into 5 step

1. Send http request to server at www.foo.com
2. Check status of request
3. If its 200 then move to step 4 otherwise show error
4. Parse all link
5. Send http request to server to parsed link

I am able to do first 4 step, But facing challenge in 5th step

I am getting 3 type of links

1.)absolute link : http://www.foo.com/file.asp
2.)links from root directory, which need domain name eg /folder2/file2.asp
3.)relative link : ../file3.asp

Challenge

When I am requesting www.foo.com , which is 301 redirected to www.foo.com/folder3/folder3/file3.asp

I am getting html content of redirected page, But don't get redirected url and not able to check 3rd type of links

Using following code

Set ServerXmlHttp = Server.CreateObject("MSXML2.ServerXMLHTTP.6.0")
ServerXmlHttp.open "GET", "http://www.foo.com"
ServerXmlHttp.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
ServerXmlHttp.send PostData
If ServerXmlHttp.status = 200 Then
 //My CODE

Hope for quick response... or any other idea for link finder in asp , vb.net

Answer 1

It's out of the ServerXMLHTTP capabilities.
Instead you have to use IWinHttpRequest or another third-party component that able to manage redirects.
In the following example, req.Option(WHR_URL) returns the current url even if redirected.
Option req.option(WHR_EnableRedirects) is True by default like ServerXMLHTTP.
So, I've added a line commented out showing how to disable redirects.

Const WHR_URL = 1
Const WHR_EnableRedirects = 6
'Enum constants are listed at http://msdn.microsoft.com/en-us/library/windows/desktop/aa384108(v=vs.85).aspx
Dim req
Set req = Server.CreateObject("WinHttp.WinHttpRequest.5.1")
    'req.Option(WHR_EnableRedirects) = False 'don't follow the redirects
    req.open "GET", "http://www.foo.com", False
    req.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
    req.send PostData
If req.Status = 200 Then
    Response.Write "Last URL : " & req.Option(WHR_URL)
End If

Answer 2

I have been struggling too, but the solution is to verify if the final URL is equal to the start URL.

Calling getOption(-1) right after the 'send' will give you the final URL.

Then resend your request with this final URL.

Here my working example from SQL server using MSXML2.ServerXMLHTTP.3.0:

declare @err_code INT,@response_text VARCHAR(MAX),@oa_object INT, @status int
declare @url varchar(4000)='https://api.insocial.nl/v2/customers/'
declare @final_url varchar(4000)
DECLARE @result_table Table (result varchar(max))
 
EXEC @err_code=sp_OACreate 'MSXML2.ServerXMLHTTP.3.0', @oa_object OUT;
Exec @err_code=sp_OAMethod @oa_object, 'open', NULL, 'GET', @url,'false', null, null
Exec @err_code=sp_OAMethod @oa_object, 'setTimeouts',NULL,5000,5000,10000,1200000
Exec @err_code=sp_OAMethod @oa_object, 'setRequestHeader', NULL, 'X-AUTH-TOKEN', '***'
Exec @err_code=sp_OAMethod @oa_object, 'send', Null, null
 
Exec @err_code=sp_OAMethod @oa_object,'getOption(-1)',@final_url OUT
print @final_url
if (@final_url <> @url)
begin
    Exec @err_code=sp_OAMethod @oa_object, 'open', NULL, 'GET', @final_url,'false', null, null
    Exec @err_code=sp_OAMethod @oa_object, 'setTimeouts',NULL,5000,5000,10000,1200000
    Exec @err_code=sp_OAMethod @oa_object, 'setRequestHeader', NULL, 'X-AUTH-TOKEN', '***'
    Exec @err_code=sp_OAMethod @oa_object, 'send', Null, null
end
 
SET @response_text = null --make sure we don't return garbage
INSERT @result_table (result)
EXEC @err_code = sp_OAGetProperty @oa_object, 'responseText' 
SELECT * FROM @result_table
 
EXEC sp_OADestroy @oa_object