Getting connect timeout when sending request to www.fnde.gov.br from deployed app on Hetzner

Question

I hope all is well with you.

I need a solution to a time-out problem when web scraping a URL. The program is running in the cloud, at Hetzner.

This effect can be simulated with the curl command below:

curl https://www.fnde.gov.br/distribuicaosimadnet/confirmarCancelar

Failure Message:

curl https://www.fnde.gov.br/distribuicaosimadnet/confirmarCancelar curl: (28) Failed to connect to www.fnde.gov.br port 443 after 131364 ms: Connection timed out

The curious thing is that this access works correctly in our local machine and in AWS, both on Linux.

Another fact that draws attention is that the access below, via curl, can work on Hetzner:

curl https://www.fnde.gov.br/distribuicaosimadnet/selecionar?numeroEntidade=000001406302&anoPrograma=2023&codigoPrograma=01&ufSeleciona=AC&criterios=

The original application is written in Python, but I used curl as an alternative to simulate the problem.

Answer 1

It seems to me that www.fnde.gov.br site admin restricted the access of this website outside the country (Brazil here). This happens with many countries restricting their websites for access.

I assume the Hetzner VM is somewhere in Germany same as where I am. The traceroute command fails after few hops. You could verify the same from ping.eu

Nevertheless, traceroute works from perfops.net Brazil nodes. Their web application seems to be running when I checked with pingdom tools. Test results for you here.

As far as the timeouts are concerned, it may not work for you from Hetzner infrastructure as it's either based in Europe & North America only.

Another option could be using vpn/Cloudflare ZTNA connecting to a server located in Brazil

Answer 2

This is with very high likelyhood the website blocking you from outside brazil.

If this isnt fixable any other way you can click a cheap VPN like https://mullvad.net/en and install it on the server. I would go with their wireguard option. Make sure to not set the default route via the VPN or your server might become unaccessible. Here is an example config:

[Interface]
# you need this
PrivateKey = secret-private-key-here
# and this - you can remove the ipv6 address or leave it in, doesnt rly matter if you only use it to access one server
Address = your-private-internal-vpn-ipv4-here,your-private-internal-ipv6-here
# REMOVE THIS!!
#DNS = 100.61.0.24
# REMOVE THIS TOO (if its in your config)
#PostUp = iptables -I OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT && ip6tables -I OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT
#PreDown = iptables -D OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT && ip6tables -D OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT

[Peer]
# You need this
PublicKey = mullvad-wireguard-public-key-here
# CHANGE THIS
AllowedIPs = 0.0.0.0/0,::0/0
# This would set your default route (route to 0.0.0.0/0) to the wireguard server. You don't want to use it by default, only for specific hosts:
# Here you only define the IP you want to access. For stackoverflow.com:
# user@host:~$ host -t A stackoverflow.com
# stackoverflow.com has address 172.64.155.249
# stackoverflow.com has address 104.18.32.7
# you would write:
AllowedIPs = 172.64.155.249/32,104.18.32.7/32

Installing that:

sudo apt install wireguard-tools
sudo editor /etc/wireguard/mullvad.conf

Then to test if it works:

sudo apt install tmux
tmux
# then inside the tmux
sudo systemctl start [email protected]; sleep 10; sudo systemctl stop [email protected]

If you dont loose the connection to the server you're all set. If you do it will be back after 10 seconds. If it goes offline, paste your VPN config here (but redact private information!). You can then enable the wireguard service to start on (re)boot:

sudo systemctl enable [email protected]

Now your server will go over the mullvad VPN if it is accessing the IP you defined.