My local machine is using MacOSX/linux and am able to connect to a Windows Remote Desktop machine (using Windows Remote Desktop/Remote Desktop Connection).
The Windows Remote Desktop machine is able to connect to certain servers that I cannot connect to locally on my MAC. For example, connection to database servers, specific web servers, etc.
How can I setup some kind of tunnel so that I can access the servers locally on my mac?
You can configure port forwarding on the Windows machine
where 192.168.1.111 is IP of your Windows machine and 192.168.0.33 is destination server. From your Mac you will connect to Windows machine specifying port in RDP connection 192.168.1.111:3390 and you will be connected to the remote server. You have to provide credentials from the server, not your local Win machine. This configuration survive restarting Windows machine.