I've got my Xcode server setup on a mac mini. Downloading the .ipa from https://pandora.local/xcode/ using Itunes works just fine.
I tried installing it directly from the phone from safari. It installs a certificate but when I click install it gives an error, saying that it can't connect to the server.
For the mention issue, I got solved by going into iphone settings -> general - > About - > Scroll to the bottom , Click on Certificate Trust Settings -> trun on for Xcode Server Root Certificate Authority. See screenshot.