Before diving into the SDK Version, i simply tried to stream an x264 encdoed video to VLC. Streaming MP3 Works, but x264 takes a while then i get an error, that vlc cannot open the file.
1) Downloaded Live555 Server
2) Started EXE which states the url to use as: rtsp://172.18.1.85/<filename>
3) In VLC i am using: rtsp://172.18.1.85/fantastic.264
As the server output says "Each files type is inferred from it's name suffix" so i renamed the file from fantastic.mp4
to fantastic.264
As i've stated an mp3 stream works fine in the same directory where fantastic.264 is placed.
The file ending was correct but it wasn't an "elementary h264 stream". So I had to reconvert it to an elementary h264 file.
Quick Fix:
You can get here more Information: What does Elementary Stream mean in Terms of H264