ESP transport mode is incompatible with NAT (not NAPT or PAT)
I saw on many papers that because NAT device should calculate TCP checksom so transport mode wouldn't work with NAT.
the question is - how the NAT device can differ between Transport mode or Tunnel mode given that next-header in ESP is encrypted.
The short answer is you wouldn't. Even though TCP/UDP traffic might not be able to pass through in transport mode, other traffic might be OK. I believe as a NAT box you'd do your best to get all ESP traffic to go where it belongs, based on the SPI.
You should read RFC 3715 for more details.