Could anyone explain what would be the best approach for this scenario?: There are 2 separate Service Providers:
System A is a server that works as a rest API for a mobile application.
System B is a website which login is handled with simple and plain cookies.
I have read a lot and Ive found that the most used protocols in use right now for these kind of systems are SAML and OAuth, where OAuth works specially to give security to API services and SAML for web applications (cookie based).
Ideally, what is being looked for is a single log in, which is right now handled for System B through cookies in the main domain.
Is there a way to use both SAML and OAuth? Or What would be a good way to go?
The answer depends on what IDP you currently have.
Is the IDP SAML based or is that just what you got from researching?
SAML and OAuth don't really play well together.
The ideal would be OpenID Connect where you could use the OpenID Connect token to get a REST API token. But that would involve using the same IDP for authentication for both.