how do I Integrate Watson text to speech with speech to text in unity

Question

I'm building an AR CV app in unity using the watson SDK. I'm a complete noob but I've managed to follow the videos and create something kinda cool.

The idea is that it will give the candidate a more interesting way to describe themselves than a sheet of paper. my problem is that while I've managed to get speech to text streaming done I don't know what my next steps are. It's for a university project but my tutor doesn't know either. Also if TAJ reads this thank you so much for those youtube videos!

my question is how do I add text to speech and assistant?

Answer 1

The basic idea here is that you will use the Watson Unity SDK services to bring speech via the microphone and convert it to text. You shouldn't send this text back to text to speech since it's what you just input (unless that's what you wanted). This text can be used in many ways. One way would be to use the Watson Assistant service and create a kind of script that you can use in natural language. The output of the message method is text that you could feed into Watson Text to Speech resulting in an audio file that could be played back. Essentially from the StreamingExample

private void OnRecognize(SpeechRecognitionEvent result, Dictionary<string, object> customData)
{
    if (result != null && result.results.Length > 0)
    {
        foreach (var res in result.results)
        {
            foreach (var alt in res.alternatives)
            {
                // Is final for the utternace?
                if (res.final)
                {
                    MessageRequest messageRequest = new MessageRequest()
                    {
                        Input = new MessageInput()
                        {
                            Text = alt.transcript
                        }
                    };
                    // Send the text to Assistant
                    assistant.Messsage(OnMessage, OnFail, assistantId, sessionId, messageRequest);
                }
            }
        }
    }
}

private void OnMessage(MessageResponse response, Dictionary<string, object> customData)
{
    // Send Assistant output to TextToSpeech
    textToSpeech.ToSpeech(OnSynthesize, OnFail, response.output.generic[0].text, true)
}

private void OnSynthesize(AudioClip clip, Dictionary<string, object> customData)
{
    // Play the clip from TextToSpeech
    PlayClip(clip);
}

private void PlayClip(AudioClip clip)
{
    if (Application.isPlaying && clip != null)
    {
        GameObject audioObject = new GameObject("AudioObject");
        AudioSource source = audioObject.AddComponent<AudioSource>();
        source.spatialBlend = 0.0f;
        source.loop = false;
        source.clip = clip;
        source.Play();

        Destroy(audioObject, clip.length);
    }
}

You will need to properly instantiate and authenticate the services.