Azure Speech SDK: createPushStream from GetUserMedia in javascript

Question

I am trying to find examples on how to use getUserMedia stream object to createPushStream with the Azure Speech SDK. Note that I intend to run the code in Safari, so the use of MediaRecorder is not possible. The intent is to use getUserMedia stream to feed both the SpeechSDK.SpeechRecognizer (SpeechSDK.AudioConfig.fromStreamInput) and to save stream as an audio file. SpeechSDK.AudioConfig.fromDefaultMicrophoneInput does not allow that.

Answer 1

One way to do this would be to grab the audio data sent to the service via the connection.messageSent callback. Build the recognizer in the usual way (you should be able to use fromDefaultMicrophoneInput), then add this callback:

    const con: sdk.Connection = sdk.Connection.fromRecognizer(r);

    let wavFragmentCount: number = 0;

    const wavFragments: { [id: number]: ArrayBuffer; } = {};

    con.messageSent = (args: sdk.ConnectionMessageEventArgs): void => {
        if (args.message.path === "audio" && args.message.isBinaryMessage && args.message.binaryMessage !== null) {
            wavFragments[wavFragmentCount++] = args.message.binaryMessage;
        }
    };

Once recognition is completed, wavFragments has the audio data captured, which you can write to a file.

(This example code was taken from ConnectionTest.ts )