I'm implementing WebRTC Perfect Negotiation in my Vue 2 application. The app will have multiple viewers and a single streamer.
After a lot of logging and debugging, I've resolved some of the problems that I was having. I removed the TURN server in the iceServers configuration, and that allowed the ICE Candidate gathering to finish. Previously it was stuck at "gathering". Now, the two peers have exchanged local/remote descriptions and added ICE candidates, but there still is not a change in the connectionState.
Here is my RTCPeerConnection object:
RTCPeerConnection
canTrickleIceCandidates: true
connectionState: "new"
currentLocalDescription: RTCSessionDescription {type: 'offer', sdp: 'v=0\r\no=- 4764627134364341061 2 IN IP4 127.0.0.1\r\ns…4754 label:f12fee59-268c-4bc3-88c1-8ac27aec8a9c\r\n'}
currentRemoteDescription: RTCSessionDescription {type: 'answer', sdp: 'v=0\r\no=- 3069477756847576830 2 IN IP4 127.0.0.1\r\ns…Nd1pO\r\na=ssrc:1149065622 cname:VquHLgyd/d3Nd1pO\r\n'}
iceConnectionState: "new"
iceGatheringState: "complete"
localDescription: RTCSessionDescription {type: 'offer', sdp: 'v=0\r\no=- 4764627134364341061 2 IN IP4 127.0.0.1\r\ns…4754 label:f12fee59-268c-4bc3-88c1-8ac27aec8a9c\r\n'}
onaddstream: null
onconnectionstatechange: ƒ (e)
ondatachannel: null
onicecandidate: ƒ (_ref)
onicecandidateerror: ƒ (e)
oniceconnectionstatechange: ƒ ()
onicegatheringstatechange: ƒ (e)
onnegotiationneeded: ƒ ()
onremovestream: null
onsignalingstatechange: null
ontrack: ƒ (_ref3)
pendingLocalDescription: null
pendingRemoteDescription: null
remoteDescription: RTCSessionDescription {type: 'answer', sdp: 'v=0\r\no=- 3069477756847576830 2 IN IP4 127.0.0.1\r\ns…Nd1pO\r\na=ssrc:1149065622 cname:VquHLgyd/d3Nd1pO\r\n'}
sctp: null
signalingState: "stable"
[[Prototype]]: RTCPeerConnection
Here is LiveStream.vue:
<template>
<div>
<main>
<div>
<div id="video-container">
<h2>LiveStream</h2>
<video id="local-video" ref="localVideo" autoplay="true"></video>
</div>
</div>
</main>
<aside>
<div>
<div>
<p>ViewStream</p>
<div v-for="(item, key) in participants" :key="key">
<Video :videoId="key" :videoStream="participants[key].peerStream" />
</div>
<div></div>
</div>
</div>
</aside>
</div>
</template>
<script>
import { videoConfiguration } from "../mixins/WebRTC";
import Video from "../components/Video.vue";
export default {
name: "LiveStream",
components: {
Video,
},
data() {
return {
participants: {},
localStream: null,
pc: null,
roomInfo: {
room: undefined,
username: "testUser",
},
constraints: {
video: {
width: 450,
height: 348,
},
},
};
},
mixins: [videoConfiguration],
methods: {
async initializeWebRTC(user, desc) {
console.log("initializeWebRTC called", { user, desc });
this.participants[user] = {
...this.participants[user],
pc: this.setupRTCPeerConnection(
new RTCPeerConnection(this.configuration),
user,
this.roomInfo.username,
this.roomInfo.room
),
peerStream: null,
peerVideo: null,
};
for (const track of this.localStream.getTracks()) {
this.participants[user].pc.addTrack(track, this.localStream);
console.log("local track added", track);
}
this.createOffer(
this.participants[user].pc,
user,
this.roomInfo.room,
true
);
this.onIceCandidates(
this.participants[user].pc,
user,
this.roomInfo.room,
true
);
},
createPeerConnection() {
this.pc = new RTCPeerConnection(this.configuration);
},
},
created() {
this.roomInfo.room = this.getRoomName();
},
async mounted() {
this.myVideo = document.getElementById("local-video");
await this.getUserMedia();
await this.getAudioVideo();
this.$socket.client.emit("joinRoom", {
...this.roomInfo,
creator: true,
});
},
beforeDestroy() {
this.pc.close();
this.pc = null;
this.$socket.$unsubscribe("newParticipant");
this.$socket.$unsubscribe("onMessage");
this.$socket.client.emit("leaveRoom", {
to: this.to,
from: this.username,
room: this.roomInfo.room,
});
},
sockets: {
connect() {
console.log("connected socket");
},
newParticipant(userObject) {
if (userObject.username === this.roomInfo.username) return;
this.$set(this.participants, userObject.username, {
user: userObject.username,
});
this.initializeWebRTC(userObject.username);
},
async onMessage({ desc, from, room, candidate }) {
if (from === this.username) return;
try {
if (desc) {
const offerCollision =
desc.type === "offer" &&
(this.makingOffer ||
this.participants[from].pc.signalingState !== "stable");
this.ignoreOffer = !this.isPolitePeer && offerCollision;
if (this.ignoreOffer) {
return;
}
if (desc.type === "offer") {
this.handleAnswer(desc, this.participants[from].pc, from, room);
} else {
this.addRemoteTrack(this.participants[from], from);
await this.setRemoteDescription(desc, this.participants[from].pc);
}
} else if (candidate) {
try {
await this.addCandidate(
this.participants[from].pc,
candidate.candidate
);
} catch (err) {
if (!this.ignoreOffer) {
throw err;
}
}
}
} catch (err) {
console.error(err);
}
},
},
};
</script>
Here is the mixin I created to handle a lot of the connection functionality:
export const videoConfiguration = {
data() {
return {
// Media config
constraints: {
audio: {
echoCancellation: true,
noiseSuppression: true,
autoGainControl: false
},
video: {
width: 400,
height: 250
}
},
configuration: {
iceServers: [
{
urls: [
"stun:stun1.l.google.com:19302",
"stun:stun2.l.google.com:19302"
]
}
]
},
offerOptions: {
offerToReceiveAudio: 1,
offerToReceiveVideo: 1
},
myVideo: null,
localStream: null,
username: null,
isPolitePeer: false,
makingOffer: false,
ignoreOffer: false
};
},
async created() {
this.username = await this.getUsername();
},
beforeDestroy() {
this.localStream.getTracks().forEach((track) => track.stop());
},
methods: {
/**
* Get permission to read from user's microphone and camera.
* Returns audio and video streams to be added to video element
*/
async getUserMedia() {
if ("mediaDevices" in navigator) {
try {
const stream = await navigator.mediaDevices.getUserMedia(
this.constraints
);
if ("srcObject" in this.myVideo) {
this.myVideo.srcObject = stream;
this.myVideo.volume = 0;
} else {
this.myVideo.src = stream;
}
this.localStream = stream;
} catch (error) {
// eslint-disable-next-line no-console
console.error(error);
}
}
},
getAudioVideo() {
const video = this.localStream.getVideoTracks();
// eslint-disable-next-line no-console
console.log(video);
const audio = this.localStream.getAudioTracks();
// eslint-disable-next-line no-console
console.log(audio);
},
async setRemoteDescription(remoteDesc, pc) {
try {
await pc.setRemoteDescription(remoteDesc);
} catch (error) {
// eslint-disable-next-line no-console
console.error(error);
}
},
addCandidate(pc, candidate) {
try {
const rtcIceCandidate = new RTCIceCandidate(candidate);
pc.addIceCandidate(rtcIceCandidate);
console.log(`${this.username} added a candidate`);
} catch (error) {
console.error(
`Error adding a candidate in ${this.username}. Error: ${error}`
);
}
},
onIceCandidates(pc, to, room) {
pc.onicecandidate = ({ candidate }) => {
if (!candidate) return;
this.$socket.client.emit("new-ice-candidate", {
candidate,
to: to,
from: this.username,
room: room
});
};
},
async createOffer(pc, to, room) {
console.log(`${this.roomInfo.username} wants to start a call with ${to}`);
pc.onnegotiationneeded = async () => {
try {
this.makingOffer = true;
await pc.setLocalDescription();
this.sendSignalingMessage(pc.localDescription, true, to, room);
} catch (err) {
console.error(err);
} finally {
this.makingOffer = false;
}
};
},
async createAnswer(pc, to, room) {
try {
const answer = await pc.createAnswer();
await pc.setLocalDescription(answer);
this.sendSignalingMessage(pc.localDescription, false, to, room);
} catch (error) {
console.error(error);
}
},
async handleAnswer(desc, pc, from, room) {
await this.setRemoteDescription(desc, pc);
this.createAnswer(pc, from, room);
},
sendSignalingMessage(desc, offer, to, room) {
const isOffer = offer ? "offer" : "answer";
// Send the offer to the other peer
if (isOffer === "offer") {
this.$socket.client.emit("offer", {
desc: desc,
to: to,
from: this.username,
room: room,
offer: isOffer
});
} else {
this.$socket.client.emit("answer", {
desc: desc,
to: to,
from: this.username,
room: room,
offer: isOffer
});
}
},
addRemoteTrack(user, video) {
user.peerVideo = user.peerVideo || document.getElementById(video);
user.pc.ontrack = ({ track, streams }) => {
user.peerStream = streams[0];
track.onunmute = () => {
if (user.peerVideo.srcObject) {
return;
}
user.peerVideo.srcObject = streams[0];
};
};
},
/**
* Using handleRemoteTrack temporarily to add the tracks to the RTCPeerConnection
* for ViewStream since the location of pc is different.
* @param {*} user
*/
handleRemoteTrack(pc, user) {
this.peerVideo = document.getElementById(user);
pc.ontrack = ({ track, streams }) => {
this.peerStream = streams[0];
track.onunmute = () => {
if (this.peerVideo.srcObject) {
return;
}
this.peerVideo.srcObject = streams[0];
};
};
},
setupRTCPeerConnection(pc) {
pc.onconnectionstatechange = (e) => {
console.log(
"WebRTC: Signaling State Updated: ",
e.target.signalingState
);
};
pc.oniceconnectionstatechange = () => {
console.log("WebRTC: ICE Connection State Updated");
};
pc.onicegatheringstatechange = (e) => {
console.log(
"WebRTC: ICE Gathering State Updated: ",
e.target.iceGatheringState
);
};
pc.onicecandidateerror = (e) => {
if (e.errorCode === 701) {
console.log("ICE Candidate Error: ", e);
}
};
return pc;
}
}
};
I created a CodeSandbox that has the ViewStream.vue file and the directory structure for how I'm trying to set it up. (It's just too much code to post here.)
When the viewer joins the room created by the streamer, I can see that they exchange offer/answer and ice candidates. However, I still do not see any change in the
connectionStateoriceConnectionState. Is there a piece that I'm not doing?One thing I noticed when logging data and digging through chrome://webrtc-internals/ is that the
MediaStreamID's don't match. I log out the tracks after the call togetUserMedia(), and note the track ID's.
This image shows the stream IDs for the caller (top) and the callee (bottom)
I then log when I'm adding the local tracks to the RTCPeerConnection, and they match what was generated for both peers.
Here, the tracks for the streamer are added to the RTCPeerConnection. The IDs match from above.
However, I'm also logging for each peer when I receive a remote track, and that's when the ID's don't match.
I don't know what is generating the ID in this picture. It's different from the ID of the callee in the first picture.
Is that normal behavior? Would the fact that the IDs don't match be the cause of the streams not starting on either end? I don't know what would cause this. The IDs are the same when added to the RTCPeerConnection on either end of the call.
Edit 5/1: I removed the TURN server from my config, and that fixed part of the connection process. Still having a problem getting media to flow between peers. But I can see that I've captured a MediaStream on each side of the connection.
Can I have the Server side code? So that I can recreate whats going on?
Edit:
If you are talking about this in your second question.
I guess it just explains how the collision between a polite and impolite peer is resolved by polite peer becoming callee from caller.
And that is just the definition of
Polite Peerand I thing you don't need to worry much about explicitly changing the roles in this context.A polite peer, essentially, is one which may send out offers, but then responds if an offer arrives from the other peer with "Okay, never mind, drop my offer and I'll consider yours instead.~ MDN DocsI hope this answers your Second Question