I'm working on an Xamarin Forms iOS app that streams the user's screen using the Broadcast Upload Extension via WebRTC. It works well in the most part, but I'm seeing unusual behaviour when the method doesn't have much processing to do, the ProcessSampleBuffer method is called less frequently. Here's the code:
public override void ProcessSampleBuffer(CoreMedia.CMSampleBuffer sampleBuffer, RPSampleBufferType sampleBufferType)
{
switch (sampleBufferType)
{
case RPSampleBufferType.Video:
// Handle audio sample buffer
var imageBuffer = sampleBuffer.GetImageBuffer() as CVPixelBuffer;
var orientation = GetOrientation(sampleBuffer);
var newOrientation = RTCVideoRotation.Rotation0;
if (orientation == InterfaceOrientationType.Portrait)
{
newOrientation = RTCVideoRotation.Rotation0;
}
else if (orientation == InterfaceOrientationType.LandscapeLeft)
{
newOrientation = RTCVideoRotation.Rotation180;
}
else if (orientation == InterfaceOrientationType.LandscapeRight)
{
newOrientation = RTCVideoRotation.Rotation270;
}
// UNCOMMENTING THE FOLLOWING 3 LINES MAKES THE METHOD FIRE ON A MORE REGULAR BASIS.
// var ciImage = CIImage.FromImageBuffer(imageBuffer);
// var image = new UIImage(ciImage, 1.0f, UIImageOrientation.Up);
// var jpeg = image.AsJPEG(0.05f);
long timeStampNs = (long)sampleBuffer.PresentationTimeStamp.Seconds * 1000000000;
if (WebRtcService2._videoCapturer != null && WebRtcService2._videoCapturer.Delegate != null)
{
WebRtcService2._videoCapturer.Delegate.Capturer(WebRtcService2._videoCapturer, ConvertToRTCVideoFrame(imageBuffer, timeStampNs, newOrientation));
}
break;
case RPSampleBufferType.AudioApp:
// Handle audio sample buffer for app audio
break;
case RPSampleBufferType.AudioMic:
// Handle audio sample buffer for app audio
break;
}
}
When the 3 commented out lines above are included to convert to a JPEG the method is called more regularly, multiple times per second. With the code as it is above, the method is fired in short bursts of 10-20 frames every minute or so.
Why would this be? I don't want to include the Jpeg code as it's impacting the performance (framerate) of the stream.
I expect a consistent framerate with the method being called multiple times per second (when the screen has something to redraw, of course) without the need to include the following lines of code:
var ciImage = CIImage.FromImageBuffer(imageBuffer);
var image = new UIImage(ciImage, 1.0f, UIImageOrientation.Up);
var jpeg = image.AsJPEG(0.05f);