Jerky animation when scrolling image in WPF using SharpDX

Question

I am trying to smoothly scroll some images across a window using DirectX11 via SharpDX in a WPF application.

A bit of background:

The images are signal returns, varying by time, that have been loaded from a file and loaded into D3D as a texture array, and the image itself is rendered as a series of textured quads (a series of adjoining rectangles in a very long horizontal line). The viewport is set up so that the left and right edges represent time offsets, and the signal is displayed for the time period that falls between these times. The DirectX implementation is very simple; the quad vertices are generated once, and a very small per-frame buffer contains a simple 2D world-view transform that updates scale and translation according to the current visible range/zoom etc. As far as I can tell, the D3D implementation is not part of the problem I am having - it really is a very simple set up, and seems to be rendering extremely quickly (as it should), and although I do have some fancier stuff (streaming of textures from disk as required), I have disabled all of these and am running with very simple (small) textures whilst I try and resolve the issue I am having..

The problem:

Scrolling of the image is not smooth when the visible time range is animated. The image "jitters" much of the time, and frankly looks awful (when it is not jittering, it looks great).

The setup:

DirectX is rendered to a D3DImage, with a bit of work going on behind the scenes to make DX11 work - this code is taken from https://sharpdxwpf.codeplex.com/

There are multiple D3DImages (up to a total of 4), arranged in a grid (I have been testing with two, both of which contain signals for the same time period and are animated together).

These D3DImages are drawn by a DrawingVisual (which is hosted by a custom FrameworkElement) where they are used as the source of an ImageBrush. The frameworkelement object triggers a render as required, and the drawing visual handles the D3D render call and draws a rectangle to fill the control using the D3DImage brush.

The time range value is animated using a WPF DoubleAnimation. The visuals that are currently displayed are bound to this value via INotifyPropertyChanged and trigger a render (via InvalidateVisual) on each change.

DrawingVisual render code, triggered by change of "Position" value (start of visible time range):

// update scene per-frame buffer first, with world-view transform
using (DrawingContext dc = RenderOpen()
{
    _scene.Renderer.Render(_draw_args);
    _image.Invalidate();

    dc.DrawRectangle(_brush, null, new Rect(viewer.RenderSize));
}

What I have tried:

I have tried a number of things (and searched a lot) to try and determine whether the issue is due to jittery render request timing, or if the issue is further into the render process.

1. Hooking into CompositionTarget.Rendering Firstly driving the update of the position value via CompositionTarget.Rendering, and leaving the rest of the process as-is (i.e. elements react to the change of this value):

(needless to say, this is very much "test" code):

Stopwatch rsw;
long last_time = 0;
void play()
{
    rsw = new Stopwatch();
    last_time = 0;
    rsw.Start();
    CompositionTarget.Rendering += rendering;
}

void rendering(object sender, EventArgs e)
{
    double update_distance = rsw.ElapsedMilliseconds - last_time;
    Position += 10 * update_distance;
    last_time = rsw.ElapsedMilliseconds;
}

Results - worse than the simple double animation.

2. Using a DispatcherTimer. Similar to above, using a timer in conjunction with a stopwatch to judge the elapsed time a little better. Worse results again, with the interesting side note that CPU usage dropped from about 7% (half a core) to 0.7%.

The original attempt, along with variants 1 & 2, all try to update a single value that then triggers the render of interested parties. Out of interest, I logged the time differences between render requests as they reached the Drawing visual - although the DisptacherTimer actually gave the most consistent results there (both WPF animation and CompositionTarget dropped the odd frame), the DisptacherTimer also gave the most jittery animation.

3. Updating the image, but not invalidating the visual. Updating the source of an ImageBrush updates the displayed image, without having to re-render the DrawingVisual. This method produced very jittery results.

4. CompositionTarget.Rendering in the framework element itself. This produced the best results of the lot, but still not perfect. Jitter would happen, then dissipate, only to return again. In this approach, the framework element that holds the DV hooks up to CompositionTarget.Rendering, and the visual queries the current position, which is being animated independently. I could almost live with this approach.

5. Attempting the wait until the D3D scene is rendered, before invalidating the image (no discernible improvement):

   _scene.Renderer.Render(_draw_args);
   _scene.Renderer.Device.ImmediateContext.End(q);

   while (!(_scene.Renderer.Device.ImmediateContext.IsDataAvailable(q))) Thread.Yield();

   _image.Invalidate();

Observations:

1. I really don't think this is a performance issue as such. My dev machine has a good graphics card, 8 i7 cores etc, and this is a simple rendering operation.
2. This really seems like some sort of synchronisation issue between D3D and the WPF rendering, but I have no idea how to begin looking into this.
3. If I have two images animating in parallel, the jitter is much more pronounced on the first of the two (usually).
4. If I actively resize the window whilst animating, animation is perfectly smooth (despite the fact that a lot of extra work is being done, as the D3D context is being resized constantly).

EDIT

I've taken things back as far as possible to try and isolate the problem, and the issue seems to be fundamental to the way in which D3DImage is updated & rendered by WPF.

I have modified the WPFHost example in the SharpDX Samples solution so that the simple trigangle that is displayed is animated across the screen. This example is hosted in a DX10ImageSource that is rendered by a DPFCanvas on CompositionTarget.Rendering.

This example couldn't be more simple, and is about as "close to the metal" as you can get whilst rendering a D3DImage in WPF. A single triangle, translated across the screen by a value calculated from the time difference between renders. The stutter remains, coming and going, as if some sort of synchronisation issue. It baffles me, but essentially makes SharpDX unusable within WPF for any sort of smooth animations, which is extremely disappointing.

If anyone is interested in reproducing this problem, the SharpDX samples are available here: https://github.com/sharpdx/SharpDX-Samples

I made the following simple changes to the WPFHost example:

void IScene.Render()
{
...

        EffectMatrixVariable wv = this.SimpleEffect.GetVariableBySemantic("WorldView").AsMatrix();
        wv.SetMatrix(this.WorldViewMatrix);

...
}

void IScene.Update(TimeSpan sceneTime)
{
    float x = (float)sceneTime.Milliseconds * 0.001f - 0.5f;
    WorldViewMatrix = Matrix.Translation(x, 0, 0);
}

and in shader Simple.fx:

float4x4 WorldViewTransform : WorldView;

PS_IN VS( VS_IN input )
{
    PS_IN output = (PS_IN)0;

    output.pos = mul(input.pos, WorldViewTransform);
    output.col = input.col * Overlay;

    return output;
}

Answer 1

D3DImage is fundamentally broken.

If you don't need to overlay XAML elements on top of D3D or you do not need to resize too frequently the D3D surface, prefer to host a HWND/WinForm into your WPF apps and render to it directly using regular rendering loop. If you want a bit more details about the reasons, you can check this issue.