Troubleshooting Windows Preview Handler Issues: GDI+ and Direct2D Integration

Question

I'm trying to create a Windows preview handler for a file type (.abcd in the following) that draws in the preview window using GDI+ and Direct2D based on the contents of the selected file. I used the example given by Microsoft, explained here and with the code example given here. While this works in principle (see the reduced example below), I encountered the problem that the preview is only shown for every other clicked .abcd file; in the other cases the error "This file can't be previewed" appears. This is not the case in Microsoft's example and is probably related to loading/unloading resources.

Screenshots of the behavior of the preview handler

I rely on Direct2D to efficiently create a bitmap figure from the contents of the file, while GDI+ can be omitted and replaced with Direct2D components in the preview if needed.

A stripped down but still comprehensive code example is given below. It consists of the two files dll.cpp and main.cpp, where dll.cpp handles the registration of the preview handler. Using Visual Studio 2022, the example can be executed as follows:

1. Get the sample code provided here.
2. Replace RecipePreviewHandler.cpp with main.cpp given below.
3. Replace the code in dll.cpp with the code below.
4. Build the solution.
5. Register the preview handler for .abcd files by running regsvr32.exe RecipePreviewHandler.dll in Powershell from the build folder and restart the 'explorer' process.
6. Create (two) .abcd files, click on them, and observe the preview pane behavior described above.
7. The preview handler can be unregistered by running regsvr32.exe /u RecipePreviewHandler.dll in Powershell and restarting the 'prevhost' process.

In trying to fix the problem, I found that removing PostQuitMessage(0); eliminates the problem described above, but introduces a lot of artifacts in the preview when resizing the preview pane and when switching between .abcd files, where previews of multiple files overlay each other (not for this example, but for more complex and changing drawings). Since I really don't know what to do now, how can this be solved?

main.cpp:

#define _CRT_SECURE_NO_WARNINGS
#include <shlwapi.h>
#include <shobjidl.h>
#include <gdiplus.h>
#include <new>
#include <windows.h>
#include <commctrl.h>
#include <objidl.h>
#include <iostream>
#include <string>
#include <sstream>
#include <stdio.h>
#include <cstdio>
#include <d2d1.h>
using namespace Gdiplus;
using namespace std;
#pragma comment(lib,"Gdiplus.lib")
#pragma comment(lib,"d2d1.lib")
#pragma comment(lib,"Comctl32.lib")
#pragma comment(lib,"shlwapi.lib")

// Utility function to safely release COM pointers
template <class T> void SafeRelease(T** ppT)
{
    if (*ppT)
    {
        (*ppT)->Release();
        *ppT = NULL;
    }
} // SafeRelease

inline int RECTWIDTH(const RECT& rc)
{
    return (rc.right - rc.left);
}

inline int RECTHEIGHT(const RECT& rc)
{
    return (rc.bottom - rc.top);
}

class CABCDPreviewHandler : public IObjectWithSite,
    public IPreviewHandler,
    public IOleWindow,
    public IInitializeWithFile
{

public:
    GdiplusStartupInput gdiplusStartupInput;
    ULONG_PTR gdiplusToken;

    ID2D1Factory* pFactory = nullptr;
    ID2D1HwndRenderTarget* pRenderTarget = nullptr;

    // Constructor
    CABCDPreviewHandler() : _cRef(1), _hwndParent(NULL), _hwndPreview(NULL), _punkSite(NULL)
    {
        // Initialize GDI+
        GdiplusStartup(&gdiplusToken, &gdiplusStartupInput, NULL);

        // Initialize _rcParent to an initial value (for example, zero size rectangle)
        _rcParent = RECT{ 0, 0, 0, 0 };

        // Initialize Direct2D
        D2D1CreateFactory(D2D1_FACTORY_TYPE_SINGLE_THREADED, &pFactory);
    }

    // Destructor
    virtual ~CABCDPreviewHandler()
    {
        // Shutdown GDI+
        GdiplusShutdown(gdiplusToken);

        if (_hwndPreview)
        {
            DestroyWindow(_hwndPreview);
        }
        SafeRelease(&_punkSite);
        SafeRelease(&pRenderTarget);
        SafeRelease(&pFactory);
    }

    // IUnknown
    IFACEMETHODIMP QueryInterface(REFIID riid, void** ppv)
    {
        *ppv = NULL;
        static const QITAB qit[] =
        {
            QITABENT(CABCDPreviewHandler, IObjectWithSite),
            QITABENT(CABCDPreviewHandler, IOleWindow),
            QITABENT(CABCDPreviewHandler, IInitializeWithFile),
            QITABENT(CABCDPreviewHandler, IPreviewHandler),
            { 0 },
        };
        return QISearch(this, qit, riid, ppv);
    }

    IFACEMETHODIMP_(ULONG) AddRef()
    {
        return InterlockedIncrement(&_cRef);
    }

    IFACEMETHODIMP_(ULONG) Release()
    {
        ULONG cRef = InterlockedDecrement(&_cRef);
        if (!cRef)
        {
            delete this;
        }
        return cRef;
    }

    // IObjectWithSite
    IFACEMETHODIMP SetSite(IUnknown* punkSite);
    IFACEMETHODIMP GetSite(REFIID riid, void** ppv);

    // IPreviewHandler
    IFACEMETHODIMP SetWindow(HWND hwnd, const RECT* prc);
    IFACEMETHODIMP SetFocus();
    IFACEMETHODIMP QueryFocus(HWND* phwnd);
    IFACEMETHODIMP TranslateAccelerator(MSG* pmsg);
    IFACEMETHODIMP SetRect(const RECT* prc);
    IFACEMETHODIMP DoPreview();
    IFACEMETHODIMP Unload();

    // IOleWindow
    IFACEMETHODIMP GetWindow(HWND* phwnd);
    IFACEMETHODIMP ContextSensitiveHelp(BOOL fEnterMode);

    // IInitializeWithFile
    IFACEMETHODIMP Initialize(LPCWSTR pszFilePath, DWORD);

    // Custom
    LRESULT CALLBACK PreviewWindowSubclassProc(HWND hwnd, UINT uMsg, WPARAM wParam, LPARAM lParam, UINT_PTR uIdSubclass, DWORD_PTR dwRefData);
    static LRESULT CALLBACK PreviewWindowStaticSubclassProc(HWND hwnd, UINT uMsg, WPARAM wParam, LPARAM lParam, UINT_PTR uIdSubclass, DWORD_PTR dwRefData);

private:
    HRESULT _CreatePreviewWindow();

    long        _cRef;            // Reference count of this object
    HWND        _hwndParent;      // parent window that hosts the previewer window; do NOT DestroyWindow this
    RECT        _rcParent;        // bounding rect of the parent window
    HWND        _hwndPreview;     // the actual previewer window
    IUnknown*   _punkSite;        // site pointer from host
};

// IPreviewHandler
// This method gets called when the previewer gets created
HRESULT CABCDPreviewHandler::SetWindow(HWND hwnd, const RECT* prc)
{
    if (hwnd && prc)
    {
        _hwndParent = hwnd; // cache the HWND for later use
        _rcParent = *prc; // cache the RECT for later use

        if (_hwndPreview)
        {
            // Update preview window parent and rect information
            SetParent(_hwndPreview, _hwndParent);
            SetWindowPos(_hwndPreview, NULL, _rcParent.left, _rcParent.top,
                RECTWIDTH(_rcParent), RECTHEIGHT(_rcParent), SWP_NOMOVE | SWP_NOZORDER | SWP_NOACTIVATE);
        }
    }
    return S_OK;
} // SetWindow

HRESULT CABCDPreviewHandler::SetFocus()
{
    HRESULT hr = S_FALSE;
    if (_hwndPreview)
    {
        ::SetFocus(_hwndPreview);
        hr = S_OK;
    }
    return hr;
} // SetFocus

HRESULT CABCDPreviewHandler::QueryFocus(HWND* phwnd)
{
    HRESULT hr = E_INVALIDARG;
    if (phwnd)
    {
        *phwnd = ::GetFocus();
        if (*phwnd)
        {
            hr = S_OK;
        }
        else
        {
            hr = HRESULT_FROM_WIN32(GetLastError());
        }
    }
    return hr;
} // QueryFocus

HRESULT CABCDPreviewHandler::TranslateAccelerator(MSG* pmsg)
{
    HRESULT hr = S_FALSE;
    IPreviewHandlerFrame* pFrame = NULL;
    if (_punkSite && SUCCEEDED(_punkSite->QueryInterface(&pFrame)))
    {
        hr = pFrame->TranslateAccelerator(pmsg);
        SafeRelease(&pFrame);
    }
    return hr;
} // TranslateAccelerator

// This method gets called when the size of the previewer window changes (user resizes the Reading Pane)
HRESULT CABCDPreviewHandler::SetRect(const RECT* prc)
{
    HRESULT hr = E_INVALIDARG;
    if (prc)
    {
        _rcParent = *prc;
        if (_hwndPreview)
        {
            // Preview window is already created, so set its size and position
            SetWindowPos(_hwndPreview, NULL, _rcParent.left, _rcParent.top,
                RECTWIDTH(_rcParent), RECTHEIGHT(_rcParent), SWP_NOMOVE | SWP_NOZORDER | SWP_NOACTIVATE);
        }
        hr = S_OK;
    }

    return hr;
} // SetRect

// The main method that renders graphics
HRESULT CABCDPreviewHandler::DoPreview()
{
    HRESULT hr = E_FAIL;
    if (_hwndPreview == NULL)
    {
        hr = _CreatePreviewWindow();
    }
    return hr;
} // DoPreview

// This method gets called when a shell item is de-selected in the listview
HRESULT CABCDPreviewHandler::Unload()
{
    if (_hwndPreview)
    {
        DestroyWindow(_hwndPreview);
        _hwndPreview = NULL;
    }
    return S_OK;
} // Unload

// IObjectWithSite methods
HRESULT CABCDPreviewHandler::SetSite(IUnknown* punkSite)
{
    SafeRelease(&_punkSite);
    return punkSite ? punkSite->QueryInterface(&_punkSite) : S_OK;
} // SetSite

HRESULT CABCDPreviewHandler::GetSite(REFIID riid, void** ppv)
{
    *ppv = NULL;
    return _punkSite ? _punkSite->QueryInterface(riid, ppv) : E_FAIL;
} // GetSite

// IOleWindow methods
HRESULT CABCDPreviewHandler::GetWindow(HWND* phwnd)
{
    HRESULT hr = E_INVALIDARG;
    if (phwnd)
    {
        *phwnd = _hwndParent;
        hr = S_OK;
    }
    return hr;
} // GetWindow

HRESULT CABCDPreviewHandler::ContextSensitiveHelp(BOOL)
{
    return E_NOTIMPL;
}

// IInitializeWithFile methods
// This method gets called when an item gets selected in listview
HRESULT CABCDPreviewHandler::Initialize(LPCWSTR pszFilePath, DWORD)
{
    return S_OK;
} // Initialize

LRESULT CALLBACK CABCDPreviewHandler::PreviewWindowStaticSubclassProc(HWND hwnd, UINT uMsg, WPARAM wParam, LPARAM lParam, UINT_PTR uIdSubclass, DWORD_PTR dwRefData)
{
    CABCDPreviewHandler* pThis = reinterpret_cast<CABCDPreviewHandler*>(dwRefData);

    // Forward the call to the member function
    return pThis->PreviewWindowSubclassProc(hwnd, uMsg, wParam, lParam, uIdSubclass, dwRefData);
}

LRESULT CALLBACK CABCDPreviewHandler::PreviewWindowSubclassProc(HWND hwnd, UINT uMsg, WPARAM wParam, LPARAM lParam, UINT_PTR /* uIdSubclass */, DWORD_PTR dwRefData)
{
    CABCDPreviewHandler* pThis = reinterpret_cast<CABCDPreviewHandler*>(dwRefData);

    switch (uMsg)
    {
    case WM_SIZE:
        if (pRenderTarget)
        {
            int windowWidth = LOWORD(lParam);
            int windowHeight = HIWORD(lParam);
            pRenderTarget->Resize(D2D1::SizeU(windowWidth, windowHeight));
        }
        return 0;
    case WM_PAINT:
    {
        if (pRenderTarget)
        {
            // Begin drawing with Direct2D
            pRenderTarget->BeginDraw();

            // Draw the white background
            D2D1_COLOR_F clearColor = D2D1::ColorF(1.0f, 1.0f, 1.0f); // White color
            pRenderTarget->Clear(clearColor);

            pRenderTarget->EndDraw();
            // End drawing with Direct2D

            // Begin drawing with GDI+
            PAINTSTRUCT ps;
            HDC hdcPaint = BeginPaint(hwnd, &ps);

            // Use GDI+ to render the red circle
            Gdiplus::Graphics g(hdcPaint);
            Gdiplus::SolidBrush brush(Gdiplus::Color::Red);
            int radius = 50;
            int centerX = pRenderTarget->GetSize().width / 2;
            int centerY = pRenderTarget->GetSize().height / 2;
            g.FillEllipse(&brush, centerX - radius, centerY - radius, 2 * radius, 2 * radius);

            EndPaint(hwnd, &ps);
            // End drawing with GDI+
        }
    }
    return 0;
    case WM_DESTROY:
        PostQuitMessage(0);
        return 0;
    default:
        return DefWindowProc(hwnd, uMsg, wParam, lParam);
    }
} // PreviewWindowSubclassProc

HRESULT CABCDPreviewHandler::_CreatePreviewWindow()
{
    // Create the preview window
    _hwndPreview = CreateWindowExW(0, L"STATIC", NULL,
        WS_CHILD | WS_VISIBLE,
        _rcParent.left, _rcParent.top, RECTWIDTH(_rcParent), RECTHEIGHT(_rcParent),
        _hwndParent, NULL, NULL, this); // Pass the instance pointer as lParam

    if (_hwndPreview)
    {
        // Set the static subclass procedure
        SetWindowSubclass(_hwndPreview, PreviewWindowStaticSubclassProc, 0, reinterpret_cast<DWORD_PTR>(this));

        RECT rc;
        GetClientRect(_hwndPreview, &rc);

        // Create render target
        pFactory->CreateHwndRenderTarget(
            D2D1::RenderTargetProperties(),
            D2D1::HwndRenderTargetProperties(_hwndPreview, D2D1::SizeU(rc.right, rc.bottom)),
            &pRenderTarget);

        ShowWindow(_hwndPreview, SW_SHOW);
        UpdateWindow(_hwndPreview);

        return S_OK;
    }

    return E_FAIL;
} // _CreatePreviewWindow

HRESULT CABCDPreviewHandler_CreateInstance(REFIID riid, void** ppv)
{
    *ppv = NULL;

    CABCDPreviewHandler* pNew = new (std::nothrow) CABCDPreviewHandler();
    HRESULT hr = pNew ? S_OK : E_OUTOFMEMORY;
    if (SUCCEEDED(hr))
    {
        hr = pNew->QueryInterface(riid, ppv);
        pNew->Release();
    }
    return hr;
}

dll.cpp:

#include <objbase.h>
#include <shlwapi.h>
#include <new>

extern HRESULT CABCDPreviewHandler_CreateInstance(REFIID riid, void** ppv);

#define SZ_CLSID_ABCDPreviewHandler     L"{B9197DE3-9813-494E-978C-08AA1973BD4A}"
#define SZ_ABCDPREVIEWHANDLER           L"ABCD Preview Handler"

const CLSID CLSID_ABCDPreviewHandler = { 0xb9197de3, 0x9813, 0x494e, { 0x97, 0x8c, 0x8, 0xaa, 0x19, 0x73, 0xbd, 0x4a } };

typedef HRESULT(*PFNCREATEINSTANCE)(REFIID riid, void** ppvObject);
struct CLASS_OBJECT_INIT
{
    const CLSID* pClsid;
    PFNCREATEINSTANCE pfnCreate;
};

// add classes supported by this module here
const CLASS_OBJECT_INIT c_rgClassObjectInit[] =
{
    { &CLSID_ABCDPreviewHandler, CABCDPreviewHandler_CreateInstance }
};

long g_cRefModule = 0;

// Handle the the DLL's module
HINSTANCE g_hInst = NULL;

// Standard DLL functions
STDAPI_(BOOL) DllMain(HINSTANCE hInstance, DWORD dwReason, void*)
{
    if (dwReason == DLL_PROCESS_ATTACH)
    {
        g_hInst = hInstance;
        DisableThreadLibraryCalls(hInstance);
    }
    return TRUE;
}

STDAPI DllCanUnloadNow()
{
    // Only allow the DLL to be unloaded after all outstanding references have been released
    return (g_cRefModule == 0) ? S_OK : S_FALSE;
}

void DllAddRef()
{
    InterlockedIncrement(&g_cRefModule);
}

void DllRelease()
{
    InterlockedDecrement(&g_cRefModule);
}

class CClassFactory : public IClassFactory
{
public:
    static HRESULT CreateInstance(REFCLSID clsid, const CLASS_OBJECT_INIT* pClassObjectInits, size_t cClassObjectInits, REFIID riid, void** ppv)
    {
        *ppv = NULL;
        HRESULT hr = CLASS_E_CLASSNOTAVAILABLE;
        for (size_t i = 0; i < cClassObjectInits; i++)
        {
            if (clsid == *pClassObjectInits[i].pClsid)
            {
                IClassFactory* pClassFactory = new (std::nothrow) CClassFactory(pClassObjectInits[i].pfnCreate);
                hr = pClassFactory ? S_OK : E_OUTOFMEMORY;
                if (SUCCEEDED(hr))
                {
                    hr = pClassFactory->QueryInterface(riid, ppv);
                    pClassFactory->Release();
                }
                break; // match found
            }
        }
        return hr;
    }

    CClassFactory(PFNCREATEINSTANCE pfnCreate) : _cRef(1), _pfnCreate(pfnCreate)
    {
        DllAddRef();
    }

    // IUnknown
    IFACEMETHODIMP QueryInterface(REFIID riid, void** ppv)
    {
        static const QITAB qit[] =
        {
            QITABENT(CClassFactory, IClassFactory),
            { 0 }
        };
        return QISearch(this, qit, riid, ppv);
    }

    IFACEMETHODIMP_(ULONG) AddRef()
    {
        return InterlockedIncrement(&_cRef);
    }

    IFACEMETHODIMP_(ULONG) Release()
    {
        long cRef = InterlockedDecrement(&_cRef);
        if (cRef == 0)
        {
            delete this;
        }
        return cRef;
    }

    // IClassFactory
    IFACEMETHODIMP CreateInstance(IUnknown* punkOuter, REFIID riid, void** ppv)
    {
        return punkOuter ? CLASS_E_NOAGGREGATION : _pfnCreate(riid, ppv);
    }

    IFACEMETHODIMP LockServer(BOOL fLock)
    {
        if (fLock)
        {
            DllAddRef();
        }
        else
        {
            DllRelease();
        }
        return S_OK;
    }

private:
    ~CClassFactory()
    {
        DllRelease();
    }

    long _cRef;
    PFNCREATEINSTANCE _pfnCreate;
};


STDAPI DllGetClassObject(REFCLSID clsid, REFIID riid, void** ppv)
{
    return CClassFactory::CreateInstance(clsid, c_rgClassObjectInit, ARRAYSIZE(c_rgClassObjectInit), riid, ppv);
}

// A struct to hold the information required for a registry entry

struct REGISTRY_ENTRY
{
    HKEY   hkeyRoot;
    PCWSTR pszKeyName;
    PCWSTR pszValueName;
    PCWSTR pszData;
};

// Creates a registry key (if needed) and sets the default value of the key

HRESULT CreateRegKeyAndSetValue(const REGISTRY_ENTRY* pRegistryEntry)
{
    HKEY hKey;
    HRESULT hr = HRESULT_FROM_WIN32(RegCreateKeyExW(pRegistryEntry->hkeyRoot, pRegistryEntry->pszKeyName,
        0, NULL, REG_OPTION_NON_VOLATILE, KEY_SET_VALUE, NULL, &hKey, NULL));
    if (SUCCEEDED(hr))
    {
        hr = HRESULT_FROM_WIN32(RegSetValueExW(hKey, pRegistryEntry->pszValueName, 0, REG_SZ,
            (LPBYTE)pRegistryEntry->pszData,
            ((DWORD)wcslen(pRegistryEntry->pszData) + 1) * sizeof(WCHAR)));
        RegCloseKey(hKey);
    }
    return hr;
}

//
// Registers this COM server
//
STDAPI DllRegisterServer()
{
    HRESULT hr;

    WCHAR szModuleName[MAX_PATH];

    if (!GetModuleFileNameW(g_hInst, szModuleName, ARRAYSIZE(szModuleName)))
    {
        hr = HRESULT_FROM_WIN32(GetLastError());
    }
    else
    {
        // List of registry entries we want to create

        const REGISTRY_ENTRY rgRegistryEntries[] =
        {
            // RootKey            KeyName                                                                              ValueName                       Data
            {HKEY_CURRENT_USER,   L"Software\\Classes\\CLSID\\" SZ_CLSID_ABCDPreviewHandler,                           NULL,                           SZ_ABCDPREVIEWHANDLER},
            {HKEY_CURRENT_USER,   L"Software\\Classes\\CLSID\\" SZ_CLSID_ABCDPreviewHandler L"\\InProcServer32",       NULL,                           szModuleName},
            {HKEY_CURRENT_USER,   L"Software\\Classes\\CLSID\\" SZ_CLSID_ABCDPreviewHandler L"\\InProcServer32",       L"ThreadingModel",              L"Apartment"},
            {HKEY_CURRENT_USER,   L"Software\\Classes\\CLSID\\" SZ_CLSID_ABCDPreviewHandler,                           L"AppID",                       L"{6d2b5079-2f0b-48dd-ab7f-97cec514d30b}"},
            {HKEY_CURRENT_USER,   L"Software\\Classes\\.abcd\\ShellEx\\{8895b1c6-b41f-4c1c-a562-0d564250836f}",        NULL,                           SZ_CLSID_ABCDPreviewHandler},
            {HKEY_CURRENT_USER,   L"Software\\Microsoft\\Windows\\CurrentVersion\\PreviewHandlers",                     SZ_CLSID_ABCDPreviewHandler,  L"ABCDPreviewHandler"},
        };

        hr = S_OK;
        for (int i = 0; i < ARRAYSIZE(rgRegistryEntries) && SUCCEEDED(hr); i++)
        {
            hr = CreateRegKeyAndSetValue(&rgRegistryEntries[i]);
        }
    }
    return hr;
}

//
// Unregisters this COM server
//
STDAPI DllUnregisterServer()
{
    HRESULT hr = S_OK;

    const PCWSTR rgpszKeys[] =
    {
        L"Software\\Classes\\CLSID\\" SZ_CLSID_ABCDPreviewHandler,
        L"Software\\Classes\\.abcd\\ShellEx\\{8895b1c6-b41f-4c1c-a562-0d564250836f}"
    };

    // Delete the registry entries
    for (int i = 0; i < ARRAYSIZE(rgpszKeys) && SUCCEEDED(hr); i++)
    {
        hr = HRESULT_FROM_WIN32(RegDeleteTreeW(HKEY_CURRENT_USER, rgpszKeys[i]));
        if (hr == HRESULT_FROM_WIN32(ERROR_FILE_NOT_FOUND))
        {
            // If the registry entry has already been deleted, say S_OK.
            hr = S_OK;
        }
    }
    if (SUCCEEDED(hr))
    {
        HKEY hKey;
        if (RegOpenKeyEx(HKEY_CURRENT_USER, L"Software\\Microsoft\\Windows\\CurrentVersion\\PreviewHandlers", 0, KEY_WRITE, &hKey) == ERROR_SUCCESS)
        {
            RegDeleteValue(hKey, SZ_CLSID_ABCDPreviewHandler);
            RegCloseKey(hKey);
        }
    }
    return hr;
}

Answer 1

The number one problem is your call to PostQuitMessage(0). It will stop the message pump that the parent window and your window uses, so it will stop the previewing process.

So, the first time you click an .abcd file everything works. The second time, the system asks your first instance to unload, and then you stop the message pump, and the file cannot be previewed. But the system is resilient and the next time, it will work again, etc.

So you really must stop calling PostQuitMessage(0).

As for artifacts you mention, you shouldn't mix GDI+ and Direct2D like you do. Everything you can do in GDI+ you should be able to do with pure Direct2D code.

For example here is the red ellipse drawn with Direct2D (zero GDI+ code):

// Begin drawing with Direct2D
pRenderTarget->BeginDraw();

// Draw the white background
D2D1_COLOR_F clearColor = D2D1::ColorF(1, 1, 1); // White color
pRenderTarget->Clear(clearColor);

int radius = 50;
int centerX = pRenderTarget->GetSize().width / 2;
int centerY = pRenderTarget->GetSize().height / 2;
D2D1_ELLIPSE ellipse = D2D1::Ellipse(D2D1::Point2F(centerX, centerY), radius, radius);
pRenderTarget->FillEllipse(ellipse, pRedBrush);

pRenderTarget->EndDraw();
// End drawing with Direct2D

Presuming you have created a pRedBrush after the render target creation, something like this:

pRenderTarget->CreateSolidColorBrush(D2D1::ColorF(1, 0, 0), &pRedBrush);

And you get antialiasing for free.

If you really want to use GDI+ and HDC, then read this Direct2D and GDI Interoperability Overview and create an ID2D1DCRenderTarget instead of an ID2D1HwndRenderTarget.

Other remarks:

• The ID2D1Factory can be cached in per-process
• When you destroy the window, release the associated Direct2D render target (and other resources created on it) first. In Unload in this code.
• You should check for all errors (for example the result to EndDraw)