Write image to a specific Device Memory Space defined by an IntPtr

Question

I'd like to write an image from a kernel to a specified place in device memory which I define by an IntPtr.

Although it's not directly related to this problem, it's the RenderTexture from Unity which I want to change from inside the kernel in order to pass it to a shader which will visualize my algorithm on the GPU.

So far I tried this:

CLkernel.SetArgument (0, (IntPtr)(width*height*4*sizeof(float)), pointerToRT);

Which threw InvalidArgumentSize as I cannot specify it as an image, and this:

renderTexture = new ComputeImage2D (CLcontext, ComputeMemoryFlags.WriteOnly | ComputeMemoryFlags.UseHostPointer, 
                                    new ComputeImageFormat (ComputeImageChannelOrder.Rgba, ComputeImageChannelType.Float), 
                                    width, height, 0, pointerToRT);
CLkernel.SetMemoryArgument (0, renderTexture);

which resulted in an InvalidHostPointer Exception, as the pointer points to a place already in the device memory.

This is the Kernel code:

kernel void WriteToRenderTexture (
write_only image2d_t bdsRT,
global float* terrainHeight )
{
int width = get_global_size (0);
int x = get_global_id (0);
int y = get_global_id (1);
int pixel = x + y * width;

int2 coords = { x, y };
float4 value = { 0, terrainHeight [pixel * 3], 0.5, 0 };

write_imagef (bdsRT, coords, value);
}

Any ideas how I'm able to do so with Cloo?