Is there a way in .Net Framwork to reference an object cross AppDomains without Serialization?

Question

I am developing an addon to a commercial software which has an object that I have no control with it. By "no control with it", I mean I can't access its source code and make it serializable. It just sits in the current AppDomain for me to reference when my addon gets attached.

Now I want to make my addon (dll library) hot plug&play and to achieve that I need load my addon to a separate AppDomain. I was able to do that with a dummy addon, for example, in one version, I let it MessageBox.Show("A"), and I can hot swap to a different version and have it MessageBox.Show("B").

Now to the real battle: I need have it output MessageBox.Show(obj.Title) in one version and in another MessageBox.Show(obj.Id). Granted I can pass Title or Id as string, but I need do a lot more than that and want to pass the whole obj to my addon so I can do things with it.

Since I can't make it serializable, direct passing it as an argument is out of question. I tried to create a container for it by creating a wrapper class like below:

[Serializable]
public class Arg : MarshalByRefObject
{
    public ObjectIHaveNoControl Obj { get; set; }
}

and then tried to pass it to my addon via proxy (I use AppDomainToolkit):

var proxy = Remote<Addon>.CreateProxy(addonDomain);
proxy.RemoteObject.DoSomething(new Arg() { Obj = currentDomainObject });

It throws the same exception saying the obj is not serializable.

Did I hit a dead-end? I'm about to give up, then I recalled we are able to use VSTO or Office Interop to control Excel from another process. So there's gonna be a way to accomplish this, isn't it?