I'm need to make a WYSIWYG XML editor for a custom XML grammar. I have explored all the OpenSource tools that I could find under this domain. Includes VEX, Oxygen, Bitflux...
None of them served my purpose.
I'm also looking for javascript-based WYSIWYG HTML editors which can support and additional DOCTYPE with DTD or any grammar. I already checked out FCK and CKEditor and TinyMCE kind of editors.
Jaxe is ideal to create a GUI with a custom XML grammar (although you could say I'm biased, since I wrote most of it ;-)
See How to create an XML language and its XML editor in 5 minutes for an example (you can skip the "Definition of the XML language" part since you already have it). Or you can open an XSD file and choose the export to config menu as explained in the configuration files syntax documentation, to automatically create a config file from a schema.
However, as opposed to other XML editors, Jaxe does not try to be "WYSIWYG", it tries to be "WYSIWYM", even though it has a graphical interface. So, what you see generally does not look like the end result, it tries to convey the meaning of the XML elements, with visible tags. This results in semantically better documents, but users need to be aware of what semantic editing means, so that might or might not be what you're looking for...