I would like to run my own executable binary file (generated from C code) bundled along with other regular files in my Firefox add-on. Since, it needs to create a new process for that, I think I should use either sdk/system/child_process or a native OS API by using js-ctypes.
My question is that does creating a new process have any negative impact on the add-on publishing process? Does it violate any Mozilla's rule?
Creating a new process has a severe impact on the add-on publishing process but it does not violate the rules.
By interacting with the system in such a low-level way, your add-on will be flagged for administrative review which can result in even trivial updates taking up to 3 months to be approved. It certainly won't always take that long (and the reviewers are very good at rushing through reviews which fix critical or security impacting bugs if you explain the mitigating circumstances to them) but for a typical new version it can be months between submitting the XPI file for review and it actually being delivered to users.
At the moment you can avoid that delay by hosting your add-on outside of the AMO ecosystem but that will change in the next 1-2 months when the add-on signing requirements come into play. They say that the reviews for non-hosted add-ons will be faster and I hope that this will be the case but even so, the need for a manual review might be a serious problem if you're looking to get a regular automated test and deployment process in place for your add-on.
I wouldn't be surprised if Mozilla make further changes in this area later in the year in order to stem the loss of their add-on developer community. In particular, it seems crazy that even unchanged code will trigger the need for a review for every minor update.
As for how you go about implementing it, I've not used the SDK before so your suggestions are probably best but a third option to consider is the use of the XPCOM nsIProcess interface. I don't use ctypes in my add-on any more but I would expect that to be flagged for admin review in the same way as any other approach that could feasibly launch an executable.
As far as the bundling is concerned, that shouldn't be a problem but you will need to supply the source code to Mozilla so they can ensure it is safe.