Calling Python code from Twisted

Question

First of all, I should say that this might more a design question rather than about code itself.

I have a network with one Server and multiple Clients (written in Twisted cause I need these asynchronous non-blocking features), such server-client couple it's just only receiving-sending messages.

However, at some point, I want one client to run a python file when received a certain message. That client should keep listening and talking to the Server, also, I should be able to stop that file if needed, so my first thought is starting a thread for that python file and forget about it.

At the end it should go like this: Server sends message to ClientA, ClientA and its dataReceived function interprets the message and decides to run that python file (which I don't know how long will take and maybe contains blocking calls), when that python file finishes running should send the result to ClientB.

So, questions are:

• Would it be starting a thread a good idea for that python file in ClientA?
• As I want to send the result of that python file to ClientB, can I have another reactor loop inside that python file?

In any case I would highly appreciate any kind of advice as both python and twisted are not my specialty and all these ideas may not be the best ones.

Thanks!

Answer 1

At first reading, I though you were implying twisted isn't python. If you are thinking that, keep in mind the following:

Twisted is a python framework, I.E. it is python. Specifically it's about getting the most out of a single process/thread/core by allowing the programmer to hand-tune the scheduling/order-of-ops in their own code (which is nearly the opposite of the typical use of threads).

While you can interact with threads in twisted, its quite tricky to do without ruining the efficiency of twisted. (for longer description of threads vs events see SO: https://stackoverflow.com/a/23876498/3334178 )

If you really want to spawn your new python away from your twisted python (I.E. get this work running on a different core) then I would look at spawning it off as a process, see Glyph's answer in this SO: https://stackoverflow.com/a/5720492/3334178 for good libraries to get that done.

Processes give the proper separation to allow your twisted apps to run without meaningful slowdown and you should find all your starting/stoping/pausing/killing needs will be fulfilled.

Answering your questions specifically:

Would it be starting a thread a good idea for that python file in ClientA?

I would say "No" its generally not a good idea, and in your specific case you should look at using processes instead.

Can I have another reactor loop inside that python file?

Strictly speaking "no you can't have multiple reactors" - but what twisted can do is concurrently manage hundreds or thousands of separate tasks, all in the same reactor, will do what you need. I.E. run all your different async tasks in one reactor, that is what twisted is built for.

BTW I always recommend the following tutorial for twisted: krondo Twisted Introduction http://krondo.com/?page_id=1327 Its long, but if you get through it this kind of work will become very clear.

All the best!