How to Deploy and Manage a Python Application with Systemd

Question

What is Deployment and Process Lifetime Managment with Systemd?

A Python Application is an executable Python script which may also have a dependency on some shared Python code which exists in local Python modules or Packages.

To deploy an application (any application, not necessarily a Python one) we would normally run some kind of build process and then copy the output of the build to a pre-determined directory location on a target machine. The target machine might be the same machine as the one we are using to develop our application.

If we do the above, then it is easy to manage our processes with systemd. We can write a service file which references an executable in a known location.

What is the Python Development Process with venv?

venv ships by default with Python 3. I have been using it to manage a Python virtual environment as part of the development process. It performs two functions:

1. Isolating the install of libraries required by one project from all other projects and the Operating System
2. It provides a way for me to build Python Shared Libraries containing Python Code which should be shared between multiple executable processes which are part of the same project

It is easier to understand (2.) if I present my project directory structure:

my-python-project/
  bin/
    webscraper-processes/
      webscraper.py
      html_processor.py
      feature_extractor.py
      database_uploader.py
      local_library/
        __init__.py
        ... other files ...
    cron-download-process/
      pycron-download-trigger.py
      downloader.py
      data-processor.py
      database_uploader_2.py
      lib_cron_download/
        __init__.py
        ... other files ...
  src/
    lib_database/
      __init__.py
      ... other files ...
    lib_kafka/
      __init__.py
      ... other files ...
  test/
    ... tests ...
  .venv/
    bin/activate
    ... other files ...
  pyproject.toml

Please excuse the slightly silly names, hopefully this structure makes it clear why I have structured the project this way.

• There are multiple "groups" of executables. Each executable is a microservice which performs some function. A group of executables is a group of microservices. Each group of microservices performs a set of related operations as part of a sequence or chain. They work together to perform some higher level process or operation. Executables in different groups are independent.
• Although the group webscraper-process and cron-download-process are independent, they do share some common code. That common code goes in 1 or more shared libraries under the src directory.
• Any code which is shared by a group but not between multiple groups goes in a "local" shared library. (I'm not totally convinced this is a good idea. Perhaps it should go in src as well.)

venv is used for two purposes:

• To manage and install external dependencies locally. This would include confluent-kafka, Confluent's Kafka Python library and sqlalchemy.
• To provide a mechanism for the executable Python scripts to be able to "see" the shared library code which lives in the src directory. Otherwise, the interpreter would only be able to see the code in the current working directory, which includes "local" libraries but not "shared" libraries which are shared across the two groups.

If there is a better approach than the one I am using, please let me know.

I also recently experimented with Poetry and if I understand correctly, it provides a more convenient interface to manage virtual environments. I am open to using something other than venv if it provides an easier way to manage deployments.

How to deploy this?

I don't fully understand how to deploy this and manage the process lifecycle with systemd.

• I know that I need to chose some target for the deployment. A fixed location which systemd can reference in the Service Files which I will write for systemd to use.
• Let's choose /opt/my-python-project/, which seems like a sensible location

Remaining questions include:

• How should I "deploy" code to this location?
• A simple approach would be to copy cp -r my-python-project/bin /opt/my-python-project/
• But that doesn't include the shared library code under src
• I suspect the shared library code should be built into a wheel and that wheel should be copied to /opt/my-python-project/dist/, and then installed into some venv in /opt/my-python-project/ ?
• Should the library code in the bin/webscraper-process and bin/cron-download-process be built into a wheel as well?
• Do I need another venv or suitable alternative in the target location /opt/my-python-project?
• Is my approach fundamentally flawed because the project directory structure setup is flawed? Is there a better approach to the one I am using regarding the use of directory structure and venv?