Python Project Structure for Esky

Question

My question is essentially, "How should I structure the files and folders of my frozen, deployed Python-based Windows application." To understand my situation, here's some background:

I'm building a desktop application with Python 2.7 for my workplace. It is a GUI-based application built on PyQt. I am building the application with Esky which is a cross-platform freezing and updating framework. Esky basically wraps/calls py2exe, py2app, bb_freeze, or whatever tool you have installed that is appropriate for the current platform. Esky creates a zipped package that looks like this:

prog.exe                     - esky bootstrapping executable
appdata/                     - container for all the esky magic
  appname-X.Y.platform/      - specific version of the application
    prog.exe                 - executable(s) as produced by freezer module
    library.zip              - pure-python frozen modules
    pythonXY.dll             - python DLL
    esky-files/              - esky control files
      bootstrap/             - files not yet moved into bootstrapping env
      bootstrap-manifest.txt - list of files expected in bootstrap env
      lockfile.txt           - lockfile to block removal of in-use versions
      ...other deps...
  updates/                   - work area for fetching/unpacking updates

These zipped packages can then be placed on a file server which Esky looks to for updates. A number of methods are provided for managing updates including a very simple auto_update(). When an update occurs, the appname-X.Y.platform folder is essentially replaced with the next version folder... so the myApp.0.1.win32 folder is replaced by a myApp.0.2.win32 folder.

The other aspect of background you should know is that I am distributing the application to my coworkers, who do not have Python installed. I'm not distributing a Python package or library, I'm deploying a desktop application (my coworkers don't particularly care what it's written in, just that it works). I've built an Inno installer which installs the application, provides an uninstaller, and various shortcuts. Because everyone on the team has essentially the same Windows 7 64-bit environment, I'm pretty safe building for just that platform.

So, back to the issue of structure. I've read guides that recommend a certain format for a project skeleton, such as Learn Python the Hard Way, Exercise 46 or the Hitchhiker's Guide to Packaging. However these guides are oriented toward Python package developers, not compiled application developers.

I've also run into problems with Esky's appname-X.Y.platform folder, since it changes names every time the program is updated (to reflect the version number). Because I want some shortcuts in the Start Menu to always refer to documentation, changelog, etc, I have the installer place some of those files under the appdata folder. When the program updates, I have some code to check for newer versions of those files I want to be externally "visible" and copy the newer versions out of the appname-X.Y.platform folder and overwrite the copies in the appdata folder. I then also needed a means of storing persistent user settings, so the program generates and uses an appdata\settings folder (otherwise the settings would be wiped with each update).

Should I continue the process of having the application push new files out to the appdata folder post-update? Should I build my own structure of Docs, Examples, Settings, etc. and let the program populate those folders with newer files whenever necessary? Should I attempt to alter or take better advantage of Esky's behavior to better fit my usage? Perhaps I should rework my application to be destributable as both a Python package and an end-user application?

This question relates to this one about static files with Esky, this one about Python deployed application structure, and numerous generic questions about Python project structure which don't specifically address using Esky. Some videos discussing Esky are also available here and here.

I'm seeking recommendations for "best practice" methods to handle these challenges. If this doesn't fit the StackOverflow Question format, I'll gladly attempt to reword or narrow the focus of my question.

Answer 1

So here's my solution to the problem mentioned about about making files available to shortcuts at a static location despite the fact that Esky's auto-updating changes the name of my application folder every update. The function below I have within a class definition for a QMainWindow.

Logging statements could be replaced with print statements if your application doesn't use the logging module, though I highly recommend logging, especially if deploying a standalone application like this.

import os
import shutil
import logging

def push_updated_files(self):
    """
    Manually push auto-updated files from the application folder up to the appdata folder
    This enables shortcuts and other features on the computer to point to these files since the application
      directory name changes with each update.
    """
    logger = logging.getLogger(__name__)

    #Verify whether running script or frozen/deployed application
    if getattr(sys, 'frozen', False):
        logger.info("Verifying Application Integrity...")

        #Files which should by copied to appdata directory to be easily referenced by shortcuts, etc.
        data_files = ['main.ico',
                      'uninstall.ico',
                      'ReadMe.txt',
                      'changelog.txt',
                      'WhatsNew.txt',
                      'copyright.txt',
                      'Documentation.pdf']

        logger.debug("  App Path: {0}".format(self._app_path))

        #Get application top directory
        logger.debug("  AppData Directory: {0}".format(self._appdata_path))

        #Get application internal file path
        for f in data_files:
            a_file = f
            int_path = os.path.join(self._app_path, a_file)
            logger.debug("  Internal File Path: {0}".format(int_path))

            #Get file's creation time
            mtime_int = os.stat(int_path).st_mtime
            logger.debug("  Internal File Modified Time: {0}".format(time.ctime(mtime_int)))

            #Get external file path
            ext_path = os.path.join(self._appdata_path, a_file)
            if os.path.exists(ext_path):
                mtime_ext = os.stat(ext_path).st_mtime
                logger.debug("  External File Modified Time: {0}".format(time.ctime(mtime_ext)))

                if mtime_int > mtime_ext:
                    logger.debug("  Replacing external file with new file...")
                    try:
                        os.remove(ext_path)
                        shutil.copy(int_path, ext_path)
                    except Exception, e:
                        logger.error("  Failed to replace the external file...", exc_info=True)
                else:
                    logger.debug("  External file is newer than internal file - all is well.")
            else:
                logger.debug("  Copying file to appdata to be externally accessible")
                shutil.copy(int_path, ext_path)

Also related to this, when dealing with user settings (which currently is only a history.txt file used to populate a recent files list) I have a settings folder under appdata but outside the application folder so that settings aren't lost each update. I may make similar folders for documentation and icons.