flask alembic: avoid check DB revision ID

Question

I am deploying a Flask-based app based on this amazing tutorial. I deploy my DB through:

flask db init
flask db migrate
flask db upgrade

sometimes it checks for older version of the DB inside migrations directory and I get this error (full log below, here just what I think are the most important parts):

# ...
alembic.util.exc.CommandError: Directory migrations already exists
# ...
alembic.script.revision.ResolutionError: No such revision or branch 'f05567e712eb'
# ...
alembic.util.exc.CommandError: Can't locate revision identified by 'f05567e712eb'
# ...

I know how to fix this in several ways (as example recreate DB or drop the revision ID from the table etc.). But is there any way to prevent this "revision-ID check" alembic behavior?

Here the full error log:

Traceback (most recent call last):
File "/home/diagnosticator/venv/bin/flask", line 8, in <module>
  sys.exit(main())
File "/home/diagnosticator/venv/lib/python3.8/site-packages/flask/cli.py", line 894, in main
  cli.main(args=args, prog_name=name)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/flask/cli.py", line 557, in main
  return super(FlaskGroup, self).main(*args, **kwargs)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/click/core.py", line 697, in main
  rv = self.invoke(ctx)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/diagnosticator/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/diagnosticator/venv/lib/python3.8/site-packages/click/core.py", line 895, in invoke
  return ctx.invoke(self.callback, **ctx.params)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/click/core.py", line 535, in invoke
  return callback(*args, **kwargs)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/click/decorators.py", line 17, in new_func
  return f(get_current_context(), *args, **kwargs)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/flask/cli.py", line 412, in decorator
  return __ctx.invoke(f, *args, **kwargs)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/click/core.py", line 535, in invoke
  return callback(*args, **kwargs)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/flask_migrate/cli.py", line 31, in init
  _init(directory, multidb)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/flask_migrate/__init__.py", line 118, in init
  command.init(config, directory, 'flask')
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/command.py", line 42, in init
  raise util.CommandError("Directory %s already exists" % directory)
alembic.util.exc.CommandError: Directory migrations already exists
[2022-01-04 18:30:59,314] INFO in __init__: Diagnosticator-local startup
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
20
Traceback (most recent call last):
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/base.py", line 143, in _catch_revision_errors
  yield
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/base.py", line 206, in get_revisions
  return self.revision_map.get_revisions(id_)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/revision.py", line 299, in get_revisions
  return sum([self.get_revisions(id_elem) for id_elem in id_], ())
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/revision.py", line 299, in <listcomp>
  return sum([self.get_revisions(id_elem) for id_elem in id_], ())
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/revision.py", line 302, in get_revisions
  return tuple(
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/revision.py", line 303, in <genexpr>
  self._revision_for_ident(rev_id, branch_label)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/revision.py", line 360, in _revision_for_ident
  raise ResolutionError(
alembic.script.revision.ResolutionError: No such revision or branch 'f05567e712eb'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/diagnosticator/venv/bin/flask", line 8, in <module>
  sys.exit(main())
File "/home/diagnosticator/venv/lib/python3.8/site-packages/flask/cli.py", line 894, in main
  cli.main(args=args, prog_name=name)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/flask/cli.py", line 557, in main
  return super(FlaskGroup, self).main(*args, **kwargs)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/click/core.py", line 697, in main
  rv = self.invoke(ctx)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/diagnosticator/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/diagnosticator/venv/lib/python3.8/site-packages/click/core.py", line 895, in invoke
  return ctx.invoke(self.callback, **ctx.params)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/click/core.py", line 535, in invoke
  return callback(*args, **kwargs)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/click/decorators.py", line 17, in new_func
  return f(get_current_context(), *args, **kwargs)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/flask/cli.py", line 412, in decorator
  return __ctx.invoke(f, *args, **kwargs)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/click/core.py", line 535, in invoke
  return callback(*args, **kwargs)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/flask_migrate/cli.py", line 89, in migrate
  _migrate(directory, message, sql, head, splice, branch_label, version_path,
File "/home/diagnosticator/venv/lib/python3.8/site-packages/flask_migrate/__init__.py", line 195, in migrate
  command.revision(config, message, autogenerate=True, sql=sql,
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/command.py", line 176, in revision
  script_directory.run_env()
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/base.py", line 425, in run_env
  util.load_python_file(self.dir, 'env.py')
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 81, in load_python_file
  module = load_module_py(module_id, path)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/util/compat.py", line 83, in load_module_py
  spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "migrations/env.py", line 87, in <module>
  run_migrations_online()
File "migrations/env.py", line 80, in run_migrations_online
  context.run_migrations()
File "<string>", line 8, in run_migrations
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/runtime/environment.py", line 836, in run_migrations
  self.get_context().run_migrations(**kw)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/runtime/migration.py", line 321, in run_migrations
  for step in self._migrations_fn(heads, self):
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/command.py", line 156, in retrieve_migrations
  revision_context.run_autogenerate(rev, context)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/autogenerate/api.py", line 415, in run_autogenerate
  self._run_environment(rev, migration_context, True)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/autogenerate/api.py", line 425, in _run_environment
  if set(self.script_directory.get_revisions(rev)) != \
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/base.py", line 206, in get_revisions
  return self.revision_map.get_revisions(id_)
File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
  self.gen.throw(type, value, traceback)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/base.py", line 174, in _catch_revision_errors
  compat.raise_from_cause(util.CommandError(resolution))
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/util/compat.py", line 194, in raise_from_cause
  reraise(type(exception), exception, tb=exc_tb, cause=exc_value)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/util/compat.py", line 187, in reraise
  raise value.with_traceback(tb)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/base.py", line 143, in _catch_revision_errors
  yield
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/base.py", line 206, in get_revisions
  return self.revision_map.get_revisions(id_)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/revision.py", line 299, in get_revisions
  return sum([self.get_revisions(id_elem) for id_elem in id_], ())
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/revision.py", line 299, in <listcomp>
  return sum([self.get_revisions(id_elem) for id_elem in id_], ())
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/revision.py", line 302, in get_revisions
  return tuple(
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/revision.py", line 303, in <genexpr>
  self._revision_for_ident(rev_id, branch_label)
File "/home/diagnosticator/venv/lib/python3.8/site-packages/alembic/script/revision.py", line 360, in _revision_for_ident
  raise ResolutionError(
alembic.util.exc.CommandError: Can't locate revision identified by 'f05567e712eb'

Answer 1

this is what might be happening

1. You create a migration and run "flask db migrate -m 'msg'". This will look through your model and create a migrations file. If you open the migrations file, among other things you will see "revision=REV1" and "down_revision=None". The very first migration will have "down_revision = None" as there is nothing to downgrade do.
2. You then deploy the migration by "flask db upgrade". At this point, the migration (only migration you have created in step.1) gets applied. If you were to go into the db and do "select * from alembic_version", you will see the same version that was there in the revision field..
3. You now make some changes to the model and create another migration. The migration file will now have "revision=REV2", "down_revision=REV1"
4. You do "flask db upgrade" and if all goes well alembic_version will show "REV2"
5. Now - if for some reason you decide that you had made a mistake in the model and delete the migrations/version/REV2-migration-created-in-step-3 and say make some DB fixes and create another migration (REV3).
6. If you try to do "flask db upgrade" the alembic version (REV2) does not match any of the files you have in migrations/versions and you will see "Can't locate the revision identified by REV2".

If you create a migration and apply it, the way to un-apply it is to do "flask db downgrade" and then delete the migrations file - if you have to.

If you happen to delete the migrations file after doing flask db upgrade (and its not in source control for you to rever the deletion), then you have to manually deal with the situation that will require you to undo the db changes and also "update alembic version set version_num=version" where version is the previous version to which you manually reverted to.

While at this, I'd like to point out another common pitfall..

When multiple developers are on a project and there is parallel development going on in 2 branches. Let's say main branch was on REV10 and 2 developers pull a private-branch. They both have "revision=REV10, down_revision=REV9".. At the end of their development let's say

DEVELOPER1: has created 2 migrations thus having "revision=REV12, down_revision=REV11" and "revision=REV11, down_revision=REV10" in the 2 migration files he has checked-in to his private-branch [now - we can debate why not just one revision but I am taking an example here]

DEVELOPER2: has created 1 migration thus having "revision=REV11, down_revision=REV10" in his private-branch

Say Developer1 merges to main ..

Now when developer2 merges main to his private branch (before sending out a pull) - the fun (or the trouble) begins for him.

This will require developer2 to manually edit his migrations file(s) change the down_revisions so the migrations chain is in-tact at the end of combining the migration files.

Answer 2

Luckily, it's not possible. Such behaviour could make a lot harm - when we command alembic to do some operation it creates list of migrations from upper to lower in some topological order. One of them is given at command (e.g. head in alembic upgrade head), second is took from that table. Then it can safely apply migrations one by one.

Even if we come with some hack, there is no good way to answer what should be done in such case. And whatever we do, it would mean that some migration was applied but will never be downgraded.

Saying that, if you really want act some logical way instead of throwing error, it's a workaround I invented. It replaces current version of alembic table with current head. Use it at your own risk.

# It's alembic env.py file

from sqlalchemy.sql.elements import literal_column
from alembic import context
from alembic.util import CommandError
from alembic.script import ScriptDirectory

def run_migrations_online():
    """Run migrations in 'online' mode.

    In this scenario we need to create an Engine
    and associate a connection with the context.

    """
    connectable = engine_from_config(
        config.get_section(config.config_ini_section),
        prefix="sqlalchemy.",
        poolclass=pool.NullPool,
    )

    with connectable.connect() as connection:
        context.configure(connection=connection, target_metadata=target_metadata)

        with context.begin_transaction():
            # Until now everything is just defaults generated by alembic
            try:
                context.run_migrations()
            except CommandError: # in case run_migrations failed
                c = context.get_context()
                c.impl._exec(c._version.delete()) # delete current version from alembic table
                script = ScriptDirectory.from_config(config)
                head = script.revision_map.get_current_head() # get last migration in topological order
                c.impl._exec(
                    c._version.insert().values(
                        version_num=literal_column("'%s'" % head),
                    )
                ) # insert it into alembic table
                context.run_migrations() # run again as usual