Capistrano rails | undefined method `already_invoked' for Rake::Task when setting rails_env while migrations during gitlab deploy

Question

I've spent a whole lot of time debugging this and I doesn't seem to be getting close to an answer. Also I thought about creating an issue in capistrano repository, but I don't really think this is an issue of capistrano itself.

The Context:

I have a rails 5 project in a private repository hosted in Gitlab EE. My repo is fully configured to use Gitlab CI, for automating deployments using capistrano, capistrano/rails, and capistrano/rvm. I have one job for deployment, in which I execute cap review deploy (being review the environment i want to deploy)

Everything runs well, until I reach the capistrano hook deploy:migrate in which it breaks with the following error.

The Error:

  (Backtrace restricted to imported tasks) 
  cap aborted!
  NoMethodError: undefined method `already_invoked' for <Rake::Task deploy:migrating => [set_rails_env]>:Rake::Task

  Tasks: TOP => deploy:migrate
  (See full trace by running task with --trace)
  The deploy has failed with an error: undefined method `already_invoked' for <Rake::Task deploy:migrating => [set_rails_env]>:Rake::Task

From what I've been able to grasp of it, it seems to be a problem with the rake tasks that sets up the rails_env (from the gem capistrano/rails). I've tracked to see if capistrano/rvm wasn't setting the correct ruby from rvm, but that wasn't the case. cappistrano/rvm setup was ok. I think maybe it have to do with something about gem versions or else.

My weak patch:

I've patched it to work when no new migrations are added, using an optional parameter provided by the folks at capistrano/rails:

set :conditionally_migrate, true

But it's still failing otherwise (when there are new migrations, which is most of the time)

Aditionally, capistrano works correctly when using it from my local environment (using bundle exec cap review deploy).

Thanks in advance! Hope you can help me find the answer...

EDIT: Adding some details about my files and stack:

In my Rails configuration I use an git-ignored environment_variables.yml file in which I set all those ENV['var'] configs that are shown in the capistrano config files. I also guarantee that those are correctly setted up in the runner that is doing the deploy job. (Using Gitlab Variables).

These CI jobs are runt using a gitlab-runner service installed on a DigitalOcean droplet inside a docker runner with the images specified per-job.

.gitlab-ci.yml

stages:
  - test
  - deploy
variables:
  MYSQL_DATABASE: "test_linting"
  MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
  DB_NAME: 'test_linting'
  DB_USER: 'root'
  DB_PASS: ''
  DB_HOST: 'mysql'
services:
  - mysql
testing:
  image: heroku/ruby
  stage: test
  cache: 
    paths:
      - vendor/cache
  script:
    - bundle install --without=development production --jobs $(nproc) --path=vendor/cache
    - bundle exec rails db:create RAILS_ENV=test
    - bundle exec rails db:migrate RAILS_ENV=test
    - bundle exec rails test
  artifacts:
    paths:
      - coverage/
dev-deploy:
  image: ruby:2.3
  stage: deploy
  environment: 
    name: $CI_BUILD_REF_NAME
    url: http://dev.linting.com
  before_script:
    - gem install capistrano -v '~> 3.7'
    - gem install capistrano-rails
    - gem install capistrano-rvm
    - gem install slackistrano
  script:
    - cap $CI_BUILD_REF_NAME deploy
  only:
    - development
deploy:
  image: ruby:2.3
  stage: deploy
  environment: 
    name: $CI_BUILD_REF_NAME
    url: http://$CI_BUILD_REF_NAME.linting.com
  before_script:
    - gem install capistrano -v '~> 3.7'
    - gem install capistrano-rails
    - gem install capistrano-rvm
    - gem install slackistrano
  script:
    - cap $CI_BUILD_REF_NAME deploy
  only:
    - qa
    - staging
review_deploy:
  image: ruby:2.3
  stage: deploy
  environment: 
    name: review
    url: http://rev.linting.com
  when: manual
  before_script:
    - gem install capistrano -v '~> 3.7'
    - gem install capistrano-rails
    - gem install capistrano-rvm
    - gem install slackistrano
  script:
    - cap review deploy
  except:
    - development
    - qa
    - staging
    - master

config/deploy.rb

lock '~> 3.7'

env_file = "./config/environment_variables.yml"
if File.exists?(env_file)
  YAML.load_file(env_file)['capistrano'].each do |key, value|
    ENV[key.to_s] = value
  end
end

set :application, ENV['PROJECT_NAME']
set :repo_url, ENV['REPO_URL']

set :rvm_type, :user
set :rvm_ruby_version, '2.3.1'

set :conditionally_migrate, true

set :linked_files, %w{config/environment_variables.yml}
set :linked_dirs, %w{log tmp/pids tmp/cache tmp/sockets vendor/bundle public/system}

set :slackistrano, {
  klass: Slackistrano::CustomMessaging,
  channel: ENV['SLACK_CHANNEL'],
  webhook: ENV['SLACK_HOOK']
}

namespace :deploy do
  desc 'Restart application'
  task :restart do
    on roles(:app), in: :sequence, wait: 5 do
      execute :touch, release_path.join('tmp/restart.txt')
    end
  end

  desc 'Reset environment (rake db:reset)'
  task :db_reset do
    on roles(:app) do
      within "#{current_path}" do
        with rails_env: "#{fetch(:rails_env)}" do
          execute :rake, "db:migrate:reset"
          execute :rake, "db:seed"
        end
      end
    end
  end

  after :published, 'deploy:restart'
  after :finishing, 'deploy:cleanup'
end

config/deploy/review.rb

# Server definiton: (define ip in local env, and pass in gitlab)
server ENV['DEV_DROPLET_IP'], user: 'deploy', roles: %w{web app db}, password: ENV["SSH_DEPLOY_PASS"]

# ONLY WORKS IF IT WAS RUN BY THE GITLAB CI RUNNER
set :branch, ENV['CI_BUILD_REF_NAME'] ? ENV['CI_BUILD_REF_NAME'] : 'development'
# Capistrano Variables
set :stage, 'review'
set :rails_env, 'review'

# Variables for rev
set :commit, ENV['CI_BUILD_REF'] ? ENV['CI_BUILD_REF'][0..7] : 'local'
set :user, ENV['GITLAB_USER_EMAIL'] ? ENV['GITLAB_USER_EMAIL']: 'local user'

# Capistrano Deployment Route
set :deploy_to, "/home/deploy/rev.#{ENV['PROJECT_NAME']}"

namespace :deploy do
  desc 'Set review branch in the review server'
  task :set_review_branch do
    on roles(:app) do
      within "#{current_path}" do
        execute "echo '#{fetch(:branch)}@#{fetch(:commit)}' >> #{release_path.join('tmp/rev_branch')}"
        execute "echo '#{fetch(:user)}' >> #{release_path.join('tmp/rev_user')}"
      end
    end
  end

  after :publishing, 'deploy:set_review_branch'

  before :finishing, 'deploy:db_reset'
end

Capfile

require "capistrano/setup"
require "capistrano/deploy"

require 'capistrano/rvm'
require 'capistrano/rails'

require 'slackistrano/capistrano'
require_relative 'lib/custom_messaging'

require "capistrano/scm/git"
install_plugin Capistrano::SCM::Git

# Load custom tasks from `lib/capistrano/tasks` if you have any defined
Dir.glob("lib/capistrano/tasks/*.rake").each { |r| import r }

Gemfile

source 'https://rubygems.org'

gem 'rails', '~> 5.0.0', '>= 5.0.0.1'
gem 'puma', '~> 3.0'
gem 'sass-rails', '~> 5.0'
gem 'bootstrap-sass'
gem 'uglifier', '>= 1.3.0'
gem 'jquery-rails'
gem 'turbolinks', '~> 5'
gem 'jbuilder', '~> 2.5'

gem 'mysql2'
gem 'haml-rails'
gem 'devise'
gem 'paperclip', git: 'git://github.com/thoughtbot/paperclip.git'
gem 'administrate', '~> 0.3.0'
gem 'bourbon'
gem 'rails_real_favicon', '~> 0.0.6'
gem 'listen', '~> 3.0.5'

group :development, :test do
  gem 'byebug', platform: :mri
  gem 'minitest-rails'
  gem 'minitest-reporters'
  gem 'simplecov'
  gem 'simplecov-json', require: false
  gem 'factory_girl_rails'
  gem 'faker'
end

group :development do
  gem 'annotate'
  gem 'better_errors'
  gem 'binding_of_caller'

  # Capistrano for Deployments
  gem 'capistrano', '~> 3.7'
  gem 'capistrano-rvm'
  gem 'capistrano-rails'
  gem 'slackistrano'

  gem 'web-console'
  gem 'spring'
  gem 'spring-watcher-listen', '~> 2.0.0'
end

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby]

Answer 1

My guess is that because you are installing the Capistrano Gems in the CI environment manually, there is a version mismatch. If you run bundle install and bundle exec on the CI server, or pin the installed versions, it will probably start working.

One solution I've used for this is to add all of the required deployment gems to a :deployment group in my Gemfile. Then you need to add the following to your deploy.rb:

set :bundle_without, %w{development test deployment}.join(' ')

Then change your script to be:

bundle install --except development test default production ...
bundle exec cap $CI_BUILD_REF_NAME deploy