Adding attribute to thinking sphinx index config causing indexing fail

Question

Given schema:

create_table "letters", force: :cascade do |t|
t.string   "title",      limit: 255
t.text     "content",    limit: 16777215
t.date     "issue_date"
t.datetime "created_at",                  null: false
t.datetime "updated_at",                  null: false
t.integer  "osha_id",    limit: 4
t.string   "status",     limit: 255
end

indices/letter_index.rb

ThinkingSphinx::Index.define :letter, :with => :real_time do
# fields
indexes title, :sortable => true
indexes content
indexes issue_date_as_string

# attributes
has status, :type => :string
has created_at, :type => :timestamp
has updated_at, :type => :timestamp
end

When I rake ts:regenerate without the has status, :type => :string line, it generates the index just fine.

But when I add that line in, I get a screenful of error messages, of which the most relevant seems to be

/home/tim/.rvm/gems/ruby-2.2.1/gems/rake-12.0.0/exe/rake:27:in '<top (required)>' 
Mysql2::Error: unknown column: 'status'

I'm absolutely befuddled at what the error is trying to tell me and why it's failing in the first place. Am I missing something obvious?

[edited to add more information]

complete error output (after it dumps the contents of the database):

/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/connection.rb:93:in `rescue in query'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/connection.rb:96:in `query'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/connection.rb:73:in `execute'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/real_time/transcriber.rb:21:in `block (2 levels) in copy'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/connection.rb:39:in `block in take'
/home/tim/.rvm/gems/ruby-2.2.1/gems/innertube-1.1.0/lib/innertube.rb:138:in `take'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/connection.rb:37:in `take'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/real_time/transcriber.rb:20:in `block in copy'
/home/tim/.rvm/gems/ruby-2.2.1/gems/activesupport-4.2.4/lib/active_support/notifications.rb:164:in `block in instrument'
/home/tim/.rvm/gems/ruby-2.2.1/gems/activesupport-4.2.4/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/home/tim/.rvm/gems/ruby-2.2.1/gems/activesupport-4.2.4/lib/active_support/notifications.rb:164:in `instrument'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/logger.rb:3:in `log'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/real_time/transcriber.rb:19:in `copy'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/real_time/populator.rb:16:in `block in populate'
/home/tim/.rvm/gems/ruby-2.2.1/gems/activerecord-4.2.4/lib/active_record/relation/batches.rb:124:in `find_in_batches'
/home/tim/.rvm/gems/ruby-2.2.1/gems/activerecord-4.2.4/lib/active_record/querying.rb:9:in `find_in_batches'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/real_time/populator.rb:15:in `populate'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/real_time/populator.rb:3:in `populate'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/rake_interface.rb:30:in `block in generate'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/rake_interface.rb:29:in `each'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/rake_interface.rb:29:in `generate'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/tasks.rb:28:in `block (2 levels) in <top (required)>'
/home/tim/.rvm/gems/ruby-2.2.1/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
Mysql2::Error: unknown column: 'status'
/home/tim/.rvm/gems/ruby-2.2.1/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in `_query'
/home/tim/.rvm/gems/ruby-2.2.1/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in `block in query'
/home/tim/.rvm/gems/ruby-2.2.1/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in `handle_interrupt'
/home/tim/.rvm/gems/ruby-2.2.1/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in `query'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/connection.rb:121:in `results_for'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/connection.rb:88:in `query'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/connection.rb:73:in `execute'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/real_time/transcriber.rb:21:in `block (2 levels) in copy'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/connection.rb:39:in `block in take'
/home/tim/.rvm/gems/ruby-2.2.1/gems/innertube-1.1.0/lib/innertube.rb:138:in `take'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/connection.rb:37:in `take'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/real_time/transcriber.rb:20:in `block in copy'
/home/tim/.rvm/gems/ruby-2.2.1/gems/activesupport-4.2.4/lib/active_support/notifications.rb:164:in `block in instrument'
/home/tim/.rvm/gems/ruby-2.2.1/gems/activesupport-4.2.4/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/home/tim/.rvm/gems/ruby-2.2.1/gems/activesupport-4.2.4/lib/active_support/notifications.rb:164:in `instrument'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/logger.rb:3:in `log'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/real_time/transcriber.rb:19:in `copy'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/real_time/populator.rb:16:in `block in populate'
/home/tim/.rvm/gems/ruby-2.2.1/gems/activerecord-4.2.4/lib/active_record/relation/batches.rb:124:in `find_in_batches'
/home/tim/.rvm/gems/ruby-2.2.1/gems/activerecord-4.2.4/lib/active_record/querying.rb:9:in `find_in_batches'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/real_time/populator.rb:15:in `populate'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/real_time/populator.rb:3:in `populate'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/rake_interface.rb:30:in `block in generate'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/rake_interface.rb:29:in `each'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/rake_interface.rb:29:in `generate'
/home/tim/.rvm/gems/ruby-2.2.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/tasks.rb:28:in `block (2 levels) in <top (required)>'
/home/tim/.rvm/gems/ruby-2.2.1/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => ts:regenerate => ts:generate
(See full trace by running task with --trace)

further gratuitous information-- This shows the various things I've tried, all of which have failed with the same unknown column error:

ThinkingSphinx::Index.define :letter, :with => :real_time do
  # fields
  indexes title, :sortable => true
  indexes content
  #indexes issue_date
  indexes issue_date_as_string # this one works!
  #indexes status
  #indexes status_as_string
  #indexes osha_id
  #indexes issue_date

  # attributes
  #has status, :type => :string
  has created_at, :type => :timestamp
  has updated_at, :type => :timestamp
end

The *_as_string methods are just a shot in the dark:

models/letter.rb:

# experimental for thinking sphinx indexing on issue date
def issue_date_as_string
  self.issue_date.to_s
end

def status_as_string
  self.status.to_s
end

Answer 1

Appreciate all the extra detail Tim.

I think the issue here is related to a bug in Thinking Sphinx which leads to index files not being properly cleared out by ts:regenerate and thus they're still using the old schema (without the status attribute). This means any attempts to populate new data is going to hit that error.

However, I've fixed this bug a little while ago, and the v3.3.0 release of Thinking Sphinx includes it. Can you upgrade (there are no breaking changes) and see if that fixes the problem?