I'm trying to build a ckan (v2.3) data portal on my MacBook Pro running OS X 10.10.3, with the help of Homebrew. While I've had to make small adjustments to the official documentation to make installing from source work for me, I have been able to run it using Solr and Tomcat and install both the DataStore and FileStore components with no incident. I can, using a sysadmin account, upload files and download them again with no problems.
The problem is when I try to install DataPusher to try to make sure that user-uploaded data is automatically put into the DataStore. Following the "Development" section in the guide located here causes a few very strange things to happen.
First of all, although the "PORT" variable in my settings for the DataStore installation is set to the default (8800) and there is nothing blocking it on port 8800, the actual service seems to insist on running through port 8000. I wouldn't think that this ought to affect anything, but in the interest of documenting all strange behavior I'm leaving it here.
Secondly, as soon as I point my ckan installation's DataPusher url to the correct port and enable the app and all its default settings in ckan's development.ini file, my ckan installation instantly becomes completely unusable. The front page displays fine, but every single time I try to log in or register as any user I get this traceback (apologies for length):
File '/usr/lib/ckan/default/lib/python2.7/site-packages/weberror/evalexception.py', line 431 in respond
app_iter = self.application(environ, detect_start_response)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
resp = self.call_func(req, *args, **self.kwargs)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
return self.func(req, *args, **kwargs)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in __call__
return request.get_response(self.app)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
application, catch_exc_info=False)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
app_iter = application(self.environ, start_response)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
resp = self.call_func(req, *args, **self.kwargs)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
return self.func(req, *args, **kwargs)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/fanstatic/injector.py', line 54 in __call__
response = request.get_response(self.app)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
application, catch_exc_info=False)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
app_iter = application(self.environ, start_response)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__
return self.app(environ, start_response)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/beaker/middleware.py', line 155 in __call__
return self.wrap_app(environ, session_start_response)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/routes/middleware.py', line 131 in __call__
response = self.app(environ, start_response)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__
response = self.dispatch(controller, environ, start_response)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch
return controller(environ, start_response)
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/lib/base.py', line 338 in __call__
res = WSGIController.__call__(self, environ, start_response)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__
response = self._dispatch_call()
File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call
response = self._inspect_call(func)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
result = self._perform_call(func, args)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call
return func(**args)
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/controllers/home.py', line 182 in index
return base.render('home/index.html', cache_force=True)
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/lib/base.py', line 217 in render
loader_class=loader_class)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/templating.py', line 249 in cached_template
return render_func()
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/lib/base.py', line 155 in render_template
return render_jinja2(template_name, globs)
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/lib/base.py', line 104 in render_jinja2
return template.render(**extra_vars)
File '/usr/lib/ckan/default/lib/python2.7/site-packages/jinja2/environment.py', line 894 in render
return self.environment.handle_exception(exc_info, True)
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/templates/home/index.html', line 2 in top-level template code
{% set homepage_style = ( g.homepage_style or '1' ) %}
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/templates/page.html', line 1 in top-level template code
{% extends "base.html" %}
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/templates/base.html', line 103 in top-level template code
{%- block page %}{% endblock -%}
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/templates/page.html', line 14 in block "page"
{%- block header %}
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/templates/page.html', line 15 in block "header"
{% include "header.html" %}
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/templates/header.html', line 1 in top-level template code
{% block header_wrapper %}
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/templates/header.html', line 2 in block "header_wrapper"
{% block header_account %}
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/templates/header.html', line 5 in block "header_account"
{% block header_account_container_content %}
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/templates/header.html', line 9 in block "header_account_container_content"
{% block header_account_logged %}
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/templates/header.html', line 23 in block "header_account_logged"
{% set new_activities = h.new_activities() %}
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/lib/helpers.py', line 1855 in new_activities
return action({}, {})
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/logic/__init__.py', line 424 in wrapped
result = _action(context, data_dict, **kw)
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/logic/action/get.py', line 3259 in dashboard_new_activities_count
context, data_dict)
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/logic/__init__.py', line 424 in wrapped
result = _action(context, data_dict, **kw)
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/logic/__init__.py', line 510 in wrapper
return action(context, data_dict)
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/logic/action/get.py', line 3190 in dashboard_activity_list
limit=limit, offset=offset)
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/model/activity.py', line 298 in dashboard_activity_list
q = _dashboard_activity_query(user_id, limit + offset)
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/model/activity.py', line 283 in _dashboard_activity_query
q1 = _user_activity_query(user_id, limit)
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/model/activity.py', line 126 in _user_activity_query
return _activities_union_all(q1, q2)
File '/Users/<user>/ckan/lib/default/src/ckan/ckan/model/activity.py', line 96 in _activities_union_all
return model.Session.query(model.Activity).select_entity_from(
AttributeError: 'Query' object has no attribute 'select_entity_from'
It appears to be related to how users' recent activity is displayed, but after that I have no clue what is going on. Common sense would seem to dictate that I could just disable the plugin and things would be alright again, but sadly this is not the case. Even once I disconnect from the DataPusher plugin in the ini file and stop the service completely, this error persists. It effectively destroys my entire ckan installation, and the only way I have been able to make things work again is by installing ckan from source all over again.
The first time this happened it appeared to be a fluke. I figured something had to have been corrupted in my database, so I uninstalled ckan, deleted all the PostgreSQL databases and reinstalled everything from a clean slate. The same thing happened. If anyone would be able to help me make sense of why this error is happening and more broadly why installing the DataPusher is wrecking my ckan websites permanently, I would be very grateful. Thanks for your assistance.
So the problem seems to be that using the normal ckan documentation for installing from source led me down the wrong path. Their installation still installs several wrong versions of SQLAlchemy-related plugins. Following the instructions here worked out for me and eliminated this crash. I would suggest to anyone who maintains the ckan sources to check that SQLAlchemy 0.8 is the version that is automatically installed, rather than 0.7.8. Thanks to Mikko Ohtamaa for the assistance.