I don't know exactly where is the problem so i write here to get some tips or clues about it. I would like to know if anybody has an opinion or an idea about it. The harvesting with ckan seems to work (i am able to get the datas on the open data site) but at the end its state stays "ongoing", so it stays blocked and i'm unable to reharvest other datas. Thank you
Here are the logs from fetch_consumer.log
2016-09-02 13:05:17,824 INFO [ckanext.harvest.queue] Received harvest object id: bd9dbc78-0b81-4566-87b4-61cdaf783b46
2016-09-02 13:05:17,840 DEBUG [ckanext.mydata.harvester] In Mydata Harvest fetch_stage
2016-09-02 13:05:17,845 DEBUG [ckanext.harvest.harvesters.ckanharvester] Using config: {u'force_all': False, u'user': u'admin'}
2016-09-02 13:05:17,956 DEBUG [ckanext.mydata.harvester] In Mydata Harvest import_stage
2016-09-02 13:05:17,957 DEBUG [ckanext.harvest.harvesters.ckanharvester] Using config: {u'force_all': False, u'user': u'admin'}
2016-09-02 13:05:17,990 DEBUG [ckanext.mydata.harvester] Package id d64484b6-1447-4c9e-a8dd-fde132842de0 doesn't exist, moving extras to base metadata
2016-09-02 13:05:17,995 INFO [ckanext.mydata.harvester] Package with GUID d64484b6-1447-4c9e-a8dd-fde132842de0 does not exist, let's create it
2016-09-02 13:05:18,292 INFO [ckanext.mydata.harvester] Attempting to upload resource 29b781c7-06d6-4a7d-bee4-09d12f2e29cf
2016-09-02 13:05:22,097 INFO [ckanext.mydata.harvester] Resource 29b781c7-06d6-4a7d-bee4-09d12f2e29cf has to be created
2016-09-02 13:05:26,046 INFO [ckanext.mydata.harvester] Resource 29b781c7-06d6-4a7d-bee4-09d12f2e29cf has been created
Below the logs from gather_consumer.log:
2016-09-02 13:04:43,226 DEBUG [ckanext.mydata.harvester] Delta packages to remove locally: [u'9f058088-056f-433a-b9f4-c61b2fb07230']
2016-09-02 13:04:43,226 DEBUG [ckanext.mydata.harvester] Candidate package_id for removal: u'9f058088-056f-433a-b9f4-c61b2fb07230'
/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py:80: SAWarning: Usage of the 'related attribute set' operation is not currently supported within the execution stage of the flush process. Results may not be consistent. Consider using alternative event listeners or connection-level operations instead.
sess._flush_warning("related attribute set")
/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py:38: SAWarning: Usage of the 'collection append' operation is not currently supported within the execution stage of the flush process. Results may not be consistent. Consider using alternative event listeners or connection-level operations instead.
sess._flush_warning("collection append")
/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py:1919: SAWarning: Attribute history events accumulated on 1 previously clean instances within inner-flush event handlers have been reset, and will not result in database updates. Consider using set_committed_value() within inner-flush event handlers to avoid this warning.
self._flush(objects)
2016-09-02 13:04:43,905 DEBUG [ckanext.harvest.queue] Received from plugin gather_stage: 6 objects (first: [u'b5ca80a0-6301-47c0-a592-338f6adfddd2'] last: [u'bd9dbc78-0b81-4566-87b4-61cdaf783b46'])
2016-09-02 13:04:43,908 DEBUG [ckanext.harvest.queue] Sent 6 objects to the fetch queue