I'm almost successful in my efforts to transfer an excerpt of my ZODB (a small subset of folders, including resources taken from other folders) to another Plone instance; the showstopper is: during import, my objects are not written. Here is my export script:
[transmogrifier]
pipeline =
sitewalker
xmlwalker
uidextractor
encapsulator
manifestexporter
fileexporter
marshaller
propertiesexporter
commentsexporter
datacorrector
portletsexporter
writer
EXPORTING
[sitewalker]
blueprint = quintagroup.transmogrifier.sitewalker
path =
folder1
folder2
[xmlwalker]
blueprint = collective.transmogrifier.sections.xmlwalker
[uidextractor]
# my own section which parses the html text of the given fields,
# checks the href and src attributes for UIDs, and injects entries
# for the refered objects into the pipeline
blueprint = my.transmogrifier.uidextractor
inspect_fields =
text
description
notes
[encapsulator]
blueprint = plone.app.transmogrifier.mimeencapsulator
mimetype = item/_mimetype
field = string:datafield
[manifestexporter]
blueprint = quintagroup.transmogrifier.manifestexporter
[fileexporter]
blueprint = quintagroup.transmogrifier.fileexporter
[marshaller]
blueprint = quintagroup.transmogrifier.marshaller
[propertiesexporter]
blueprint = quintagroup.transmogrifier.propertiesexporter
[commentsexporter]
blueprint = quintagroup.transmogrifier.commentsexporter
[datacorrector]
blueprint = quintagroup.transmogrifier.datacorrector
sources =
marshall
[portletsexporter]
blueprint = quintagroup.transmogrifier.portletsexporter
[writer]
blueprint = quintagroup.transmogrifier.writer
context = directory
path = var/export/
prefix = structure
[EXPORTING]
blueprint = quintagroup.transmogrifier.logger
keys =
_type
_path
Based on the default export and import scripts from quintagroup.transmogrifier, here is my import script:
[transmogrifier]
pipeline =
reader
pathfixer
constructor
schemaupdater
datacorrector
demarshaller
uidupdater
referencesimporter
propertiesimporter
commentsimporter
portletsimporter
printcounters
IMPORTING
[reader]
blueprint = quintagroup.transmogrifier.reader
prefix =
path = /path/to/var/export
context = directory
# unchanged entries I don't really understand:
.objects.xml = manifest
.marshall.xml = marshall
.properties.xml = propertymanager
.comments.xml = comments
.file-fields.xml = file-fields
.interfaces.xml = interfaces
.portlets.xml = portlets
[pathfixer]
blueprint = plone.app.transmogrifier.pathfixer
stripstring = Plone/
prependstring =
[constructor]
blueprint = collective.transmogrifier.sections.constructor
[schemaupdater]
blueprint = plone.app.transmogrifier.atschemaupdater
[datacorrector]
blueprint = quintagroup.transmogrifier.datacorrector
type = import
sources =
marshall
[fileimporter]
blueprint = quintagroup.transmogrifier.fileimporter
[demarshaller]
blueprint = quintagroup.transmogrifier.demarshaller
[uidupdater]
blueprint = plone.app.transmogrifier.uidupdater
[referencesimporter]
blueprint = quintagroup.transmogrifier.referencesimporter
[propertiesimporter]
blueprint = quintagroup.transmogrifier.propertiesimporter
[commentsimporter]
blueprint = quintagroup.transmogrifier.commentsimporter
[portletsimporter]
blueprint = quintagroup.transmogrifier.portletsimporter
[printcounters]
blueprint = collective.transmogrifier.sections.summary
count = true
[IMPORTING]
blueprint = quintagroup.transmogrifier.logger
keys =
_type
_path
As far as I understand, the reader comes up to the writer;
the marshaller comes up for the demarshaller, etc.
However, while importing, the objects are not actually written to the
database. Tweaking the blueprints with a counting facility (in the
add-info branch of my forks of the packages), I got the following
overview:
Items summary
~~~~~~~~~~~~~
[reader]:
created: 714
[pathfixer]:
got: 714
forwarded: 714
stripped: 390
[constructor]:
got: 714
missing-type: 714
missing-info: 714
forwarded: 714
[datacorrector]:
got: 714
forwarded: 714
[fileimporter]:
got: 31
forwarded: 31
[demarshaller]:
got: 31
forwarded: 31
[uidupdater]:
got: 31
forwarded: 31
[referencesimporter]:
passed-through: 31
[printcounters]:
passed-through: 31
[IMPORTING]:
got: 31
forwarded: 31
When the constructor doesn't do anything about the item, I count the reasons; obviously the constructor section doesn't write anything because it doesn't know the types of the objects to create. But shouldn't this
information have been created somewhere?!
(There is another problem: apparently the fileimporter receives only
31 objects of the 714 passed on by the datacorrector. But first I'd
like to see anything imported.)
What am I doing wrong?
Oh, and online documentation for the collective.transmogrifier.sections would be nice;they even lack docstrings ...
Edit: My counting-enabled forks are here (branches add-info):
2nd Edit:
When I move the IMPORTING section before constructer, the counters output is:
Items summary
~~~~~~~~~~~~~
[reader]:
created: 714
[pathfixer]:
got: 714
forwarded: 714
stripped: 390
[IMPORTING]:
got: 714
forwarded: 714
[constructor]:
got: 714
missing-type: 714
missing-info: 714
forwarded: 714
[datacorrector]:
got: 714
forwarded: 714
[fileimporter]:
got: 31
forwarded: 31
[demarshaller]:
got: 31
forwarded: 31
[uidupdater]:
got: 31
forwarded: 31
[referencesimporter]:
passed-through: 31
[printcounters]:
passed-through: 31
Thus, the collector still can't find _type information.
3rd Edit:
I added a little facility which prints a short info about the items found (the 1st item per section by default; _path and _type with values, if present, and a list of the other keys). The result is:
[reader], item #1:
_path=''
other keys: _import_context (DirectoryImportContext)
[pathfixer], item #1:
_path=''
other keys: _import_context (DirectoryImportContext)
[pathfixer], item #2:
_path=''
other keys: _import_context (DirectoryImportContext)
2015-08-18 18:39:56 INFO IMPORTING _path=
[constructor], item #1:
_path=''
other keys: _import_context (DirectoryImportContext)
[reader], item #2:
_path='some/archetypes/object/containing/a/video'
other keys: _files (dict), _import_context (DirectoryImportContext)
Indeed there are no _type keys anywhere, so I need a section which provides them.
4th Edit:
After re-inserting manifestimporter before constructor, I got:
[reader], item #1:
_path=''
other keys: _import_context (DirectoryImportContext)
[pathfixer], item #1:
_path=''
other keys: _import_context (DirectoryImportContext)
[pathfixer], item #2:
_path=''
other keys: _import_context (DirectoryImportContext)
2015-08-19 10:15:43 INFO IMPORTING _path=
[constructor], item #1:
_path=''
other keys: _import_context (DirectoryImportContext)
[manifestimporter], item #1:
_path=''
other keys: _import_context (DirectoryImportContext)
[reader], item #2:
_path=' [reader], item #1:
_path=''
other keys: _import_context (DirectoryImportContext)
[pathfixer], item #1:
_path=''
other keys: _import_context (DirectoryImportContext)
[pathfixer], item #2:
_path=''
other keys: _import_context (DirectoryImportContext)
2015-08-19 10:15:43 INFO IMPORTING _path=
[constructor], item #1:
_path=''
other keys: _import_context (DirectoryImportContext)
[manifestimporter], item #1:
_path=''
other keys: _import_context (DirectoryImportContext)
[reader], item #2:
_path='some/archetypes/object/containing/a/video'
other keys: _files (dict), _import_context (DirectoryImportContext)
...
2015-08-19 10:15:44 INFO IMPORTING
Pipeline processing time: 00:00:00
715 items were generated in source sections
2 went through full pipeline
713 were discarded in some section
The manifestimporter sections doesn't forward any previous items, so all items from reader are thrown away.
5th Edit: Tried the recommended way to use the "Site Configuration Export Step;" I edited the default export script and tried the export, but I got the following traceback:
Traceback (innermost last):
Module ZPublisher.Publish, line 138, in publish
Module ZPublisher.mapply, line 77, in mapply
Module ZPublisher.Publish, line 48, in call_object
Module Products.GenericSetup.tool, line 598, in manage_exportSelectedSteps
Module Products.GenericSetup.tool, line 1053, in _doRunExportSteps
Module quintagroup.transmogrifier.exportimport, line 74, in exportSiteStructure
Module collective.transmogrifier.utils, line 121, in constructPipeline
Module quintagroup.transmogrifier.sitewalker, line 29, in __init__
TypeError: ('Could not adapt', {'manifestexporter': {'blueprint': 'quintagroup.transmogrifier.manifestexporter'}, 'transmogrifier': {'pipeline': '\nsitewalker\nuidextractor\npathfixer\nmanifestexporter\nfileexporter\nmarshaller\npropertiesexporter\ncommentsexporter\ndatacorrector\nportletsexporter\nwriter\nEXPORTING'}, 'uidextractor': {'blueprint': 'my.transmogrifier.uidextractor', 'trace-first': 'true', 'inspect_fields': '\ntext\ndescription\nnotes'}, 'sitewalker': {'blueprint': 'quintagroup.transmogrifier.sitewalker', 'start-path': '\nakademie/vortraege/d-02-verlegung-lektion-02-leitungsgraben', 'exclude-contained': 'false'}, 'xmlwalker': {'blueprint': 'collective.transmogrifier.sections.xmlwalker'}, 'encapsulator': {'blueprint': 'plone.app.transmogrifier.mimeencapsulator', 'mimetype': 'item/_mimetype', 'field': 'string:datafield'}, 'writer': {'blueprint': 'quintagroup.transmogrifier.writer', 'path': 'var/export/', 'prefix': '', 'context': 'tarball'}, 'commentsexporter': {'blueprint': 'quintagroup.transmogrifier.commentsexporter'}, 'pathfixer': {'blueprint': 'plone.app.transmogrifier.pathfixer', 'stripstring': 'unitracc/'}, 'echo': {'blueprint': 'visaplan.transmogrifier.echo'}, 'marshaller': {'blueprint': 'quintagroup.transmogrifier.marshaller'}, 'propertiesexporter': {'blueprint': 'quintagroup.transmogrifier.propertiesexporter'}, 'datacorrector': {'blueprint': 'quintagroup.transmogrifier.datacorrector', 'sources': '\nmarshall'}, 'breakpoint': {'blueprint': 'collective.transmogrifier.sections.breakpoint'}, 'EXPORTING': {'blueprint': 'quintagroup.transmogrifier.logger', 'keys': '\n_type\n_path'}, 'portletsexporter': {'blueprint': 'quintagroup.transmogrifier.portletsexporter'}, 'fileexporter': {'blueprint': 'quintagroup.transmogrifier.fileexporter'}}, <InterfaceClass zope.annotation.interfaces.IAnnotations>)
It didn't matter whether or not my uidexporter section was included.
6th Edit:
Here is the current export profile which I used for the Site Configuration Export Step:
[transmogrifier]
pipeline =
sitewalker
manifestexporter
fileexporter
marshaller
propertiesexporter
commentsexporter
datacorrector
portletsexporter
writer
EXPORTING
[sitewalker]
blueprint = quintagroup.transmogrifier.sitewalker
exclude-contained = false
start-path =
Plone/some/existing/structure
[manifestexporter]
blueprint = quintagroup.transmogrifier.manifestexporter
[fileexporter]
blueprint = quintagroup.transmogrifier.fileexporter
[marshaller]
blueprint = quintagroup.transmogrifier.marshaller
[propertiesexporter]
blueprint = quintagroup.transmogrifier.propertiesexporter
[commentsexporter]
blueprint = quintagroup.transmogrifier.commentsexporter
[datacorrector]
blueprint = quintagroup.transmogrifier.datacorrector
sources =
marshall
[portletsexporter]
blueprint = quintagroup.transmogrifier.portletsexporter
[writer]
blueprint = quintagroup.transmogrifier.writer
context = tarball
path = var/export/
prefix =
[EXPORTING]
blueprint = quintagroup.transmogrifier.logger
keys =
_type
_path
7th Edit:
A short information which kind of data is required by the reader to enable the constructor to create objects, and which section of the export pipeline (with an idea about the necessary options) might earn easy 150 reputation points ;-)
From a comparison to quintagroup.transmogrifier's import.cfg You are missing a manifestimporter section in your pipeline. Their pipeline reads: