I'm having problems with configuring a filter that replicates specific tables only

Question

I am trying to use filters to select specific tables to replicate. I tried running this with the installer

./tools/tungsten-installer --master-slave -a \
  ...
  --svc-extractor-filters=replicate \
--property=replicator.filter.replicate.do=test,*.foo"

and got this exception in trepctl status after the master had not installed properly:

Plugin class name property is missing or null: key=replicator.filter.replicate

which file is this properties file? How do I find it? Moreover, in specifying the settings for the filter, how do I know what exactly to put?

I discovered that I am supposed to Modify the configuration template file prior to configuration according to Issue 219 but what changes am I supposed to make in tungsten-replicator-2.0.5-diff that will later on be patched to the extraction?

Issue 254 suggests that If you want to apply a filter out of the box, you can use these options with tungsten-installer:

-a --property=replicator.filter.Replicate.ignoreFilter=schema_x.tablex,schema_x,tabley,schema_y,tablez
 --svc-thl-filter=Replicate 

However when I try using this for --property=replicator.filter.replicate.do, but the problem is still the same:

pendingExceptionMessage: Plugin class name property is missing or null: key=replicator.filter.replicate

Your assistance will be greatly appreciated.

Rumbi

Update: Hi I had a look at this file: /root/tungsten/tungsten-replicator/samples/ conf/filters/default/tableignore.tpl .Acoording to this sample, a static-SERVICE_NAME.properties file is supposed to have something like this configured, please confirm if this is the correct syntax:

replicator.filter.tabledo=com.continuent.tungsten.replicator.filter.JavaScr iptFilter 
replicator.filter.tabledo.script=${replicator.home.dir}/samples/ 
scripts/javascript-advanced/tabledo.js 
replicator.filter.tabledo.tables=foo(database).bar(table) 
replicator.stage.thl-to-dbms.filters=tabledo 

However, I did not find tabledo.js (or something similar) in the directory where tableignore.js exists. Could I please have the location of this file. If there is an alternative way of specifiying --property=replicator.filter.replicate.do=test without the use of this .js file, your suggestions are most welcome.

Answer 1

Download the latest version of tungsten replicator. The missing tpl file was added about a month ago. After installation, the filtered tables should be added to static-service.properties under the section FILTERS.

Answer 2

• Locate your replicator configuration file in static-YOUR_SERVICE_NAME.properties, e.g.

/opt/continuent/tungsten/tungsten-replicator/conf/static-mysql2vertica.properties

Make sure the individual dbms properties are set, in particular the setting replicator.applier.dbms:

# Batch applier basic configuration information. 
replicator.applier.dbms=com.continuent.tungsten.replicator.applier.batch.SimpleBatchApplier
replicator.applier.dbms.url=jdbc:mysql:thin://${replicator.global.db.host}:${replicator.global.db.port}/tungsten_${service.name}?createDB=true
replicator.applier.dbms.driver=org.drizzle.jdbc.DrizzleDriver
replicator.applier.dbms.user=${replicator.global.db.user}
replicator.applier.dbms.password=${replicator.global.db.password}
replicator.applier.dbms.startupScript=${replicator.home.dir}/samples/scripts/batch/mysql-connect.sql

# Timezone and character set. 
replicator.applier.dbms.timezone=GMT+0:00
replicator.applier.dbms.charset=UTF-8

# Parameters for loading and merging via stage tables. 
replicator.applier.dbms.stageTablePrefix=stage_xxx_
replicator.applier.dbms.stageDirectory=/tmp/staging
replicator.applier.dbms.stageLoadScript=${replicator.home.dir}/samples/scripts/batch/mysql-load.sql
replicator.applier.dbms.stageMergeScript=${replicator.home.dir}/samples/scripts/batch/mysql-merge.sql
replicator.applier.dbms.cleanUpFiles=false

Depending on the database you are replicating to you may have to omit/modify some of the lines.

For more information see: https://code.google.com/p/tungsten-replicator/wiki/Replicator_Batch_Loading

Answer 3

I don't know if this problem is still open or not.

I am using this version 2.0.6-xxx and installing the service using the parameters works for me. I would like to point it out, that as the parameter says "--svc-extractor-filters" defines an extractor filter. Meaning that the parameters will guide the extraction of data in the master server.

If you intend to use it on the slave service, you should use the "--svc-applier-filters".

The parameters
--svc-extractor-filters=replicate \
--property=replicator.filter.replicate.do=test,*.foo"

supposed to create the following in the properties file:

This is the filter set up.
replicator.filter.replicate=com.continuent.tungsten.replicator.filter.ReplicateFilter
replicator.filter.replicate.ignore=
replicator.filter.replicate.do=test,*.foo

And you should also be able to find the replicator.stage.binlog-to-q.filters=replicate parameter set.

If you intend to use this filter in the slave, please find the line with:
replicator.stage.q-to-dbms.filters=mysqlsessions,pkey,bidiSlave

and change it as
replicator.stage.q-to-dbms.filters=mysqlsessions,pkey,bidiSlave,replicate

Hope this brief description did help to you!