back to top
Data::Transfer::Sync
back to top
Data::Transfer::Sync is a module providing a generic synchronization process
across arbitrary/multiple storages based on a ident/checksum mechanism.
It sits on top of Data::Storage.
back to top
x using Data::Rap and rap.pl
x broke sync.pl...
o ... re-enable via Data::Transfer::Sync::API::V1?
- Tangram <-> .csv
- Tangram <-> mysql
- Tangram <-> perl-code
- MAPI <-> LDAP
- please visit http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/
- Tangram <-> mysql
back to top
2003-01/02 Refactoring:
Split up Sync/Core.pm from Sync.pm.
Split up functionality into sister modules.
back to top
from CPAN or (for win32) a ppm-repository:
App::Config (CPAN only)
Tie::IxHash
Log::Dispatch
Log::Dispatch::Config
mixin
Hash::Merge
Tie::SecureHash
Date::Manip
Digest::MD5
... and maybe some more.
from nfo/perl/libs:
Data::Compare
Data::Storage
Data::Transform
DesignPattern:: (to be renamed into Class::)
... and all their dependencies. (like Data::UUID & Co.)
Win32::OLE
Net::LDAP
DBI
DBD::mysql
DBD::CSV
Tangram
back to top
The Data::Transfer::Sync module is Copyright (c) 2002, 2003 Andreas Motl.
All Rights Reserved.
Code contributions, suggestions, support and ideas by
Sebastian Utz <su@tunemedia.de> and Jan Hoffmann <janosch@ultrajan.de>.
You may distribute it under the terms of either the GNU General Public
License or the Artistic License, as specified in the Perl README file.
back to top
* CAESAR - Collaborative And Easy System ARchitecture
*
* Data Synchronization Module
*
* Permission to use, copy, modify and distribute this software and its
* documentation is hereby granted, provided that both the copyright
* notice and this permission notice appear in all copies of the
* software, derivative works or modified versions, and any portions
* thereof, and that both notices appear in supporting documentation.
*
* NETFRAG.ORG AND THE AUTHOR ALLOW FREE USE OF THIS SOFTWARE
* IN ITS "AS IS" CONDITION. WE DISCLAIM ANY LIABILITY OF ANY KIND
* FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE.
back to top
Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.
Please feel free to contact me at <andreas.motl@ilo.de>.
Any improvements and contributions are greatly appreciated.
We have set up a newsgroup. (currently not much traffic)
Be welcome to visit news://news.netfrag.org/nfo.caesar or - if your system
lacks a newsreader - try http://news.netfrag.org/nfo.caesar .
back to top
o try to get this stuff together with SyncML (http://syncml.org/) somehow ...?
o use CRC32 for checksum generation?
o sub _resolveIdentProvider
x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node
x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm
o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm
x introduce Sync/Compare/MyComparisonImplementation.pm
o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing
- this implies everything done is _really_ split up into generic actions - how else would we defer them???
- example uses:
- fetch whole checksum list from node
- remember source ident retransmits
- remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations
o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema)
o expand statistics / keep track of:
- touched/untouched nodes
o full sync
- just do a push and a pull for now but use stats for touched nodes in between to speed up things
o introduce some new metadata flags for a descent
- isNewNodePropagator
x isWriteProtected
back to top
notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping;
notice: OEF::Script::AbstractFeeder:
- Loading data of type [n/a] and
filtered by * from the storage named base
to the storage named backend - filtered by *.
- Will attempt to convert the data to Country.
info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 180 / total: 197 / attempt_modify: 17 / ok: 17
notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping;
notice: OEF::Script::AbstractFeeder:
- Loading data of type [n/a] and
filtered by * from the storage named base
to the storage named backend - filtered by *.
- Will attempt to convert the data to Country.
info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 197 / total: 197
back to top
When in "import" mode for windows file - DBD::AutoCSV may hang.
Hint: Maybe the source node contains an ident-, but no checksum-column?
back to top
- - - - - - - - - - - - - - - - - - - - - - - - - -
info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: MyApp::Process::Setup->startSync: Can't access mapping for node "Currency" - please check MyApp::ResourceMapping.
- - - - - - - - - - - - - - - - - - - - - - - - - -
You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match
the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how
to access the otherwise generic nodes.
- - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - -
info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )
info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p
m line 165, <GEN9> line 1.
called from C:/home/amo/develop/netfrag.org/nfo/perl/libs/Data/Storage/Handler/DBI.pm at 123.
DBI-Error: DBD::AutoCSV::st fetchrow_hashref failed: Attempt to fetch row from a Non-SELECT statement
notice: Data::Transfer::Sync->syncNodes: No nodes to synchronize.
- - - - - - - - - - - - - - - - - - - - - - - - - -
DBD::AutoCSV contains a rulebase which is spooled down while attempting to guess the style of the csv file regarding
parameters like newline (eol), column-seperation-character (sep_char), quoting character (quote_char).
If this spool runs out of entries and no style could be resolved, DBD::CSV dies causing this "Execution ERROR" which
results in a "DBI-Error" afterwards.
- - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - -
info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
critical: Data::Transfer::Sync->syncNodes: Can not synchronize: No ident found in source node, maybe try to "import" this node first.
- - - - - - - - - - - - - - - - - - - - - - - - - -
If lowlevel detection succeeds, but no other required informations are found, this message is issued.
"Other informations" might be:
- column-header-row completely missing
- ident column is empty
- - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - -
info: Data::Transfer::Sync->_prepareNode_DummyIdent( descent source )
pcritical: Data::Transfer::Sync->_modifyNode failed: "source" node is empty.
- - - - - - - - - - - - - - - - - - - - - - - - - -
The source node could not be loaded. Maybe the ident is missing. Please check manually.
Hint: Like above, the ident and/or checksum columns may be missing....
- - - - - - - - - - - - - - - - - - - - - - - - - -
back to top