--- nfo/perl/libs/Data/Transfer/Sync.pod 2003/02/09 16:47:40 1.3 +++ nfo/perl/libs/Data/Transfer/Sync.pod 2003/05/13 08:09:33 1.7 @@ -1,9 +1,29 @@ =pod +=head1 NAME + + Data::Transfer::Sync + + +=head1 DESCRIPTION + + Data::Transfer::Sync is a module providing a generic synchronization process + across arbitrary/multiple storages based on a ident/checksum mechanism. + It sits on top of Data::Storage. + + + =head1 NEWS +=head4 2003-02-22 - enabled multiple task in-process synchronization ... + + x using Data::Rap and rap.pl + x broke sync.pl... + o ... re-enable via Data::Transfer::Sync::API::V1? + + =head4 2003-02-09 - backward compatibility with custom application - Tangram <-> .csv @@ -11,7 +31,7 @@ - Tangram <-> perl-code -=head4 2003-01-18 - start integration with Torus +=head4 2003-01-18 - started integration with Torus - MAPI <-> LDAP - please visit http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/ @@ -22,13 +42,11 @@ - Tangram <-> mysql +=head1 ChangeLog -=head1 DESCRIPTION - - Data::Transfer::Sync is a module providing a generic synchronization process - across arbitrary/multiple storages based on a ident/checksum mechanism. - It sits on top of Data::Storage. - + 2003-01/02 Refactoring: + Split up Sync/Core.pm from Sync.pm. + Split up functionality into sister modules. =head1 REQUIREMENTS @@ -71,29 +89,86 @@ =head1 AUTHORS / COPYRIGHT / LICENSE - The Data::Transfer::Sync module is Copyright (c) 2002 Andreas Motl. - All rights reserved. + The Data::Transfer::Sync module is Copyright (c) 2002, 2003 Andreas Motl. + All Rights Reserved. Code contributions, suggestions, support and ideas by - Sebastian Utz and Jan Hoffmann . + Sebastian Utz and Jan Hoffmann . You may distribute it under the terms of either the GNU General Public License or the Artistic License, as specified in the Perl README file. +=head1 COPYRIGHT NOTE + + * CAESAR - Collaborative And Easy System ARchitecture + * + * Data Synchronization Module + * + * Permission to use, copy, modify and distribute this software and its + * documentation is hereby granted, provided that both the copyright + * notice and this permission notice appear in all copies of the + * software, derivative works or modified versions, and any portions + * thereof, and that both notices appear in supporting documentation. + * + * NETFRAG.ORG AND THE AUTHOR ALLOW FREE USE OF THIS SOFTWARE + * IN ITS "AS IS" CONDITION. WE DISCLAIM ANY LIABILITY OF ANY KIND + * FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE. + + + =head1 SUPPORT / WARRANTY Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND. Please feel free to contact me at . + Any improvements and contributions are greatly appreciated. + + We have set up a newsgroup. (currently not much traffic) + Be welcome to visit news://news.netfrag.org/nfo.caesar or - if your system + lacks a newsreader - try http://news.netfrag.org/nfo.caesar . + + + + +=head1 TODO + +=head4 overall + + o try to get this stuff together with SyncML (http://syncml.org/) somehow ...? + o use CRC32 for checksum generation? + + +=head4 this code + + o sub _resolveIdentProvider + x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node + x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm + o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm + x introduce Sync/Compare/MyComparisonImplementation.pm + o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing + - this implies everything done is _really_ split up into generic actions - how else would we defer them??? + - example uses: + - fetch whole checksum list from node + - remember source ident retransmits + - remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations + o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema) + o expand statistics / keep track of: + - touched/untouched nodes + o full sync + - just do a push and a pull for now but use stats for touched nodes in between to speed up things + o introduce some new metadata flags for a descent + - isNewNodePropagator + x isWriteProtected + =head1 EXAMPLE OUTPUT CHUNKS =head2 Synchronizing orm objects against entries in a .csv file - 17 items modified - notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping; + notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping; notice: OEF::Script::AbstractFeeder: - Loading data of type [n/a] and filtered by * from the storage named base @@ -105,9 +180,9 @@ :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 180 / total: 197 / attempt_modify: 17 / ok: 17 -=head2 Synchronizing orm objects against entries in a .csv file - in-sync +=head2 Synchronizing orm objects against entries in a .csv file - all items in-sync - notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping; + notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping; notice: OEF::Script::AbstractFeeder: - Loading data of type [n/a] and filtered by * from the storage named base @@ -132,7 +207,7 @@ =head4 Mapping - - - - - - - - - - - - - - - - - - - - - - - - - - - info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: BizWorks::Process::Setup->startSync: Can't access mapping for node "Currency" - please check BizWorks::ResourceMapping. + info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: MyApp::Process::Setup->startSync: Can't access mapping for node "Currency" - please check MyApp::ResourceMapping. - - - - - - - - - - - - - - - - - - - - - - - - - - You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how @@ -143,7 +218,7 @@ =head4 DBD::AutoCSV's rulebase - - - - - - - - - - - - - - - - - - - - - - - - - - - info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 ) + info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 ) info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p @@ -202,27 +277,6 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - -=head1 TODO - - o sub _resolveIdentProvider - x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node - x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm - o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm - x introduce Sync/Compare/MyComparisonImplementation.pm - o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing - - this implies everything done is _really_ split up into generic actions - how else would we defer them??? - - example uses: - - fetch whole checksum list from node - - remember source ident retransmits - - remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations - o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema) - o expand statistics / keep track of: - - touched/untouched nodes - o full sync - - just do a push and a pull for now but use stats for touched nodes in between to speed up things - o introduce some new metadata flags for a descent - - isNewNodePropagator - x isWriteProtected =cut