/[cvs]/nfo/perl/libs/Data/Transfer/Sync.pod
ViewVC logotype

Diff of /nfo/perl/libs/Data/Transfer/Sync.pod

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.1 by joko, Sun Jan 19 02:03:48 2003 UTC revision 1.7 by joko, Tue May 13 08:09:33 2003 UTC
# Line 1  Line 1 
1  =pod  =pod
2    
3    
4    =head1 NAME
5    
6      Data::Transfer::Sync
7    
8    
9    =head1 DESCRIPTION
10    
11      Data::Transfer::Sync is a module providing a generic synchronization process
12      across arbitrary/multiple storages based on a ident/checksum mechanism.
13      It sits on top of Data::Storage.
14    
15    
16    
17  =head1 NEWS  =head1 NEWS
18    
 =head4 2003-01-18  -  integration with Torus  
19    
20    please look at http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/  =head4 2003-02-22  -  enabled multiple task in-process synchronization ...
21    
22      x using Data::Rap and rap.pl
23      x broke sync.pl...
24      o ... re-enable via Data::Transfer::Sync::API::V1?
25    
 =head4 2002-10-10  -  initial check-in  
26    
27    =head4 2003-02-09  -  backward compatibility with custom application
28    
29  =head1 DESCRIPTION    - Tangram <-> .csv
30      - Tangram <-> mysql
31      - Tangram <-> perl-code
32    
33  Data::Transfer::Sync is a module providing a generic synchronization process  
34  across arbitrary/multiple storages based on a ident/checksum mechanism.  =head4 2003-01-18  -  started integration with Torus
35  It sits on top of Data::Storage.  
36      - MAPI <-> LDAP
37      - please visit http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/
38    
39    
40    =head4 2002-10-10  -  initial commit
41    
42      - Tangram <-> mysql
43    
44    
45    =head1 ChangeLog
46    
47      2003-01/02    Refactoring:
48                              Split up Sync/Core.pm from Sync.pm.
49                              Split up functionality into sister modules.
50    
51    
52  =head1 REQUIREMENTS  =head1 REQUIREMENTS
53    
54    For full functionality:  =head3 dependencies
55    
56      from CPAN or (for win32) a ppm-repository:
57        App::Config (CPAN only)
58        Tie::IxHash
59        Log::Dispatch
60        Log::Dispatch::Config
61        mixin
62        Hash::Merge
63        Tie::SecureHash
64        Date::Manip
65        Digest::MD5
66        ... and maybe some more.
67    
68      from nfo/perl/libs:
69        Data::Compare
70      Data::Storage      Data::Storage
71      Data::Transform      Data::Transform
72      Data::Compare      DesignPattern:: (to be renamed into Class::)
73      ... and all their dependencies      ... and all their dependencies. (like Data::UUID & Co.)
74    
75    
76    =head3 additional modules required for 'outlook2ldap'
77    
78        Win32::OLE
79        Net::LDAP
80    
81  =head1 AUTHORS / COPYRIGHT  =head3 additional modules required for 'db2db'
82    
83  The Data::Storage module is Copyright (c) 2002 Andreas Motl.      DBI
84  All rights reserved.      DBD::mysql
85        DBD::CSV
86        Tangram
87    
88  You may distribute it under the terms of either the GNU General Public  
89  License or the Artistic License, as specified in the Perl README file.  
90    =head1 AUTHORS / COPYRIGHT / LICENSE
91    
92      The Data::Transfer::Sync module is Copyright (c) 2002, 2003 Andreas Motl.
93      All Rights Reserved.
94    
95      Code contributions, suggestions, support and ideas by
96      Sebastian Utz <su@tunemedia.de> and Jan Hoffmann <janosch@ultrajan.de>.
97      
98      You may distribute it under the terms of either the GNU General Public
99      License or the Artistic License, as specified in the Perl README file.
100    
101    
102    =head1 COPYRIGHT NOTE
103    
104      * CAESAR - Collaborative And Easy System ARchitecture
105      *
106      * Data Synchronization Module
107      *
108      * Permission to use, copy, modify and distribute this software and its
109      * documentation is hereby granted, provided that both the copyright
110      * notice and this permission notice appear in all copies of the
111      * software, derivative works or modified versions, and any portions
112      * thereof, and that both notices appear in supporting documentation.
113      *
114      * NETFRAG.ORG AND THE AUTHOR ALLOW FREE USE OF THIS SOFTWARE
115      * IN ITS "AS IS" CONDITION. WE DISCLAIM ANY LIABILITY OF ANY KIND
116      * FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE.
117    
118    
119    
120  =head1 SUPPORT / WARRANTY  =head1 SUPPORT / WARRANTY
121    
122  Data::Storage is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.    Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.
123    
124      Please feel free to contact me at <andreas.motl@ilo.de>.
125    
126      Any improvements and contributions are greatly appreciated.
127    
128      We have set up a newsgroup. (currently not much traffic)
129      Be welcome to visit news://news.netfrag.org/nfo.caesar or - if your system
130      lacks a newsreader - try http://news.netfrag.org/nfo.caesar .
131      
132    
133    
134    
135    =head1 TODO
136    
137    =head4 overall
138    
139      o try to get this stuff together with SyncML (http://syncml.org/) somehow ...?
140      o use CRC32 for checksum generation?
141    
142    
143    =head4 this code
144    
145      o sub _resolveIdentProvider
146      x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node
147      x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm
148      o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm
149      x introduce Sync/Compare/MyComparisonImplementation.pm
150      o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing
151         - this implies everything done is _really_ split up into generic actions - how else would we defer them???
152         - example uses:
153            - fetch whole checksum list from node
154            - remember source ident retransmits
155         - remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations
156      o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema)
157      o expand statistics / keep track of:
158        - touched/untouched nodes
159      o full sync
160        - just do a push and a pull for now but use stats for touched nodes in between to speed up things
161      o introduce some new metadata flags for a descent
162        - isNewNodePropagator
163        x isWriteProtected
164    
165    
166    
167    =head1 EXAMPLE OUTPUT CHUNKS
168    
169    =head2 Synchronizing orm objects against entries in a .csv file - 17 items modified
170    
171      notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping;
172      notice: OEF::Script::AbstractFeeder:
173          - Loading data of type [n/a] and
174            filtered by * from the storage named base
175            to the storage named backend - filtered by *.
176          - Will attempt to convert the data to Country.
177      info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
178      :c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
179      :c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c
180      :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
181      info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 180  /  total: 197  /  attempt_modify: 17  /  ok: 17
182    
183    =head2 Synchronizing orm objects against entries in a .csv file - all items in-sync
184      
185      notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping;
186      notice: OEF::Script::AbstractFeeder:
187          - Loading data of type [n/a] and
188            filtered by * from the storage named base
189            to the storage named backend - filtered by *.
190          - Will attempt to convert the data to Country.
191      info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
192      :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
193      :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
194      :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
195      info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 197  /  total: 197
196    
197    
198    
# Line 53  Data::Storage is free software. IT COMES Line 207  Data::Storage is free software. IT COMES
207  =head4 Mapping  =head4 Mapping
208    
209    - - - - - - - - - - - - - - - - - - - - - - - - - -    - - - - - - - - - - - - - - - - - - - - - - - - - -
210    info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: BizWorks::Process::Setup->startSync: Can't access mapping for node "Currency" - please check BizWorks::ResourceMapping.    info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: MyApp::Process::Setup->startSync: Can't access mapping for node "Currency" - please check MyApp::ResourceMapping.
211    - - - - - - - - - - - - - - - - - - - - - - - - - -    - - - - - - - - - - - - - - - - - - - - - - - - - -
212    You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match    You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match
213    the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how    the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how
# Line 64  Data::Storage is free software. IT COMES Line 218  Data::Storage is free software. IT COMES
218  =head4 DBD::AutoCSV's rulebase  =head4 DBD::AutoCSV's rulebase
219        
220    - - - - - - - - - - - - - - - - - - - - - - - - - -    - - - - - - - - - - - - - - - - - - - - - - - - - -
221    info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )    info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )
222    info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv    info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
223        
224    Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p    Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p
# Line 123  Data::Storage is free software. IT COMES Line 277  Data::Storage is free software. IT COMES
277    - - - - - - - - - - - - - - - - - - - - - - - - - -    - - - - - - - - - - - - - - - - - - - - - - - - - -
278    
279    
 =head1 TODO  
   
   - sub _resolveIdentProvider  
   - wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node  
   - split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm  
      - introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm  
   - introduce Sync/Compare/MyComparisonImplementation.pm  
   - some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing  
      - this implies everything done is _really_ split up into generic actions - how else would we defer them???  
      - example uses:  
         - fetch whole checksum list from node  
         - remember source ident retransmits  
      - remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations  
   - feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema)  
   - expand statistics / keep track of:  
     - touched/untouched nodes  
   - full sync  
     - just do a push and a pull for now but use stats for touched nodes in between to speed up things  
   - introduce some new metadata flags for a synchronization partner which is (e.g.) of "source" or "target":  
     - isNewNodePropagator  
     - isWriteProtected  
280    
281    
282  =cut  =cut
Legend:
Removed from v.1.1  
changed lines
  Added in v.1.7
MailToCvsAdmin">MailToCvsAdmin
 ViewVC Help
Powered by ViewVC 1.1.26 RSS 2.0 feed