/[cvs]/nfo/perl/libs/Data/Transfer/Sync.pod
ViewVC logotype

Diff of /nfo/perl/libs/Data/Transfer/Sync.pod

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.1 by joko, Sun Jan 19 02:03:48 2003 UTC revision 1.6 by joko, Sat Feb 22 18:25:38 2003 UTC
# Line 3  Line 3 
3    
4  =head1 NEWS  =head1 NEWS
5    
 =head4 2003-01-18  -  integration with Torus  
6    
7    please look at http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/  =head4 2003-02-22  -  enabled multiple task in-process synchronization ...
8    
9      x using Data::Rap and rap.pl
10      x broke sync.pl...
11      o ... re-enable via Data::Transfer::Sync::API::V1?
12    
13    
14    =head4 2003-02-09  -  backward compatibility with custom application
15    
16      - Tangram <-> .csv
17      - Tangram <-> mysql
18      - Tangram <-> perl-code
19    
20    
21    =head4 2003-01-18  -  started integration with Torus
22    
23      - MAPI <-> LDAP
24      - please visit http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/
25    
26    
27    =head4 2002-10-10  -  initial commit
28    
29      - Tangram <-> mysql
30    
 =head4 2002-10-10  -  initial check-in  
31    
32    
33  =head1 DESCRIPTION  =head1 DESCRIPTION
34    
35  Data::Transfer::Sync is a module providing a generic synchronization process    Data::Transfer::Sync is a module providing a generic synchronization process
36  across arbitrary/multiple storages based on a ident/checksum mechanism.    across arbitrary/multiple storages based on a ident/checksum mechanism.
37  It sits on top of Data::Storage.    It sits on top of Data::Storage.
38    
39    
40    
41  =head1 REQUIREMENTS  =head1 REQUIREMENTS
42    
43    For full functionality:  =head3 dependencies
44    
45      from CPAN or (for win32) a ppm-repository:
46        App::Config (CPAN only)
47        Tie::IxHash
48        Log::Dispatch
49        Log::Dispatch::Config
50        mixin
51        Hash::Merge
52        Tie::SecureHash
53        Date::Manip
54        Digest::MD5
55        ... and maybe some more.
56    
57      from nfo/perl/libs:
58        Data::Compare
59      Data::Storage      Data::Storage
60      Data::Transform      Data::Transform
61      Data::Compare      DesignPattern:: (to be renamed into Class::)
62      ... and all their dependencies      ... and all their dependencies. (like Data::UUID & Co.)
63    
64    
65    =head3 additional modules required for 'outlook2ldap'
66    
67        Win32::OLE
68        Net::LDAP
69    
70  =head1 AUTHORS / COPYRIGHT  =head3 additional modules required for 'db2db'
71    
72  The Data::Storage module is Copyright (c) 2002 Andreas Motl.      DBI
73  All rights reserved.      DBD::mysql
74        DBD::CSV
75        Tangram
76    
77  You may distribute it under the terms of either the GNU General Public  
78  License or the Artistic License, as specified in the Perl README file.  
79    =head1 AUTHORS / COPYRIGHT / LICENSE
80    
81      The Data::Transfer::Sync module is Copyright (c) 2002, 2003 Andreas Motl.
82      All Rights Reserved.
83    
84      Code contributions, suggestions, support and ideas by
85      Sebastian Utz <su@tunemedia.de> and Jan Hoffmann <janosch@ultrajan.de>.
86      
87      You may distribute it under the terms of either the GNU General Public
88      License or the Artistic License, as specified in the Perl README file.
89    
90    
91    =head1 COPYRIGHT NOTE
92    
93      * CAESAR - Collaborative And Easy System ARchitecture
94      *
95      * Data Synchronization Module
96      *
97      * Permission to use, copy, modify and distribute this software and its
98      * documentation is hereby granted, provided that both the copyright
99      * notice and this permission notice appear in all copies of the
100      * software, derivative works or modified versions, and any portions
101      * thereof, and that both notices appear in supporting documentation.
102      *
103      * NETFRAG.ORG AND THE AUTHOR ALLOW FREE USE OF THIS SOFTWARE
104      * IN ITS "AS IS" CONDITION. WE DISCLAIM ANY LIABILITY OF ANY KIND
105      * FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE.
106    
107    
108    
109  =head1 SUPPORT / WARRANTY  =head1 SUPPORT / WARRANTY
110    
111  Data::Storage is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.    Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.
112    
113      Please feel free to contact me at <andreas.motl@ilo.de>.
114    
115      Any improvements and contributions are greatly appreciated.
116    
117      We have set up a newsgroup. (currently not much traffic)
118      Be welcome to visit news://news.netfrag.org/nfo.caesar or - if your system
119      lacks a newsreader - try http://news.netfrag.org/nfo.caesar .
120      
121    
122    
123    
124    =head1 TODO
125    
126    =head4 overall
127    
128      o try to get this stuff together with SyncML (http://syncml.org/) somehow ...?
129    
130    
131    =head4 this code
132    
133      o sub _resolveIdentProvider
134      x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node
135      x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm
136      o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm
137      x introduce Sync/Compare/MyComparisonImplementation.pm
138      o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing
139         - this implies everything done is _really_ split up into generic actions - how else would we defer them???
140         - example uses:
141            - fetch whole checksum list from node
142            - remember source ident retransmits
143         - remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations
144      o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema)
145      o expand statistics / keep track of:
146        - touched/untouched nodes
147      o full sync
148        - just do a push and a pull for now but use stats for touched nodes in between to speed up things
149      o introduce some new metadata flags for a descent
150        - isNewNodePropagator
151        x isWriteProtected
152    
153    
154    
155    =head1 EXAMPLE OUTPUT CHUNKS
156    
157    =head2 Synchronizing orm objects against entries in a .csv file - 17 items modified
158    
159      notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping;
160      notice: OEF::Script::AbstractFeeder:
161          - Loading data of type [n/a] and
162            filtered by * from the storage named base
163            to the storage named backend - filtered by *.
164          - Will attempt to convert the data to Country.
165      info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
166      :c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
167      :c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c
168      :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
169      info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 180  /  total: 197  /  attempt_modify: 17  /  ok: 17
170    
171    =head2 Synchronizing orm objects against entries in a .csv file - all items in-sync
172      
173      notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping;
174      notice: OEF::Script::AbstractFeeder:
175          - Loading data of type [n/a] and
176            filtered by * from the storage named base
177            to the storage named backend - filtered by *.
178          - Will attempt to convert the data to Country.
179      info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
180      :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
181      :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
182      :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
183      info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 197  /  total: 197
184    
185    
186    
# Line 53  Data::Storage is free software. IT COMES Line 195  Data::Storage is free software. IT COMES
195  =head4 Mapping  =head4 Mapping
196    
197    - - - - - - - - - - - - - - - - - - - - - - - - - -    - - - - - - - - - - - - - - - - - - - - - - - - - -
198    info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: BizWorks::Process::Setup->startSync: Can't access mapping for node "Currency" - please check BizWorks::ResourceMapping.    info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: MyApp::Process::Setup->startSync: Can't access mapping for node "Currency" - please check MyApp::ResourceMapping.
199    - - - - - - - - - - - - - - - - - - - - - - - - - -    - - - - - - - - - - - - - - - - - - - - - - - - - -
200    You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match    You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match
201    the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how    the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how
# Line 64  Data::Storage is free software. IT COMES Line 206  Data::Storage is free software. IT COMES
206  =head4 DBD::AutoCSV's rulebase  =head4 DBD::AutoCSV's rulebase
207        
208    - - - - - - - - - - - - - - - - - - - - - - - - - -    - - - - - - - - - - - - - - - - - - - - - - - - - -
209    info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )    info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )
210    info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv    info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
211        
212    Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p    Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p
# Line 123  Data::Storage is free software. IT COMES Line 265  Data::Storage is free software. IT COMES
265    - - - - - - - - - - - - - - - - - - - - - - - - - -    - - - - - - - - - - - - - - - - - - - - - - - - - -
266    
267    
 =head1 TODO  
   
   - sub _resolveIdentProvider  
   - wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node  
   - split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm  
      - introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm  
   - introduce Sync/Compare/MyComparisonImplementation.pm  
   - some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing  
      - this implies everything done is _really_ split up into generic actions - how else would we defer them???  
      - example uses:  
         - fetch whole checksum list from node  
         - remember source ident retransmits  
      - remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations  
   - feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema)  
   - expand statistics / keep track of:  
     - touched/untouched nodes  
   - full sync  
     - just do a push and a pull for now but use stats for touched nodes in between to speed up things  
   - introduce some new metadata flags for a synchronization partner which is (e.g.) of "source" or "target":  
     - isNewNodePropagator  
     - isWriteProtected  
268    
269    
270  =cut  =cut
Legend:
Removed from v.1.1  
changed lines
  Added in v.1.6
MailToCvsAdmin">MailToCvsAdmin
 ViewVC Help
Powered by ViewVC 1.1.26 RSS 2.0 feed