1 |
=pod |
=pod |
2 |
|
|
3 |
|
|
4 |
|
=head1 NAME |
5 |
|
|
6 |
|
Data::Transfer::Sync |
7 |
|
|
8 |
|
|
9 |
|
=head1 DESCRIPTION |
10 |
|
|
11 |
|
Data::Transfer::Sync is a module providing a generic synchronization process |
12 |
|
across arbitrary/multiple storages based on a ident/checksum mechanism. |
13 |
|
It sits on top of Data::Storage. |
14 |
|
|
15 |
|
|
16 |
|
|
17 |
=head1 NEWS |
=head1 NEWS |
18 |
|
|
|
=head4 2003-01-18 - integration with Torus |
|
19 |
|
|
20 |
please look at http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/ |
=head4 2003-02-22 - enabled multiple task in-process synchronization ... |
21 |
|
|
22 |
|
x using Data::Rap and rap.pl |
23 |
|
x broke sync.pl... |
24 |
|
o ... re-enable via Data::Transfer::Sync::API::V1? |
25 |
|
|
|
=head4 2002-10-10 - initial check-in |
|
26 |
|
|
27 |
|
=head4 2003-02-09 - backward compatibility with custom application |
28 |
|
|
29 |
=head1 DESCRIPTION |
- Tangram <-> .csv |
30 |
|
- Tangram <-> mysql |
31 |
|
- Tangram <-> perl-code |
32 |
|
|
33 |
Data::Transfer::Sync is a module providing a generic synchronization process |
|
34 |
across arbitrary/multiple storages based on a ident/checksum mechanism. |
=head4 2003-01-18 - started integration with Torus |
35 |
It sits on top of Data::Storage. |
|
36 |
|
- MAPI <-> LDAP |
37 |
|
- please visit http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/ |
38 |
|
|
39 |
|
|
40 |
|
=head4 2002-10-10 - initial commit |
41 |
|
|
42 |
|
- Tangram <-> mysql |
43 |
|
|
44 |
|
|
45 |
|
=head1 ChangeLog |
46 |
|
|
47 |
|
2003-01/02 Refactoring: |
48 |
|
Split up Sync/Core.pm from Sync.pm. |
49 |
|
Split up functionality into sister modules. |
50 |
|
|
51 |
|
|
52 |
=head1 REQUIREMENTS |
=head1 REQUIREMENTS |
53 |
|
|
54 |
For full functionality: |
=head3 dependencies |
55 |
|
|
56 |
|
from CPAN or (for win32) a ppm-repository: |
57 |
|
App::Config (CPAN only) |
58 |
|
Tie::IxHash |
59 |
|
Log::Dispatch |
60 |
|
Log::Dispatch::Config |
61 |
|
mixin |
62 |
|
Hash::Merge |
63 |
|
Tie::SecureHash |
64 |
|
Date::Manip |
65 |
|
Digest::MD5 |
66 |
|
... and maybe some more. |
67 |
|
|
68 |
|
from nfo/perl/libs: |
69 |
|
Data::Compare |
70 |
Data::Storage |
Data::Storage |
71 |
Data::Transform |
Data::Transform |
72 |
Data::Compare |
DesignPattern:: (to be renamed into Class::) |
73 |
... and all their dependencies |
... and all their dependencies. (like Data::UUID & Co.) |
74 |
|
|
75 |
|
|
76 |
|
=head3 additional modules required for 'outlook2ldap' |
77 |
|
|
78 |
|
Win32::OLE |
79 |
|
Net::LDAP |
80 |
|
|
81 |
=head1 AUTHORS / COPYRIGHT |
=head3 additional modules required for 'db2db' |
82 |
|
|
83 |
The Data::Storage module is Copyright (c) 2002 Andreas Motl. |
DBI |
84 |
All rights reserved. |
DBD::mysql |
85 |
|
DBD::CSV |
86 |
|
Tangram |
87 |
|
|
88 |
You may distribute it under the terms of either the GNU General Public |
|
89 |
License or the Artistic License, as specified in the Perl README file. |
|
90 |
|
=head1 AUTHORS / COPYRIGHT / LICENSE |
91 |
|
|
92 |
|
The Data::Transfer::Sync module is Copyright (c) 2002, 2003 Andreas Motl. |
93 |
|
All Rights Reserved. |
94 |
|
|
95 |
|
Code contributions, suggestions, support and ideas by |
96 |
|
Sebastian Utz <su@tunemedia.de> and Jan Hoffmann <janosch@ultrajan.de>. |
97 |
|
|
98 |
|
You may distribute it under the terms of either the GNU General Public |
99 |
|
License or the Artistic License, as specified in the Perl README file. |
100 |
|
|
101 |
|
|
102 |
|
=head1 COPYRIGHT NOTE |
103 |
|
|
104 |
|
* CAESAR - Collaborative And Easy System ARchitecture |
105 |
|
* |
106 |
|
* Data Synchronization Module |
107 |
|
* |
108 |
|
* Permission to use, copy, modify and distribute this software and its |
109 |
|
* documentation is hereby granted, provided that both the copyright |
110 |
|
* notice and this permission notice appear in all copies of the |
111 |
|
* software, derivative works or modified versions, and any portions |
112 |
|
* thereof, and that both notices appear in supporting documentation. |
113 |
|
* |
114 |
|
* NETFRAG.ORG AND THE AUTHOR ALLOW FREE USE OF THIS SOFTWARE |
115 |
|
* IN ITS "AS IS" CONDITION. WE DISCLAIM ANY LIABILITY OF ANY KIND |
116 |
|
* FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE. |
117 |
|
|
118 |
|
|
119 |
|
|
120 |
=head1 SUPPORT / WARRANTY |
=head1 SUPPORT / WARRANTY |
121 |
|
|
122 |
Data::Storage is free software. IT COMES WITHOUT WARRANTY OF ANY KIND. |
Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND. |
123 |
|
|
124 |
|
Please feel free to contact me at <andreas.motl@ilo.de>. |
125 |
|
|
126 |
|
Any improvements and contributions are greatly appreciated. |
127 |
|
|
128 |
|
We have set up a newsgroup. (currently not much traffic) |
129 |
|
Be welcome to visit news://news.netfrag.org/nfo.caesar or - if your system |
130 |
|
lacks a newsreader - try http://news.netfrag.org/nfo.caesar . |
131 |
|
|
132 |
|
|
133 |
|
|
134 |
|
|
135 |
|
=head1 TODO |
136 |
|
|
137 |
|
=head4 overall |
138 |
|
|
139 |
|
o try to get this stuff together with SyncML (http://syncml.org/) somehow ...? |
140 |
|
o use CRC32 for checksum generation? |
141 |
|
|
142 |
|
|
143 |
|
=head4 this code |
144 |
|
|
145 |
|
o sub _resolveIdentProvider |
146 |
|
x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node |
147 |
|
x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm |
148 |
|
o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm |
149 |
|
x introduce Sync/Compare/MyComparisonImplementation.pm |
150 |
|
o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing |
151 |
|
- this implies everything done is _really_ split up into generic actions - how else would we defer them??? |
152 |
|
- example uses: |
153 |
|
- fetch whole checksum list from node |
154 |
|
- remember source ident retransmits |
155 |
|
- remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations |
156 |
|
o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema) |
157 |
|
o expand statistics / keep track of: |
158 |
|
- touched/untouched nodes |
159 |
|
o full sync |
160 |
|
- just do a push and a pull for now but use stats for touched nodes in between to speed up things |
161 |
|
o introduce some new metadata flags for a descent |
162 |
|
- isNewNodePropagator |
163 |
|
x isWriteProtected |
164 |
|
|
165 |
|
|
166 |
|
|
167 |
|
=head1 EXAMPLE OUTPUT CHUNKS |
168 |
|
|
169 |
|
=head2 Synchronizing orm objects against entries in a .csv file - 17 items modified |
170 |
|
|
171 |
|
notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping; |
172 |
|
notice: OEF::Script::AbstractFeeder: |
173 |
|
- Loading data of type [n/a] and |
174 |
|
filtered by * from the storage named base |
175 |
|
to the storage named backend - filtered by *. |
176 |
|
- Will attempt to convert the data to Country. |
177 |
|
info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* [] |
178 |
|
:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
179 |
|
:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c |
180 |
|
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
181 |
|
info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 180 / total: 197 / attempt_modify: 17 / ok: 17 |
182 |
|
|
183 |
|
=head2 Synchronizing orm objects against entries in a .csv file - all items in-sync |
184 |
|
|
185 |
|
notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping; |
186 |
|
notice: OEF::Script::AbstractFeeder: |
187 |
|
- Loading data of type [n/a] and |
188 |
|
filtered by * from the storage named base |
189 |
|
to the storage named backend - filtered by *. |
190 |
|
- Will attempt to convert the data to Country. |
191 |
|
info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* [] |
192 |
|
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
193 |
|
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
194 |
|
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
195 |
|
info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 197 / total: 197 |
196 |
|
|
197 |
|
|
198 |
|
|
207 |
=head4 Mapping |
=head4 Mapping |
208 |
|
|
209 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
210 |
info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: BizWorks::Process::Setup->startSync: Can't access mapping for node "Currency" - please check BizWorks::ResourceMapping. |
info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: MyApp::Process::Setup->startSync: Can't access mapping for node "Currency" - please check MyApp::ResourceMapping. |
211 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
212 |
You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match |
You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match |
213 |
the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how |
the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how |
218 |
=head4 DBD::AutoCSV's rulebase |
=head4 DBD::AutoCSV's rulebase |
219 |
|
|
220 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
221 |
info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 ) |
info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 ) |
222 |
info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv |
info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv |
223 |
|
|
224 |
Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p |
Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p |
277 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
278 |
|
|
279 |
|
|
|
=head1 TODO |
|
|
|
|
|
- sub _resolveIdentProvider |
|
|
- wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node |
|
|
- split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm |
|
|
- introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm |
|
|
- introduce Sync/Compare/MyComparisonImplementation.pm |
|
|
- some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing |
|
|
- this implies everything done is _really_ split up into generic actions - how else would we defer them??? |
|
|
- example uses: |
|
|
- fetch whole checksum list from node |
|
|
- remember source ident retransmits |
|
|
- remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations |
|
|
- feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema) |
|
|
- expand statistics / keep track of: |
|
|
- touched/untouched nodes |
|
|
- full sync |
|
|
- just do a push and a pull for now but use stats for touched nodes in between to speed up things |
|
|
- introduce some new metadata flags for a synchronization partner which is (e.g.) of "source" or "target": |
|
|
- isNewNodePropagator |
|
|
- isWriteProtected |
|
280 |
|
|
281 |
|
|
282 |
=cut |
=cut |