1 |
=pod |
2 |
|
3 |
|
4 |
=head1 NEWS |
5 |
|
6 |
|
7 |
=head4 2003-02-22 - enabled multiple task in-process synchronization ... |
8 |
|
9 |
x using Data::Rap and rap.pl |
10 |
x broke sync.pl... |
11 |
o ... re-enable via Data::Transfer::Sync::API::V1? |
12 |
|
13 |
|
14 |
=head4 2003-02-09 - backward compatibility with custom application |
15 |
|
16 |
- Tangram <-> .csv |
17 |
- Tangram <-> mysql |
18 |
- Tangram <-> perl-code |
19 |
|
20 |
|
21 |
=head4 2003-01-18 - started integration with Torus |
22 |
|
23 |
- MAPI <-> LDAP |
24 |
- please visit http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/ |
25 |
|
26 |
|
27 |
=head4 2002-10-10 - initial commit |
28 |
|
29 |
- Tangram <-> mysql |
30 |
|
31 |
|
32 |
|
33 |
=head1 DESCRIPTION |
34 |
|
35 |
Data::Transfer::Sync is a module providing a generic synchronization process |
36 |
across arbitrary/multiple storages based on a ident/checksum mechanism. |
37 |
It sits on top of Data::Storage. |
38 |
|
39 |
|
40 |
|
41 |
=head1 REQUIREMENTS |
42 |
|
43 |
=head3 dependencies |
44 |
|
45 |
from CPAN or (for win32) a ppm-repository: |
46 |
App::Config (CPAN only) |
47 |
Tie::IxHash |
48 |
Log::Dispatch |
49 |
Log::Dispatch::Config |
50 |
mixin |
51 |
Hash::Merge |
52 |
Tie::SecureHash |
53 |
Date::Manip |
54 |
Digest::MD5 |
55 |
... and maybe some more. |
56 |
|
57 |
from nfo/perl/libs: |
58 |
Data::Compare |
59 |
Data::Storage |
60 |
Data::Transform |
61 |
DesignPattern:: (to be renamed into Class::) |
62 |
... and all their dependencies. (like Data::UUID & Co.) |
63 |
|
64 |
|
65 |
=head3 additional modules required for 'outlook2ldap' |
66 |
|
67 |
Win32::OLE |
68 |
Net::LDAP |
69 |
|
70 |
=head3 additional modules required for 'db2db' |
71 |
|
72 |
DBI |
73 |
DBD::mysql |
74 |
DBD::CSV |
75 |
Tangram |
76 |
|
77 |
|
78 |
|
79 |
=head1 AUTHORS / COPYRIGHT / LICENSE |
80 |
|
81 |
The Data::Transfer::Sync module is Copyright (c) 2002, 2003 Andreas Motl. |
82 |
All Rights Reserved. |
83 |
|
84 |
Code contributions, suggestions, support and ideas by |
85 |
Sebastian Utz <su@tunemedia.de> and Jan Hoffmann <janosch@ultrajan.de>. |
86 |
|
87 |
You may distribute it under the terms of either the GNU General Public |
88 |
License or the Artistic License, as specified in the Perl README file. |
89 |
|
90 |
|
91 |
=head1 COPYRIGHT NOTE |
92 |
|
93 |
* CAESAR - Collaborative And Easy System ARchitecture |
94 |
* |
95 |
* Data Synchronization Module |
96 |
* |
97 |
* Permission to use, copy, modify and distribute this software and its |
98 |
* documentation is hereby granted, provided that both the copyright |
99 |
* notice and this permission notice appear in all copies of the |
100 |
* software, derivative works or modified versions, and any portions |
101 |
* thereof, and that both notices appear in supporting documentation. |
102 |
* |
103 |
* NETFRAG.ORG AND THE AUTHOR ALLOW FREE USE OF THIS SOFTWARE |
104 |
* IN ITS "AS IS" CONDITION. WE DISCLAIM ANY LIABILITY OF ANY KIND |
105 |
* FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE. |
106 |
|
107 |
|
108 |
|
109 |
=head1 SUPPORT / WARRANTY |
110 |
|
111 |
Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND. |
112 |
|
113 |
Please feel free to contact me at <andreas.motl@ilo.de>. |
114 |
|
115 |
Any improvements and contributions are greatly appreciated. |
116 |
|
117 |
We have set up a newsgroup. (currently not much traffic) |
118 |
Be welcome to visit news://news.netfrag.org/nfo.caesar or - if your system |
119 |
lacks a newsreader - try http://news.netfrag.org/nfo.caesar . |
120 |
|
121 |
|
122 |
|
123 |
|
124 |
=head1 TODO |
125 |
|
126 |
=head4 overall |
127 |
|
128 |
o try to get this stuff together with SyncML (http://syncml.org/) somehow ...? |
129 |
|
130 |
|
131 |
=head4 this code |
132 |
|
133 |
o sub _resolveIdentProvider |
134 |
x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node |
135 |
x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm |
136 |
o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm |
137 |
x introduce Sync/Compare/MyComparisonImplementation.pm |
138 |
o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing |
139 |
- this implies everything done is _really_ split up into generic actions - how else would we defer them??? |
140 |
- example uses: |
141 |
- fetch whole checksum list from node |
142 |
- remember source ident retransmits |
143 |
- remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations |
144 |
o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema) |
145 |
o expand statistics / keep track of: |
146 |
- touched/untouched nodes |
147 |
o full sync |
148 |
- just do a push and a pull for now but use stats for touched nodes in between to speed up things |
149 |
o introduce some new metadata flags for a descent |
150 |
- isNewNodePropagator |
151 |
x isWriteProtected |
152 |
|
153 |
|
154 |
|
155 |
=head1 EXAMPLE OUTPUT CHUNKS |
156 |
|
157 |
=head2 Synchronizing orm objects against entries in a .csv file - 17 items modified |
158 |
|
159 |
notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping; |
160 |
notice: OEF::Script::AbstractFeeder: |
161 |
- Loading data of type [n/a] and |
162 |
filtered by * from the storage named base |
163 |
to the storage named backend - filtered by *. |
164 |
- Will attempt to convert the data to Country. |
165 |
info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* [] |
166 |
:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
167 |
:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c |
168 |
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
169 |
info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 180 / total: 197 / attempt_modify: 17 / ok: 17 |
170 |
|
171 |
=head2 Synchronizing orm objects against entries in a .csv file - all items in-sync |
172 |
|
173 |
notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping; |
174 |
notice: OEF::Script::AbstractFeeder: |
175 |
- Loading data of type [n/a] and |
176 |
filtered by * from the storage named base |
177 |
to the storage named backend - filtered by *. |
178 |
- Will attempt to convert the data to Country. |
179 |
info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* [] |
180 |
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
181 |
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
182 |
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
183 |
info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 197 / total: 197 |
184 |
|
185 |
|
186 |
|
187 |
=head1 BUGS |
188 |
|
189 |
When in "import" mode for windows file - DBD::AutoCSV may hang. |
190 |
Hint: Maybe the source node contains an ident-, but no checksum-column? |
191 |
|
192 |
|
193 |
=head1 USER LEVEL ERRORS |
194 |
|
195 |
=head4 Mapping |
196 |
|
197 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
198 |
info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: MyApp::Process::Setup->startSync: Can't access mapping for node "Currency" - please check MyApp::ResourceMapping. |
199 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
200 |
You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match |
201 |
the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how |
202 |
to access the otherwise generic nodes. |
203 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
204 |
|
205 |
|
206 |
=head4 DBD::AutoCSV's rulebase |
207 |
|
208 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
209 |
info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 ) |
210 |
info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv |
211 |
|
212 |
Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p |
213 |
m line 165, <GEN9> line 1. |
214 |
called from C:/home/amo/develop/netfrag.org/nfo/perl/libs/Data/Storage/Handler/DBI.pm at 123. |
215 |
|
216 |
DBI-Error: DBD::AutoCSV::st fetchrow_hashref failed: Attempt to fetch row from a Non-SELECT statement |
217 |
notice: Data::Transfer::Sync->syncNodes: No nodes to synchronize. |
218 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
219 |
DBD::AutoCSV contains a rulebase which is spooled down while attempting to guess the style of the csv file regarding |
220 |
parameters like newline (eol), column-seperation-character (sep_char), quoting character (quote_char). |
221 |
If this spool runs out of entries and no style could be resolved, DBD::CSV dies causing this "Execution ERROR" which |
222 |
results in a "DBI-Error" afterwards. |
223 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
224 |
|
225 |
|
226 |
=head4 Check structure of source node |
227 |
|
228 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
229 |
info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv |
230 |
critical: Data::Transfer::Sync->syncNodes: Can not synchronize: No ident found in source node, maybe try to "import" this node first. |
231 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
232 |
If lowlevel detection succeeds, but no other required informations are found, this message is issued. |
233 |
"Other informations" might be: |
234 |
- column-header-row completely missing |
235 |
- ident column is empty |
236 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
237 |
|
238 |
|
239 |
=head4 Modify structure of source node |
240 |
|
241 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
242 |
info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv |
243 |
info: Data::Transfer::Sync->_prepareNode_MetaProperties( descent source ) |
244 |
warning: Data::Transfer::Sync->_prepareNode_MetaProperties: node is lacking meta properties - will try to alter... |
245 |
SQL ERROR: Command 'ALTER' not recognized or not supported! |
246 |
|
247 |
SQL ERROR: Command 'ALTER' not recognized or not supported! |
248 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
249 |
The Engine found a node which structure does not match the required. It tries to alter this automatically - only when doing "import" - |
250 |
but the DBD driver (in this case DBD::CSV) gets in the way croaking not to be able to do this. |
251 |
This could also appear if your database connection has insufficient rights to modify the database structure. |
252 |
DBD::CSV croaks because it doesn't implement the ALTER command, so please edit your columns manually. |
253 |
Hint: Add columns with the names of your "ident" and "checksum" property specifications. |
254 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
255 |
|
256 |
|
257 |
=head4 Load source node by ident |
258 |
|
259 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
260 |
info: Data::Transfer::Sync->_prepareNode_DummyIdent( descent source ) |
261 |
pcritical: Data::Transfer::Sync->_modifyNode failed: "source" node is empty. |
262 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
263 |
The source node could not be loaded. Maybe the ident is missing. Please check manually. |
264 |
Hint: Like above, the ident and/or checksum columns may be missing.... |
265 |
- - - - - - - - - - - - - - - - - - - - - - - - - - |
266 |
|
267 |
|
268 |
|
269 |
|
270 |
=cut |