/[cvs]/nfo/perl/libs/Data/Transfer/Sync.pod
ViewVC logotype

Annotation of /nfo/perl/libs/Data/Transfer/Sync.pod

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.7 - (hide annotations)
Tue May 13 08:09:33 2003 UTC (21 years, 1 month ago) by joko
Branch: MAIN
CVS Tags: HEAD
Changes since 1.6: +19 -7 lines
updated

1 joko 1.1 =pod
2    
3    
4 joko 1.7 =head1 NAME
5    
6     Data::Transfer::Sync
7    
8    
9     =head1 DESCRIPTION
10    
11     Data::Transfer::Sync is a module providing a generic synchronization process
12     across arbitrary/multiple storages based on a ident/checksum mechanism.
13     It sits on top of Data::Storage.
14    
15    
16    
17 joko 1.1 =head1 NEWS
18    
19    
20 joko 1.6 =head4 2003-02-22 - enabled multiple task in-process synchronization ...
21    
22     x using Data::Rap and rap.pl
23     x broke sync.pl...
24     o ... re-enable via Data::Transfer::Sync::API::V1?
25    
26    
27 joko 1.2 =head4 2003-02-09 - backward compatibility with custom application
28 joko 1.1
29 joko 1.2 - Tangram <-> .csv
30     - Tangram <-> mysql
31     - Tangram <-> perl-code
32    
33    
34 joko 1.6 =head4 2003-01-18 - started integration with Torus
35 joko 1.2
36     - MAPI <-> LDAP
37 joko 1.3 - please visit http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/
38 joko 1.2
39    
40     =head4 2002-10-10 - initial commit
41    
42     - Tangram <-> mysql
43 joko 1.1
44    
45 joko 1.7 =head1 ChangeLog
46 joko 1.1
47 joko 1.7 2003-01/02 Refactoring:
48     Split up Sync/Core.pm from Sync.pm.
49     Split up functionality into sister modules.
50 joko 1.1
51    
52     =head1 REQUIREMENTS
53    
54 joko 1.3 =head3 dependencies
55    
56     from CPAN or (for win32) a ppm-repository:
57     App::Config (CPAN only)
58     Tie::IxHash
59     Log::Dispatch
60     Log::Dispatch::Config
61     mixin
62     Hash::Merge
63     Tie::SecureHash
64     Date::Manip
65     Digest::MD5
66     ... and maybe some more.
67    
68     from nfo/perl/libs:
69 joko 1.2 Data::Compare
70 joko 1.1 Data::Storage
71     Data::Transform
72 joko 1.3 DesignPattern:: (to be renamed into Class::)
73     ... and all their dependencies. (like Data::UUID & Co.)
74    
75    
76     =head3 additional modules required for 'outlook2ldap'
77    
78     Win32::OLE
79     Net::LDAP
80    
81     =head3 additional modules required for 'db2db'
82    
83     DBI
84     DBD::mysql
85     DBD::CSV
86     Tangram
87    
88 joko 1.1
89    
90 joko 1.2 =head1 AUTHORS / COPYRIGHT / LICENSE
91 joko 1.1
92 joko 1.4 The Data::Transfer::Sync module is Copyright (c) 2002, 2003 Andreas Motl.
93     All Rights Reserved.
94 joko 1.1
95 joko 1.2 Code contributions, suggestions, support and ideas by
96 joko 1.4 Sebastian Utz <su@tunemedia.de> and Jan Hoffmann <janosch@ultrajan.de>.
97 joko 1.2
98     You may distribute it under the terms of either the GNU General Public
99     License or the Artistic License, as specified in the Perl README file.
100 joko 1.1
101    
102 joko 1.4 =head1 COPYRIGHT NOTE
103    
104     * CAESAR - Collaborative And Easy System ARchitecture
105     *
106     * Data Synchronization Module
107     *
108     * Permission to use, copy, modify and distribute this software and its
109     * documentation is hereby granted, provided that both the copyright
110     * notice and this permission notice appear in all copies of the
111     * software, derivative works or modified versions, and any portions
112     * thereof, and that both notices appear in supporting documentation.
113     *
114     * NETFRAG.ORG AND THE AUTHOR ALLOW FREE USE OF THIS SOFTWARE
115     * IN ITS "AS IS" CONDITION. WE DISCLAIM ANY LIABILITY OF ANY KIND
116     * FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE.
117    
118    
119    
120 joko 1.1 =head1 SUPPORT / WARRANTY
121    
122 joko 1.2 Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.
123    
124     Please feel free to contact me at <andreas.motl@ilo.de>.
125 joko 1.5
126 joko 1.4 Any improvements and contributions are greatly appreciated.
127 joko 1.5
128 joko 1.4 We have set up a newsgroup. (currently not much traffic)
129     Be welcome to visit news://news.netfrag.org/nfo.caesar or - if your system
130 joko 1.5 lacks a newsreader - try http://news.netfrag.org/nfo.caesar .
131 joko 1.4
132    
133    
134    
135     =head1 TODO
136    
137     =head4 overall
138    
139 joko 1.7 o try to get this stuff together with SyncML (http://syncml.org/) somehow ...?
140     o use CRC32 for checksum generation?
141 joko 1.4
142    
143     =head4 this code
144    
145     o sub _resolveIdentProvider
146     x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node
147     x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm
148     o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm
149     x introduce Sync/Compare/MyComparisonImplementation.pm
150     o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing
151     - this implies everything done is _really_ split up into generic actions - how else would we defer them???
152     - example uses:
153     - fetch whole checksum list from node
154     - remember source ident retransmits
155     - remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations
156     o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema)
157     o expand statistics / keep track of:
158     - touched/untouched nodes
159     o full sync
160     - just do a push and a pull for now but use stats for touched nodes in between to speed up things
161     o introduce some new metadata flags for a descent
162     - isNewNodePropagator
163     x isWriteProtected
164 joko 1.2
165    
166    
167     =head1 EXAMPLE OUTPUT CHUNKS
168    
169     =head2 Synchronizing orm objects against entries in a .csv file - 17 items modified
170    
171 joko 1.4 notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping;
172 joko 1.2 notice: OEF::Script::AbstractFeeder:
173     - Loading data of type [n/a] and
174     filtered by * from the storage named base
175     to the storage named backend - filtered by *.
176     - Will attempt to convert the data to Country.
177     info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
178     :c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
179     :c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c
180     :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
181     info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 180 / total: 197 / attempt_modify: 17 / ok: 17
182    
183 joko 1.5 =head2 Synchronizing orm objects against entries in a .csv file - all items in-sync
184 joko 1.2
185 joko 1.4 notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping;
186 joko 1.2 notice: OEF::Script::AbstractFeeder:
187     - Loading data of type [n/a] and
188     filtered by * from the storage named base
189     to the storage named backend - filtered by *.
190     - Will attempt to convert the data to Country.
191     info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
192     :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
193     :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
194     :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
195     info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 197 / total: 197
196 joko 1.1
197    
198    
199     =head1 BUGS
200    
201     When in "import" mode for windows file - DBD::AutoCSV may hang.
202     Hint: Maybe the source node contains an ident-, but no checksum-column?
203    
204    
205     =head1 USER LEVEL ERRORS
206    
207     =head4 Mapping
208    
209     - - - - - - - - - - - - - - - - - - - - - - - - - -
210 joko 1.4 info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: MyApp::Process::Setup->startSync: Can't access mapping for node "Currency" - please check MyApp::ResourceMapping.
211 joko 1.1 - - - - - - - - - - - - - - - - - - - - - - - - - -
212     You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match
213     the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how
214     to access the otherwise generic nodes.
215     - - - - - - - - - - - - - - - - - - - - - - - - - -
216    
217    
218     =head4 DBD::AutoCSV's rulebase
219    
220     - - - - - - - - - - - - - - - - - - - - - - - - - -
221 joko 1.4 info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )
222 joko 1.1 info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
223    
224     Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p
225     m line 165, <GEN9> line 1.
226     called from C:/home/amo/develop/netfrag.org/nfo/perl/libs/Data/Storage/Handler/DBI.pm at 123.
227    
228     DBI-Error: DBD::AutoCSV::st fetchrow_hashref failed: Attempt to fetch row from a Non-SELECT statement
229     notice: Data::Transfer::Sync->syncNodes: No nodes to synchronize.
230     - - - - - - - - - - - - - - - - - - - - - - - - - -
231     DBD::AutoCSV contains a rulebase which is spooled down while attempting to guess the style of the csv file regarding
232     parameters like newline (eol), column-seperation-character (sep_char), quoting character (quote_char).
233     If this spool runs out of entries and no style could be resolved, DBD::CSV dies causing this "Execution ERROR" which
234     results in a "DBI-Error" afterwards.
235     - - - - - - - - - - - - - - - - - - - - - - - - - -
236    
237    
238     =head4 Check structure of source node
239    
240     - - - - - - - - - - - - - - - - - - - - - - - - - -
241     info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
242     critical: Data::Transfer::Sync->syncNodes: Can not synchronize: No ident found in source node, maybe try to "import" this node first.
243     - - - - - - - - - - - - - - - - - - - - - - - - - -
244     If lowlevel detection succeeds, but no other required informations are found, this message is issued.
245     "Other informations" might be:
246     - column-header-row completely missing
247     - ident column is empty
248     - - - - - - - - - - - - - - - - - - - - - - - - - -
249    
250    
251     =head4 Modify structure of source node
252    
253     - - - - - - - - - - - - - - - - - - - - - - - - - -
254     info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
255     info: Data::Transfer::Sync->_prepareNode_MetaProperties( descent source )
256     warning: Data::Transfer::Sync->_prepareNode_MetaProperties: node is lacking meta properties - will try to alter...
257     SQL ERROR: Command 'ALTER' not recognized or not supported!
258    
259     SQL ERROR: Command 'ALTER' not recognized or not supported!
260     - - - - - - - - - - - - - - - - - - - - - - - - - -
261     The Engine found a node which structure does not match the required. It tries to alter this automatically - only when doing "import" -
262     but the DBD driver (in this case DBD::CSV) gets in the way croaking not to be able to do this.
263     This could also appear if your database connection has insufficient rights to modify the database structure.
264     DBD::CSV croaks because it doesn't implement the ALTER command, so please edit your columns manually.
265     Hint: Add columns with the names of your "ident" and "checksum" property specifications.
266     - - - - - - - - - - - - - - - - - - - - - - - - - -
267    
268    
269     =head4 Load source node by ident
270    
271     - - - - - - - - - - - - - - - - - - - - - - - - - -
272     info: Data::Transfer::Sync->_prepareNode_DummyIdent( descent source )
273     pcritical: Data::Transfer::Sync->_modifyNode failed: "source" node is empty.
274     - - - - - - - - - - - - - - - - - - - - - - - - - -
275     The source node could not be loaded. Maybe the ident is missing. Please check manually.
276     Hint: Like above, the ident and/or checksum columns may be missing....
277     - - - - - - - - - - - - - - - - - - - - - - - - - -
278    
279    
280    
281    
282     =cut

MailToCvsAdmin">MailToCvsAdmin
ViewVC Help
Powered by ViewVC 1.1.26 RSS 2.0 feed