/[cvs]/nfo/perl/libs/Data/Transfer/Sync.pod
ViewVC logotype

Annotation of /nfo/perl/libs/Data/Transfer/Sync.pod

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.6 - (hide annotations)
Sat Feb 22 18:25:38 2003 UTC (21 years, 4 months ago) by joko
Branch: MAIN
Changes since 1.5: +8 -1 lines
+ updated NEWS section

1 joko 1.1 =pod
2    
3    
4     =head1 NEWS
5    
6    
7 joko 1.6 =head4 2003-02-22 - enabled multiple task in-process synchronization ...
8    
9     x using Data::Rap and rap.pl
10     x broke sync.pl...
11     o ... re-enable via Data::Transfer::Sync::API::V1?
12    
13    
14 joko 1.2 =head4 2003-02-09 - backward compatibility with custom application
15 joko 1.1
16 joko 1.2 - Tangram <-> .csv
17     - Tangram <-> mysql
18     - Tangram <-> perl-code
19    
20    
21 joko 1.6 =head4 2003-01-18 - started integration with Torus
22 joko 1.2
23     - MAPI <-> LDAP
24 joko 1.3 - please visit http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/
25 joko 1.2
26    
27     =head4 2002-10-10 - initial commit
28    
29     - Tangram <-> mysql
30 joko 1.1
31    
32    
33     =head1 DESCRIPTION
34    
35 joko 1.2 Data::Transfer::Sync is a module providing a generic synchronization process
36     across arbitrary/multiple storages based on a ident/checksum mechanism.
37     It sits on top of Data::Storage.
38    
39 joko 1.1
40    
41     =head1 REQUIREMENTS
42    
43 joko 1.3 =head3 dependencies
44    
45     from CPAN or (for win32) a ppm-repository:
46     App::Config (CPAN only)
47     Tie::IxHash
48     Log::Dispatch
49     Log::Dispatch::Config
50     mixin
51     Hash::Merge
52     Tie::SecureHash
53     Date::Manip
54     Digest::MD5
55     ... and maybe some more.
56    
57     from nfo/perl/libs:
58 joko 1.2 Data::Compare
59 joko 1.1 Data::Storage
60     Data::Transform
61 joko 1.3 DesignPattern:: (to be renamed into Class::)
62     ... and all their dependencies. (like Data::UUID & Co.)
63    
64    
65     =head3 additional modules required for 'outlook2ldap'
66    
67     Win32::OLE
68     Net::LDAP
69    
70     =head3 additional modules required for 'db2db'
71    
72     DBI
73     DBD::mysql
74     DBD::CSV
75     Tangram
76    
77 joko 1.1
78    
79 joko 1.2 =head1 AUTHORS / COPYRIGHT / LICENSE
80 joko 1.1
81 joko 1.4 The Data::Transfer::Sync module is Copyright (c) 2002, 2003 Andreas Motl.
82     All Rights Reserved.
83 joko 1.1
84 joko 1.2 Code contributions, suggestions, support and ideas by
85 joko 1.4 Sebastian Utz <su@tunemedia.de> and Jan Hoffmann <janosch@ultrajan.de>.
86 joko 1.2
87     You may distribute it under the terms of either the GNU General Public
88     License or the Artistic License, as specified in the Perl README file.
89 joko 1.1
90    
91 joko 1.4 =head1 COPYRIGHT NOTE
92    
93     * CAESAR - Collaborative And Easy System ARchitecture
94     *
95     * Data Synchronization Module
96     *
97     * Permission to use, copy, modify and distribute this software and its
98     * documentation is hereby granted, provided that both the copyright
99     * notice and this permission notice appear in all copies of the
100     * software, derivative works or modified versions, and any portions
101     * thereof, and that both notices appear in supporting documentation.
102     *
103     * NETFRAG.ORG AND THE AUTHOR ALLOW FREE USE OF THIS SOFTWARE
104     * IN ITS "AS IS" CONDITION. WE DISCLAIM ANY LIABILITY OF ANY KIND
105     * FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE.
106    
107    
108    
109 joko 1.1 =head1 SUPPORT / WARRANTY
110    
111 joko 1.2 Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.
112    
113     Please feel free to contact me at <andreas.motl@ilo.de>.
114 joko 1.5
115 joko 1.4 Any improvements and contributions are greatly appreciated.
116 joko 1.5
117 joko 1.4 We have set up a newsgroup. (currently not much traffic)
118     Be welcome to visit news://news.netfrag.org/nfo.caesar or - if your system
119 joko 1.5 lacks a newsreader - try http://news.netfrag.org/nfo.caesar .
120 joko 1.4
121    
122    
123    
124     =head1 TODO
125    
126     =head4 overall
127    
128     o try to get this stuff together with SyncML (http://syncml.org/) somehow ...?
129    
130    
131     =head4 this code
132    
133     o sub _resolveIdentProvider
134     x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node
135     x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm
136     o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm
137     x introduce Sync/Compare/MyComparisonImplementation.pm
138     o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing
139     - this implies everything done is _really_ split up into generic actions - how else would we defer them???
140     - example uses:
141     - fetch whole checksum list from node
142     - remember source ident retransmits
143     - remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations
144     o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema)
145     o expand statistics / keep track of:
146     - touched/untouched nodes
147     o full sync
148     - just do a push and a pull for now but use stats for touched nodes in between to speed up things
149     o introduce some new metadata flags for a descent
150     - isNewNodePropagator
151     x isWriteProtected
152 joko 1.2
153    
154    
155     =head1 EXAMPLE OUTPUT CHUNKS
156    
157     =head2 Synchronizing orm objects against entries in a .csv file - 17 items modified
158    
159 joko 1.4 notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping;
160 joko 1.2 notice: OEF::Script::AbstractFeeder:
161     - Loading data of type [n/a] and
162     filtered by * from the storage named base
163     to the storage named backend - filtered by *.
164     - Will attempt to convert the data to Country.
165     info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
166     :c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
167     :c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c
168     :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
169     info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 180 / total: 197 / attempt_modify: 17 / ok: 17
170    
171 joko 1.5 =head2 Synchronizing orm objects against entries in a .csv file - all items in-sync
172 joko 1.2
173 joko 1.4 notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping;
174 joko 1.2 notice: OEF::Script::AbstractFeeder:
175     - Loading data of type [n/a] and
176     filtered by * from the storage named base
177     to the storage named backend - filtered by *.
178     - Will attempt to convert the data to Country.
179     info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
180     :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
181     :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
182     :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
183     info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 197 / total: 197
184 joko 1.1
185    
186    
187     =head1 BUGS
188    
189     When in "import" mode for windows file - DBD::AutoCSV may hang.
190     Hint: Maybe the source node contains an ident-, but no checksum-column?
191    
192    
193     =head1 USER LEVEL ERRORS
194    
195     =head4 Mapping
196    
197     - - - - - - - - - - - - - - - - - - - - - - - - - -
198 joko 1.4 info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: MyApp::Process::Setup->startSync: Can't access mapping for node "Currency" - please check MyApp::ResourceMapping.
199 joko 1.1 - - - - - - - - - - - - - - - - - - - - - - - - - -
200     You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match
201     the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how
202     to access the otherwise generic nodes.
203     - - - - - - - - - - - - - - - - - - - - - - - - - -
204    
205    
206     =head4 DBD::AutoCSV's rulebase
207    
208     - - - - - - - - - - - - - - - - - - - - - - - - - -
209 joko 1.4 info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )
210 joko 1.1 info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
211    
212     Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p
213     m line 165, <GEN9> line 1.
214     called from C:/home/amo/develop/netfrag.org/nfo/perl/libs/Data/Storage/Handler/DBI.pm at 123.
215    
216     DBI-Error: DBD::AutoCSV::st fetchrow_hashref failed: Attempt to fetch row from a Non-SELECT statement
217     notice: Data::Transfer::Sync->syncNodes: No nodes to synchronize.
218     - - - - - - - - - - - - - - - - - - - - - - - - - -
219     DBD::AutoCSV contains a rulebase which is spooled down while attempting to guess the style of the csv file regarding
220     parameters like newline (eol), column-seperation-character (sep_char), quoting character (quote_char).
221     If this spool runs out of entries and no style could be resolved, DBD::CSV dies causing this "Execution ERROR" which
222     results in a "DBI-Error" afterwards.
223     - - - - - - - - - - - - - - - - - - - - - - - - - -
224    
225    
226     =head4 Check structure of source node
227    
228     - - - - - - - - - - - - - - - - - - - - - - - - - -
229     info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
230     critical: Data::Transfer::Sync->syncNodes: Can not synchronize: No ident found in source node, maybe try to "import" this node first.
231     - - - - - - - - - - - - - - - - - - - - - - - - - -
232     If lowlevel detection succeeds, but no other required informations are found, this message is issued.
233     "Other informations" might be:
234     - column-header-row completely missing
235     - ident column is empty
236     - - - - - - - - - - - - - - - - - - - - - - - - - -
237    
238    
239     =head4 Modify structure of source node
240    
241     - - - - - - - - - - - - - - - - - - - - - - - - - -
242     info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
243     info: Data::Transfer::Sync->_prepareNode_MetaProperties( descent source )
244     warning: Data::Transfer::Sync->_prepareNode_MetaProperties: node is lacking meta properties - will try to alter...
245     SQL ERROR: Command 'ALTER' not recognized or not supported!
246    
247     SQL ERROR: Command 'ALTER' not recognized or not supported!
248     - - - - - - - - - - - - - - - - - - - - - - - - - -
249     The Engine found a node which structure does not match the required. It tries to alter this automatically - only when doing "import" -
250     but the DBD driver (in this case DBD::CSV) gets in the way croaking not to be able to do this.
251     This could also appear if your database connection has insufficient rights to modify the database structure.
252     DBD::CSV croaks because it doesn't implement the ALTER command, so please edit your columns manually.
253     Hint: Add columns with the names of your "ident" and "checksum" property specifications.
254     - - - - - - - - - - - - - - - - - - - - - - - - - -
255    
256    
257     =head4 Load source node by ident
258    
259     - - - - - - - - - - - - - - - - - - - - - - - - - -
260     info: Data::Transfer::Sync->_prepareNode_DummyIdent( descent source )
261     pcritical: Data::Transfer::Sync->_modifyNode failed: "source" node is empty.
262     - - - - - - - - - - - - - - - - - - - - - - - - - -
263     The source node could not be loaded. Maybe the ident is missing. Please check manually.
264     Hint: Like above, the ident and/or checksum columns may be missing....
265     - - - - - - - - - - - - - - - - - - - - - - - - - -
266    
267    
268    
269    
270     =cut

MailToCvsAdmin">MailToCvsAdmin
ViewVC Help
Powered by ViewVC 1.1.26 RSS 2.0 feed