/[cvs]/nfo/perl/libs/Data/Transfer/Sync.pod
ViewVC logotype

Annotation of /nfo/perl/libs/Data/Transfer/Sync.pod

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.5 - (hide annotations)
Tue Feb 11 10:28:03 2003 UTC (21 years, 4 months ago) by joko
Branch: MAIN
Changes since 1.4: +4 -4 lines
+ minor update

1 joko 1.1 =pod
2    
3    
4     =head1 NEWS
5    
6    
7 joko 1.2 =head4 2003-02-09 - backward compatibility with custom application
8 joko 1.1
9 joko 1.2 - Tangram <-> .csv
10     - Tangram <-> mysql
11     - Tangram <-> perl-code
12    
13    
14     =head4 2003-01-18 - start integration with Torus
15    
16     - MAPI <-> LDAP
17 joko 1.3 - please visit http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/
18 joko 1.2
19    
20     =head4 2002-10-10 - initial commit
21    
22     - Tangram <-> mysql
23 joko 1.1
24    
25    
26     =head1 DESCRIPTION
27    
28 joko 1.2 Data::Transfer::Sync is a module providing a generic synchronization process
29     across arbitrary/multiple storages based on a ident/checksum mechanism.
30     It sits on top of Data::Storage.
31    
32 joko 1.1
33    
34     =head1 REQUIREMENTS
35    
36 joko 1.3 =head3 dependencies
37    
38     from CPAN or (for win32) a ppm-repository:
39     App::Config (CPAN only)
40     Tie::IxHash
41     Log::Dispatch
42     Log::Dispatch::Config
43     mixin
44     Hash::Merge
45     Tie::SecureHash
46     Date::Manip
47     Digest::MD5
48     ... and maybe some more.
49    
50     from nfo/perl/libs:
51 joko 1.2 Data::Compare
52 joko 1.1 Data::Storage
53     Data::Transform
54 joko 1.3 DesignPattern:: (to be renamed into Class::)
55     ... and all their dependencies. (like Data::UUID & Co.)
56    
57    
58     =head3 additional modules required for 'outlook2ldap'
59    
60     Win32::OLE
61     Net::LDAP
62    
63     =head3 additional modules required for 'db2db'
64    
65     DBI
66     DBD::mysql
67     DBD::CSV
68     Tangram
69    
70 joko 1.1
71    
72 joko 1.2 =head1 AUTHORS / COPYRIGHT / LICENSE
73 joko 1.1
74 joko 1.4 The Data::Transfer::Sync module is Copyright (c) 2002, 2003 Andreas Motl.
75     All Rights Reserved.
76 joko 1.1
77 joko 1.2 Code contributions, suggestions, support and ideas by
78 joko 1.4 Sebastian Utz <su@tunemedia.de> and Jan Hoffmann <janosch@ultrajan.de>.
79 joko 1.2
80     You may distribute it under the terms of either the GNU General Public
81     License or the Artistic License, as specified in the Perl README file.
82 joko 1.1
83    
84 joko 1.4 =head1 COPYRIGHT NOTE
85    
86     * CAESAR - Collaborative And Easy System ARchitecture
87     *
88     * Data Synchronization Module
89     *
90     * Permission to use, copy, modify and distribute this software and its
91     * documentation is hereby granted, provided that both the copyright
92     * notice and this permission notice appear in all copies of the
93     * software, derivative works or modified versions, and any portions
94     * thereof, and that both notices appear in supporting documentation.
95     *
96     * NETFRAG.ORG AND THE AUTHOR ALLOW FREE USE OF THIS SOFTWARE
97     * IN ITS "AS IS" CONDITION. WE DISCLAIM ANY LIABILITY OF ANY KIND
98     * FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE.
99    
100    
101    
102 joko 1.1 =head1 SUPPORT / WARRANTY
103    
104 joko 1.2 Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.
105    
106     Please feel free to contact me at <andreas.motl@ilo.de>.
107 joko 1.5
108 joko 1.4 Any improvements and contributions are greatly appreciated.
109 joko 1.5
110 joko 1.4 We have set up a newsgroup. (currently not much traffic)
111     Be welcome to visit news://news.netfrag.org/nfo.caesar or - if your system
112 joko 1.5 lacks a newsreader - try http://news.netfrag.org/nfo.caesar .
113 joko 1.4
114    
115    
116    
117     =head1 TODO
118    
119     =head4 overall
120    
121     o try to get this stuff together with SyncML (http://syncml.org/) somehow ...?
122    
123    
124     =head4 this code
125    
126     o sub _resolveIdentProvider
127     x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node
128     x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm
129     o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm
130     x introduce Sync/Compare/MyComparisonImplementation.pm
131     o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing
132     - this implies everything done is _really_ split up into generic actions - how else would we defer them???
133     - example uses:
134     - fetch whole checksum list from node
135     - remember source ident retransmits
136     - remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations
137     o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema)
138     o expand statistics / keep track of:
139     - touched/untouched nodes
140     o full sync
141     - just do a push and a pull for now but use stats for touched nodes in between to speed up things
142     o introduce some new metadata flags for a descent
143     - isNewNodePropagator
144     x isWriteProtected
145 joko 1.2
146    
147    
148     =head1 EXAMPLE OUTPUT CHUNKS
149    
150     =head2 Synchronizing orm objects against entries in a .csv file - 17 items modified
151    
152 joko 1.4 notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping;
153 joko 1.2 notice: OEF::Script::AbstractFeeder:
154     - Loading data of type [n/a] and
155     filtered by * from the storage named base
156     to the storage named backend - filtered by *.
157     - Will attempt to convert the data to Country.
158     info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
159     :c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
160     :c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c
161     :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
162     info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 180 / total: 197 / attempt_modify: 17 / ok: 17
163    
164 joko 1.5 =head2 Synchronizing orm objects against entries in a .csv file - all items in-sync
165 joko 1.2
166 joko 1.4 notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE MyApp::ResourceMapping;
167 joko 1.2 notice: OEF::Script::AbstractFeeder:
168     - Loading data of type [n/a] and
169     filtered by * from the storage named base
170     to the storage named backend - filtered by *.
171     - Will attempt to convert the data to Country.
172     info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
173     :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
174     :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
175     :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
176     info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 197 / total: 197
177 joko 1.1
178    
179    
180     =head1 BUGS
181    
182     When in "import" mode for windows file - DBD::AutoCSV may hang.
183     Hint: Maybe the source node contains an ident-, but no checksum-column?
184    
185    
186     =head1 USER LEVEL ERRORS
187    
188     =head4 Mapping
189    
190     - - - - - - - - - - - - - - - - - - - - - - - - - -
191 joko 1.4 info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: MyApp::Process::Setup->startSync: Can't access mapping for node "Currency" - please check MyApp::ResourceMapping.
192 joko 1.1 - - - - - - - - - - - - - - - - - - - - - - - - - -
193     You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match
194     the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how
195     to access the otherwise generic nodes.
196     - - - - - - - - - - - - - - - - - - - - - - - - - -
197    
198    
199     =head4 DBD::AutoCSV's rulebase
200    
201     - - - - - - - - - - - - - - - - - - - - - - - - - -
202 joko 1.4 info: MyApp::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )
203 joko 1.1 info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
204    
205     Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p
206     m line 165, <GEN9> line 1.
207     called from C:/home/amo/develop/netfrag.org/nfo/perl/libs/Data/Storage/Handler/DBI.pm at 123.
208    
209     DBI-Error: DBD::AutoCSV::st fetchrow_hashref failed: Attempt to fetch row from a Non-SELECT statement
210     notice: Data::Transfer::Sync->syncNodes: No nodes to synchronize.
211     - - - - - - - - - - - - - - - - - - - - - - - - - -
212     DBD::AutoCSV contains a rulebase which is spooled down while attempting to guess the style of the csv file regarding
213     parameters like newline (eol), column-seperation-character (sep_char), quoting character (quote_char).
214     If this spool runs out of entries and no style could be resolved, DBD::CSV dies causing this "Execution ERROR" which
215     results in a "DBI-Error" afterwards.
216     - - - - - - - - - - - - - - - - - - - - - - - - - -
217    
218    
219     =head4 Check structure of source node
220    
221     - - - - - - - - - - - - - - - - - - - - - - - - - -
222     info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
223     critical: Data::Transfer::Sync->syncNodes: Can not synchronize: No ident found in source node, maybe try to "import" this node first.
224     - - - - - - - - - - - - - - - - - - - - - - - - - -
225     If lowlevel detection succeeds, but no other required informations are found, this message is issued.
226     "Other informations" might be:
227     - column-header-row completely missing
228     - ident column is empty
229     - - - - - - - - - - - - - - - - - - - - - - - - - -
230    
231    
232     =head4 Modify structure of source node
233    
234     - - - - - - - - - - - - - - - - - - - - - - - - - -
235     info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
236     info: Data::Transfer::Sync->_prepareNode_MetaProperties( descent source )
237     warning: Data::Transfer::Sync->_prepareNode_MetaProperties: node is lacking meta properties - will try to alter...
238     SQL ERROR: Command 'ALTER' not recognized or not supported!
239    
240     SQL ERROR: Command 'ALTER' not recognized or not supported!
241     - - - - - - - - - - - - - - - - - - - - - - - - - -
242     The Engine found a node which structure does not match the required. It tries to alter this automatically - only when doing "import" -
243     but the DBD driver (in this case DBD::CSV) gets in the way croaking not to be able to do this.
244     This could also appear if your database connection has insufficient rights to modify the database structure.
245     DBD::CSV croaks because it doesn't implement the ALTER command, so please edit your columns manually.
246     Hint: Add columns with the names of your "ident" and "checksum" property specifications.
247     - - - - - - - - - - - - - - - - - - - - - - - - - -
248    
249    
250     =head4 Load source node by ident
251    
252     - - - - - - - - - - - - - - - - - - - - - - - - - -
253     info: Data::Transfer::Sync->_prepareNode_DummyIdent( descent source )
254     pcritical: Data::Transfer::Sync->_modifyNode failed: "source" node is empty.
255     - - - - - - - - - - - - - - - - - - - - - - - - - -
256     The source node could not be loaded. Maybe the ident is missing. Please check manually.
257     Hint: Like above, the ident and/or checksum columns may be missing....
258     - - - - - - - - - - - - - - - - - - - - - - - - - -
259    
260    
261    
262    
263     =cut

MailToCvsAdmin">MailToCvsAdmin
ViewVC Help
Powered by ViewVC 1.1.26 RSS 2.0 feed