1 |
joko |
1.3 |
NEWS |
2 |
joko |
1.4 |
2003-02-09 - backward compatibility with custom application |
3 |
joko |
1.3 |
|
4 |
joko |
1.4 |
- Tangram <-> .csv |
5 |
|
|
- Tangram <-> mysql |
6 |
|
|
- Tangram <-> perl-code |
7 |
joko |
1.3 |
|
8 |
joko |
1.4 |
2003-01-18 - start integration with Torus |
9 |
|
|
|
10 |
|
|
- MAPI <-> LDAP |
11 |
|
|
- please look at http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/ |
12 |
|
|
|
13 |
|
|
2002-10-10 - initial commit |
14 |
|
|
|
15 |
|
|
- Tangram <-> mysql |
16 |
joko |
1.3 |
|
17 |
joko |
1.1 |
DESCRIPTION |
18 |
joko |
1.4 |
Data::Transfer::Sync is a module providing a generic synchronization process |
19 |
|
|
across arbitrary/multiple storages based on a ident/checksum mechanism. |
20 |
|
|
It sits on top of Data::Storage. |
21 |
joko |
1.1 |
|
22 |
|
|
REQUIREMENTS |
23 |
|
|
For full functionality: |
24 |
joko |
1.4 |
Data::Compare |
25 |
joko |
1.1 |
Data::Storage |
26 |
|
|
Data::Transform |
27 |
joko |
1.4 |
... and all their dependencies. |
28 |
|
|
|
29 |
|
|
AUTHORS / COPYRIGHT / LICENSE |
30 |
|
|
The Data::Transfer::Sync module is Copyright (c) 2002 Andreas Motl. |
31 |
|
|
All rights reserved. |
32 |
joko |
1.1 |
|
33 |
joko |
1.4 |
Contributions by Sebastian Utz <su@tunemedia.de> and |
34 |
|
|
Jan Hoffmann <jan@romain-volk.com>. |
35 |
joko |
1.1 |
|
36 |
joko |
1.4 |
You may distribute it under the terms of either the GNU General Public |
37 |
|
|
License or the Artistic License, as specified in the Perl README file. |
38 |
joko |
1.1 |
|
39 |
|
|
SUPPORT / WARRANTY |
40 |
joko |
1.4 |
Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND. |
41 |
|
|
|
42 |
|
|
EXAMPLE OUTPUT CHUNKS |
43 |
|
|
synchronizing orm objects against entries in a .csv file - 17 items modified |
44 |
|
|
|
45 |
|
|
notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping; |
46 |
|
|
notice: OEF::Script::AbstractFeeder: |
47 |
|
|
- Loading data of type [n/a] and |
48 |
|
|
filtered by * from the storage named base |
49 |
|
|
to the storage named backend - filtered by *. |
50 |
|
|
- Will attempt to convert the data to Country. |
51 |
|
|
info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* [] |
52 |
|
|
:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
53 |
|
|
:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c |
54 |
|
|
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
55 |
|
|
info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 180 / total: 197 / attempt_modify: 17 / ok: 17 |
56 |
|
|
|
57 |
|
|
synchronizing orm objects against entries in a .csv file - in-sync |
58 |
|
|
|
59 |
|
|
notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping; |
60 |
|
|
notice: OEF::Script::AbstractFeeder: |
61 |
|
|
- Loading data of type [n/a] and |
62 |
|
|
filtered by * from the storage named base |
63 |
|
|
to the storage named backend - filtered by *. |
64 |
|
|
- Will attempt to convert the data to Country. |
65 |
|
|
info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* [] |
66 |
|
|
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
67 |
|
|
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
68 |
|
|
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
69 |
|
|
info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 197 / total: 197 |
70 |
joko |
1.1 |
|
71 |
|
|
BUGS |
72 |
|
|
When in "import" mode for windows file - DBD::AutoCSV may hang. |
73 |
joko |
1.3 |
Hint: Maybe the source node contains an ident-, but no checksum-column? |
74 |
joko |
1.1 |
|
75 |
|
|
USER LEVEL ERRORS |
76 |
|
|
Mapping |
77 |
|
|
|
78 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
79 |
|
|
info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: BizWorks::Process::Setup->startSync: Can't access mapping for node "Currency" - please check BizWorks::ResourceMapping. |
80 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
81 |
|
|
You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match |
82 |
|
|
the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how |
83 |
|
|
to access the otherwise generic nodes. |
84 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
85 |
|
|
|
86 |
|
|
DBD::AutoCSV's rulebase |
87 |
|
|
|
88 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
89 |
|
|
info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 ) |
90 |
|
|
info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv |
91 |
|
|
|
92 |
|
|
Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p |
93 |
|
|
m line 165, <GEN9> line 1. |
94 |
|
|
called from C:/home/amo/develop/netfrag.org/nfo/perl/libs/Data/Storage/Handler/DBI.pm at 123. |
95 |
|
|
|
96 |
|
|
DBI-Error: DBD::AutoCSV::st fetchrow_hashref failed: Attempt to fetch row from a Non-SELECT statement |
97 |
|
|
notice: Data::Transfer::Sync->syncNodes: No nodes to synchronize. |
98 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
99 |
|
|
DBD::AutoCSV contains a rulebase which is spooled down while attempting to guess the style of the csv file regarding |
100 |
|
|
parameters like newline (eol), column-seperation-character (sep_char), quoting character (quote_char). |
101 |
|
|
If this spool runs out of entries and no style could be resolved, DBD::CSV dies causing this "Execution ERROR" which |
102 |
|
|
results in a "DBI-Error" afterwards. |
103 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
104 |
|
|
|
105 |
|
|
Check structure of source node |
106 |
|
|
|
107 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
108 |
|
|
info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv |
109 |
|
|
critical: Data::Transfer::Sync->syncNodes: Can not synchronize: No ident found in source node, maybe try to "import" this node first. |
110 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
111 |
|
|
If lowlevel detection succeeds, but no other required informations are found, this message is issued. |
112 |
|
|
"Other informations" might be: |
113 |
|
|
- column-header-row completely missing |
114 |
|
|
- ident column is empty |
115 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
116 |
|
|
|
117 |
|
|
Modify structure of source node |
118 |
|
|
|
119 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
120 |
|
|
info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv |
121 |
|
|
info: Data::Transfer::Sync->_prepareNode_MetaProperties( descent source ) |
122 |
|
|
warning: Data::Transfer::Sync->_prepareNode_MetaProperties: node is lacking meta properties - will try to alter... |
123 |
|
|
SQL ERROR: Command 'ALTER' not recognized or not supported! |
124 |
|
|
|
125 |
|
|
SQL ERROR: Command 'ALTER' not recognized or not supported! |
126 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
127 |
|
|
The Engine found a node which structure does not match the required. It tries to alter this automatically - only when doing "import" - |
128 |
|
|
but the DBD driver (in this case DBD::CSV) gets in the way croaking not to be able to do this. |
129 |
|
|
This could also appear if your database connection has insufficient rights to modify the database structure. |
130 |
|
|
DBD::CSV croaks because it doesn't implement the ALTER command, so please edit your columns manually. |
131 |
|
|
Hint: Add columns with the names of your "ident" and "checksum" property specifications. |
132 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
133 |
|
|
|
134 |
|
|
Load source node by ident |
135 |
|
|
|
136 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
137 |
|
|
info: Data::Transfer::Sync->_prepareNode_DummyIdent( descent source ) |
138 |
|
|
pcritical: Data::Transfer::Sync->_modifyNode failed: "source" node is empty. |
139 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
140 |
|
|
The source node could not be loaded. Maybe the ident is missing. Please check manually. |
141 |
|
|
Hint: Like above, the ident and/or checksum columns may be missing.... |
142 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
143 |
|
|
|
144 |
|
|
TODO |
145 |
joko |
1.4 |
o sub _resolveIdentProvider |
146 |
|
|
x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node |
147 |
|
|
x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm |
148 |
|
|
o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm |
149 |
|
|
x introduce Sync/Compare/MyComparisonImplementation.pm |
150 |
|
|
o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing |
151 |
joko |
1.1 |
- this implies everything done is _really_ split up into generic actions - how else would we defer them??? |
152 |
|
|
- example uses: |
153 |
|
|
- fetch whole checksum list from node |
154 |
|
|
- remember source ident retransmits |
155 |
|
|
- remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations |
156 |
joko |
1.4 |
o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema) |
157 |
|
|
o expand statistics / keep track of: |
158 |
joko |
1.1 |
- touched/untouched nodes |
159 |
joko |
1.4 |
o full sync |
160 |
joko |
1.1 |
- just do a push and a pull for now but use stats for touched nodes in between to speed up things |
161 |
joko |
1.4 |
o introduce some new metadata flags for a descent |
162 |
joko |
1.1 |
- isNewNodePropagator |
163 |
joko |
1.4 |
x isWriteProtected |
164 |
joko |
1.3 |
|