Data/Transfer/Sync.pod

=pod


=head1 NEWS


=head4 2003-02-09  -  backward compatibility with custom application

  - Tangram <-> .csv
  - Tangram <-> mysql
  - Tangram <-> perl-code


=head4 2003-01-18  -  start integration with Torus

  - MAPI <-> LDAP
  - please visit http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/


=head4 2002-10-10  -  initial commit

  - Tangram <-> mysql


=head1 DESCRIPTION

  Data::Transfer::Sync is a module providing a generic synchronization process
  across arbitrary/multiple storages based on a ident/checksum mechanism.
  It sits on top of Data::Storage.


=head1 REQUIREMENTS

=head3 dependencies

  from CPAN or (for win32) a ppm-repository:
    App::Config (CPAN only)
    Tie::IxHash
    Log::Dispatch
    Log::Dispatch::Config
    mixin
    Hash::Merge
    Tie::SecureHash
    Date::Manip
    Digest::MD5
    ... and maybe some more.

  from nfo/perl/libs:
    Data::Compare
    Data::Storage
    Data::Transform
    DesignPattern:: (to be renamed into Class::)
    ... and all their dependencies. (like Data::UUID & Co.)


=head3 additional modules required for 'outlook2ldap'

    Win32::OLE
    Net::LDAP

=head3 additional modules required for 'db2db'

    DBI
    DBD::mysql
    DBD::CSV
    Tangram


=head1 AUTHORS / COPYRIGHT / LICENSE

  The Data::Transfer::Sync module is Copyright (c) 2002 Andreas Motl.
  All rights reserved.

  Code contributions, suggestions, support and ideas by
  Sebastian Utz <su@tunemedia.de> and Jan Hoffmann <jan@romain-volk.com>.
  
  You may distribute it under the terms of either the GNU General Public
  License or the Artistic License, as specified in the Perl README file.


=head1 SUPPORT / WARRANTY

  Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.

  Please feel free to contact me at <andreas.motl@ilo.de>.


=head1 EXAMPLE OUTPUT CHUNKS

=head2 Synchronizing orm objects against entries in a .csv file - 17 items modified

  notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping;
  notice: OEF::Script::AbstractFeeder:
      - Loading data of type [n/a] and
        filtered by * from the storage named base
        to the storage named backend - filtered by *.
      - Will attempt to convert the data to Country.
  info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
  :c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
  :c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c
  :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
  info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 180  /  total: 197  /  attempt_modify: 17  /  ok: 17

=head2 Synchronizing orm objects against entries in a .csv file - in-sync
  
  notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping;
  notice: OEF::Script::AbstractFeeder:
      - Loading data of type [n/a] and
        filtered by * from the storage named base
        to the storage named backend - filtered by *.
      - Will attempt to convert the data to Country.
  info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
  :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
  :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
  :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
  info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 197  /  total: 197


=head1 BUGS

  When in "import" mode for windows file - DBD::AutoCSV may hang.
  Hint: Maybe the source node contains an ident-, but no checksum-column?


=head1 USER LEVEL ERRORS

=head4 Mapping

  - - - - - - - - - - - - - - - - - - - - - - - - - -
  info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: BizWorks::Process::Setup->startSync: Can't access mapping for node "Currency" - please check BizWorks::ResourceMapping.
  - - - - - - - - - - - - - - - - - - - - - - - - - -
  You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match
  the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how 
  to access the otherwise generic nodes.
  - - - - - - - - - - - - - - - - - - - - - - - - - -


=head4 DBD::AutoCSV's rulebase
  
  - - - - - - - - - - - - - - - - - - - - - - - - - -
  info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )
  info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
  
  Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p
  m line 165, <GEN9> line 1.
   called from C:/home/amo/develop/netfrag.org/nfo/perl/libs/Data/Storage/Handler/DBI.pm at 123.
  
  DBI-Error: DBD::AutoCSV::st fetchrow_hashref failed: Attempt to fetch row from a Non-SELECT statement
  notice: Data::Transfer::Sync->syncNodes: No nodes to synchronize.
  - - - - - - - - - - - - - - - - - - - - - - - - - -
  DBD::AutoCSV contains a rulebase which is spooled down while attempting to guess the style of the csv file regarding
  parameters like newline (eol), column-seperation-character (sep_char), quoting character (quote_char).
  If this spool runs out of entries and no style could be resolved, DBD::CSV dies causing this "Execution ERROR" which
  results in a "DBI-Error" afterwards.
  - - - - - - - - - - - - - - - - - - - - - - - - - -


=head4 Check structure of source node
  
  - - - - - - - - - - - - - - - - - - - - - - - - - -
  info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
  critical: Data::Transfer::Sync->syncNodes: Can not synchronize: No ident found in source node, maybe try to "import" this node first.
  - - - - - - - - - - - - - - - - - - - - - - - - - -
  If lowlevel detection succeeds, but no other required informations are found, this message is issued.
  "Other informations" might be:
    - column-header-row completely missing
    - ident column is empty
  - - - - - - - - - - - - - - - - - - - - - - - - - -


=head4 Modify structure of source node
  
  - - - - - - - - - - - - - - - - - - - - - - - - - -
  info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
  info: Data::Transfer::Sync->_prepareNode_MetaProperties( descent source )
  warning: Data::Transfer::Sync->_prepareNode_MetaProperties: node is lacking meta properties - will try to alter...
  SQL ERROR: Command 'ALTER' not recognized or not supported!
  
  SQL ERROR: Command 'ALTER' not recognized or not supported!
  - - - - - - - - - - - - - - - - - - - - - - - - - -
  The Engine found a node which structure does not match the required. It tries to alter this automatically - only when doing "import" - 
  but the DBD driver (in this case DBD::CSV) gets in the way croaking not to be able to do this.
  This could also appear if your database connection has insufficient rights to modify the database structure.
  DBD::CSV croaks because it doesn't implement the ALTER command, so please edit your columns manually.
  Hint: Add columns with the names of your "ident" and "checksum" property specifications.
  - - - - - - - - - - - - - - - - - - - - - - - - - -


=head4 Load source node by ident

  - - - - - - - - - - - - - - - - - - - - - - - - - -
  info: Data::Transfer::Sync->_prepareNode_DummyIdent( descent source )
  pcritical: Data::Transfer::Sync->_modifyNode failed: "source" node is empty.
  - - - - - - - - - - - - - - - - - - - - - - - - - -
  The source node could not be loaded. Maybe the ident is missing. Please check manually.
  Hint: Like above, the ident and/or checksum columns may be missing....
  - - - - - - - - - - - - - - - - - - - - - - - - - -


=head1 TODO

  o sub _resolveIdentProvider
  x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node
  x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm
  o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm
  x introduce Sync/Compare/MyComparisonImplementation.pm
  o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing
     - this implies everything done is _really_ split up into generic actions - how else would we defer them???
     - example uses:
        - fetch whole checksum list from node
        - remember source ident retransmits
     - remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations
  o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema)
  o expand statistics / keep track of:
    - touched/untouched nodes
  o full sync
    - just do a push and a pull for now but use stats for touched nodes in between to speed up things
  o introduce some new metadata flags for a descent
    - isNewNodePropagator
    x isWriteProtected


=cut
1	joko	1.1	=pod
2
3
4			=head1 NEWS
5
6
7	joko	1.2	=head4 2003-02-09 - backward compatibility with custom application
8	joko	1.1
9	joko	1.2	- Tangram <-> .csv
10			- Tangram <-> mysql
11			- Tangram <-> perl-code
12
13
14			=head4 2003-01-18 - start integration with Torus
15
16			- MAPI <-> LDAP
17	joko	1.3	- please visit http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/
18	joko	1.2
19
20			=head4 2002-10-10 - initial commit
21
22			- Tangram <-> mysql
23	joko	1.1
24
25
26			=head1 DESCRIPTION
27
28	joko	1.2	Data::Transfer::Sync is a module providing a generic synchronization process
29			across arbitrary/multiple storages based on a ident/checksum mechanism.
30			It sits on top of Data::Storage.
31
32	joko	1.1
33
34			=head1 REQUIREMENTS
35
36	joko	1.3	=head3 dependencies
37
38			from CPAN or (for win32) a ppm-repository:
39			App::Config (CPAN only)
40			Tie::IxHash
41			Log::Dispatch
42			Log::Dispatch::Config
43			mixin
44			Hash::Merge
45			Tie::SecureHash
46			Date::Manip
47			Digest::MD5
48			... and maybe some more.
49
50			from nfo/perl/libs:
51	joko	1.2	Data::Compare
52	joko	1.1	Data::Storage
53			Data::Transform
54	joko	1.3	DesignPattern:: (to be renamed into Class::)
55			... and all their dependencies. (like Data::UUID & Co.)
56
57
58			=head3 additional modules required for 'outlook2ldap'
59
60			Win32::OLE
61			Net::LDAP
62
63			=head3 additional modules required for 'db2db'
64
65			DBI
66			DBD::mysql
67			DBD::CSV
68			Tangram
69
70	joko	1.1
71
72	joko	1.2	=head1 AUTHORS / COPYRIGHT / LICENSE
73	joko	1.1
74	joko	1.2	The Data::Transfer::Sync module is Copyright (c) 2002 Andreas Motl.
75			All rights reserved.
76	joko	1.1
77	joko	1.2	Code contributions, suggestions, support and ideas by
78			Sebastian Utz <su@tunemedia.de> and Jan Hoffmann <jan@romain-volk.com>.
79
80			You may distribute it under the terms of either the GNU General Public
81			License or the Artistic License, as specified in the Perl README file.
82	joko	1.1
83
84			=head1 SUPPORT / WARRANTY
85
86	joko	1.2	Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.
87
88			Please feel free to contact me at <andreas.motl@ilo.de>.
89
90
91
92			=head1 EXAMPLE OUTPUT CHUNKS
93
94			=head2 Synchronizing orm objects against entries in a .csv file - 17 items modified
95
96			notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping;
97			notice: OEF::Script::AbstractFeeder:
98			- Loading data of type [n/a] and
99			filtered by * from the storage named base
100			to the storage named backend - filtered by *.
101			- Will attempt to convert the data to Country.
102			info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
103			:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
104			:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c
105			:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
106			info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 180 / total: 197 / attempt_modify: 17 / ok: 17
107
108			=head2 Synchronizing orm objects against entries in a .csv file - in-sync
109
110			notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping;
111			notice: OEF::Script::AbstractFeeder:
112			- Loading data of type [n/a] and
113			filtered by * from the storage named base
114			to the storage named backend - filtered by *.
115			- Will attempt to convert the data to Country.
116			info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
117			:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
118			:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
119			:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
120			info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 197 / total: 197
121	joko	1.1
122
123
124			=head1 BUGS
125
126			When in "import" mode for windows file - DBD::AutoCSV may hang.
127			Hint: Maybe the source node contains an ident-, but no checksum-column?
128
129
130			=head1 USER LEVEL ERRORS
131
132			=head4 Mapping
133
134			- - - - - - - - - - - - - - - - - - - - - - - - - -
135			info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: BizWorks::Process::Setup->startSync: Can't access mapping for node "Currency" - please check BizWorks::ResourceMapping.
136			- - - - - - - - - - - - - - - - - - - - - - - - - -
137			You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match
138			the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how
139			to access the otherwise generic nodes.
140			- - - - - - - - - - - - - - - - - - - - - - - - - -
141
142
143			=head4 DBD::AutoCSV's rulebase
144
145			- - - - - - - - - - - - - - - - - - - - - - - - - -
146			info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )
147			info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
148
149			Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p
150			m line 165, <GEN9> line 1.
151			called from C:/home/amo/develop/netfrag.org/nfo/perl/libs/Data/Storage/Handler/DBI.pm at 123.
152
153			DBI-Error: DBD::AutoCSV::st fetchrow_hashref failed: Attempt to fetch row from a Non-SELECT statement
154			notice: Data::Transfer::Sync->syncNodes: No nodes to synchronize.
155			- - - - - - - - - - - - - - - - - - - - - - - - - -
156			DBD::AutoCSV contains a rulebase which is spooled down while attempting to guess the style of the csv file regarding
157			parameters like newline (eol), column-seperation-character (sep_char), quoting character (quote_char).
158			If this spool runs out of entries and no style could be resolved, DBD::CSV dies causing this "Execution ERROR" which
159			results in a "DBI-Error" afterwards.
160			- - - - - - - - - - - - - - - - - - - - - - - - - -
161
162
163			=head4 Check structure of source node
164
165			- - - - - - - - - - - - - - - - - - - - - - - - - -
166			info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
167			critical: Data::Transfer::Sync->syncNodes: Can not synchronize: No ident found in source node, maybe try to "import" this node first.
168			- - - - - - - - - - - - - - - - - - - - - - - - - -
169			If lowlevel detection succeeds, but no other required informations are found, this message is issued.
170			"Other informations" might be:
171			- column-header-row completely missing
172			- ident column is empty
173			- - - - - - - - - - - - - - - - - - - - - - - - - -
174
175
176			=head4 Modify structure of source node
177
178			- - - - - - - - - - - - - - - - - - - - - - - - - -
179			info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
180			info: Data::Transfer::Sync->_prepareNode_MetaProperties( descent source )
181			warning: Data::Transfer::Sync->_prepareNode_MetaProperties: node is lacking meta properties - will try to alter...
182			SQL ERROR: Command 'ALTER' not recognized or not supported!
183
184			SQL ERROR: Command 'ALTER' not recognized or not supported!
185			- - - - - - - - - - - - - - - - - - - - - - - - - -
186			The Engine found a node which structure does not match the required. It tries to alter this automatically - only when doing "import" -
187			but the DBD driver (in this case DBD::CSV) gets in the way croaking not to be able to do this.
188			This could also appear if your database connection has insufficient rights to modify the database structure.
189			DBD::CSV croaks because it doesn't implement the ALTER command, so please edit your columns manually.
190			Hint: Add columns with the names of your "ident" and "checksum" property specifications.
191			- - - - - - - - - - - - - - - - - - - - - - - - - -
192
193
194			=head4 Load source node by ident
195
196			- - - - - - - - - - - - - - - - - - - - - - - - - -
197			info: Data::Transfer::Sync->_prepareNode_DummyIdent( descent source )
198			pcritical: Data::Transfer::Sync->_modifyNode failed: "source" node is empty.
199			- - - - - - - - - - - - - - - - - - - - - - - - - -
200			The source node could not be loaded. Maybe the ident is missing. Please check manually.
201			Hint: Like above, the ident and/or checksum columns may be missing....
202			- - - - - - - - - - - - - - - - - - - - - - - - - -
203
204
205			=head1 TODO
206
207	joko	1.2	o sub _resolveIdentProvider
208			x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node
209			x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm
210			o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm
211			x introduce Sync/Compare/MyComparisonImplementation.pm
212			o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing
213	joko	1.1	- this implies everything done is _really_ split up into generic actions - how else would we defer them???
214			- example uses:
215			- fetch whole checksum list from node
216			- remember source ident retransmits
217			- remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations
218	joko	1.2	o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema)
219			o expand statistics / keep track of:
220	joko	1.1	- touched/untouched nodes
221	joko	1.2	o full sync
222	joko	1.1	- just do a push and a pull for now but use stats for touched nodes in between to speed up things
223	joko	1.2	o introduce some new metadata flags for a descent
224	joko	1.1	- isNewNodePropagator
225	joko	1.2	x isWriteProtected
226	joko	1.1
227
228			=cut
MailToCvsAdmin">MailToCvsAdmin	ViewVC Help
Powered by ViewVC 1.1.26