Data/Transfer/Sync.README.text

NEWS
   2003-02-09  -  backward compatibility with custom application

      - Tangram <-> .csv
      - Tangram <-> mysql
      - Tangram <-> perl-code

   2003-01-18  -  start integration with Torus

      - MAPI <-> LDAP
      - please visit http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/

   2002-10-10  -  initial commit

      - Tangram <-> mysql

DESCRIPTION
      Data::Transfer::Sync is a module providing a generic synchronization process
      across arbitrary/multiple storages based on a ident/checksum mechanism.
      It sits on top of Data::Storage.

REQUIREMENTS
   dependencies

      from CPAN or (for win32) a ppm-repository:
        App::Config (CPAN only)
        Tie::IxHash
        Log::Dispatch
        Log::Dispatch::Config
        mixin
        Hash::Merge
        Tie::SecureHash
        Date::Manip

      from nfo/perl/libs:
        Data::Compare
        Data::Storage
        Data::Transform
        DesignPattern:: (to be renamed into Class::)
        ... and all their dependencies.

   additional modules required for 'outlook2ldap'

        Win32::OLE
        Net::LDAP

   additional modules required for 'db2db'

        DBI
        DBD::mysql
        DBD::CSV
        Tangram

AUTHORS / COPYRIGHT / LICENSE
      The Data::Transfer::Sync module is Copyright (c) 2002 Andreas Motl.
      All rights reserved.

      Code contributions, suggestions, support and ideas by
      Sebastian Utz <su@tunemedia.de> and Jan Hoffmann <jan@romain-volk.com>.
  
      You may distribute it under the terms of either the GNU General Public
      License or the Artistic License, as specified in the Perl README file.

SUPPORT / WARRANTY
      Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.

      Please feel free to contact me at <andreas.motl@ilo.de>.

EXAMPLE OUTPUT CHUNKS
  Synchronizing orm objects against entries in a .csv file - 17 items modified

      notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping;
      notice: OEF::Script::AbstractFeeder:
          - Loading data of type [n/a] and
            filtered by * from the storage named base
            to the storage named backend - filtered by *.
          - Will attempt to convert the data to Country.
      info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
      :c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
      :c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c
      :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
      info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 180  /  total: 197  /  attempt_modify: 17  /  ok: 17

  Synchronizing orm objects against entries in a .csv file - in-sync

      notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping;
      notice: OEF::Script::AbstractFeeder:
          - Loading data of type [n/a] and
            filtered by * from the storage named base
            to the storage named backend - filtered by *.
          - Will attempt to convert the data to Country.
      info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
      :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
      :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
      :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
      info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 197  /  total: 197

BUGS
      When in "import" mode for windows file - DBD::AutoCSV may hang.
      Hint: Maybe the source node contains an ident-, but no checksum-column?

USER LEVEL ERRORS
   Mapping

      - - - - - - - - - - - - - - - - - - - - - - - - - -
      info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: BizWorks::Process::Setup->startSync: Can't access mapping for node "Currency" - please check BizWorks::ResourceMapping.
      - - - - - - - - - - - - - - - - - - - - - - - - - -
      You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match
      the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how 
      to access the otherwise generic nodes.
      - - - - - - - - - - - - - - - - - - - - - - - - - -

   DBD::AutoCSV's rulebase

      - - - - - - - - - - - - - - - - - - - - - - - - - -
      info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )
      info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
  
      Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p
      m line 165, <GEN9> line 1.
       called from C:/home/amo/develop/netfrag.org/nfo/perl/libs/Data/Storage/Handler/DBI.pm at 123.
  
      DBI-Error: DBD::AutoCSV::st fetchrow_hashref failed: Attempt to fetch row from a Non-SELECT statement
      notice: Data::Transfer::Sync->syncNodes: No nodes to synchronize.
      - - - - - - - - - - - - - - - - - - - - - - - - - -
      DBD::AutoCSV contains a rulebase which is spooled down while attempting to guess the style of the csv file regarding
      parameters like newline (eol), column-seperation-character (sep_char), quoting character (quote_char).
      If this spool runs out of entries and no style could be resolved, DBD::CSV dies causing this "Execution ERROR" which
      results in a "DBI-Error" afterwards.
      - - - - - - - - - - - - - - - - - - - - - - - - - -

   Check structure of source node

      - - - - - - - - - - - - - - - - - - - - - - - - - -
      info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
      critical: Data::Transfer::Sync->syncNodes: Can not synchronize: No ident found in source node, maybe try to "import" this node first.
      - - - - - - - - - - - - - - - - - - - - - - - - - -
      If lowlevel detection succeeds, but no other required informations are found, this message is issued.
      "Other informations" might be:
        - column-header-row completely missing
        - ident column is empty
      - - - - - - - - - - - - - - - - - - - - - - - - - -

   Modify structure of source node

      - - - - - - - - - - - - - - - - - - - - - - - - - -
      info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
      info: Data::Transfer::Sync->_prepareNode_MetaProperties( descent source )
      warning: Data::Transfer::Sync->_prepareNode_MetaProperties: node is lacking meta properties - will try to alter...
      SQL ERROR: Command 'ALTER' not recognized or not supported!
  
      SQL ERROR: Command 'ALTER' not recognized or not supported!
      - - - - - - - - - - - - - - - - - - - - - - - - - -
      The Engine found a node which structure does not match the required. It tries to alter this automatically - only when doing "import" - 
      but the DBD driver (in this case DBD::CSV) gets in the way croaking not to be able to do this.
      This could also appear if your database connection has insufficient rights to modify the database structure.
      DBD::CSV croaks because it doesn't implement the ALTER command, so please edit your columns manually.
      Hint: Add columns with the names of your "ident" and "checksum" property specifications.
      - - - - - - - - - - - - - - - - - - - - - - - - - -

   Load source node by ident

      - - - - - - - - - - - - - - - - - - - - - - - - - -
      info: Data::Transfer::Sync->_prepareNode_DummyIdent( descent source )
      pcritical: Data::Transfer::Sync->_modifyNode failed: "source" node is empty.
      - - - - - - - - - - - - - - - - - - - - - - - - - -
      The source node could not be loaded. Maybe the ident is missing. Please check manually.
      Hint: Like above, the ident and/or checksum columns may be missing....
      - - - - - - - - - - - - - - - - - - - - - - - - - -

TODO
      o sub _resolveIdentProvider
      x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node
      x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm
      o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm
      x introduce Sync/Compare/MyComparisonImplementation.pm
      o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing
         - this implies everything done is _really_ split up into generic actions - how else would we defer them???
         - example uses:
            - fetch whole checksum list from node
            - remember source ident retransmits
         - remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations
      o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema)
      o expand statistics / keep track of:
        - touched/untouched nodes
      o full sync
        - just do a push and a pull for now but use stats for touched nodes in between to speed up things
      o introduce some new metadata flags for a descent
        - isNewNodePropagator
        x isWriteProtected

1	joko	1.3	NEWS
2	joko	1.4	2003-02-09 - backward compatibility with custom application
3	joko	1.3
4	joko	1.4	- Tangram <-> .csv
5			- Tangram <-> mysql
6			- Tangram <-> perl-code
7	joko	1.3
8	joko	1.4	2003-01-18 - start integration with Torus
9
10			- MAPI <-> LDAP
11	joko	1.5	- please visit http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/
12	joko	1.4
13			2002-10-10 - initial commit
14
15			- Tangram <-> mysql
16	joko	1.3
17	joko	1.1	DESCRIPTION
18	joko	1.4	Data::Transfer::Sync is a module providing a generic synchronization process
19			across arbitrary/multiple storages based on a ident/checksum mechanism.
20			It sits on top of Data::Storage.
21	joko	1.1
22			REQUIREMENTS
23	joko	1.5	dependencies
24
25			from CPAN or (for win32) a ppm-repository:
26			App::Config (CPAN only)
27			Tie::IxHash
28			Log::Dispatch
29			Log::Dispatch::Config
30			mixin
31			Hash::Merge
32			Tie::SecureHash
33			Date::Manip
34
35			from nfo/perl/libs:
36	joko	1.4	Data::Compare
37	joko	1.1	Data::Storage
38			Data::Transform
39	joko	1.5	DesignPattern:: (to be renamed into Class::)
40	joko	1.4	... and all their dependencies.
41
42	joko	1.5	additional modules required for 'outlook2ldap'
43
44			Win32::OLE
45			Net::LDAP
46
47			additional modules required for 'db2db'
48
49			DBI
50			DBD::mysql
51			DBD::CSV
52			Tangram
53
54	joko	1.4	AUTHORS / COPYRIGHT / LICENSE
55			The Data::Transfer::Sync module is Copyright (c) 2002 Andreas Motl.
56			All rights reserved.
57	joko	1.1
58	joko	1.5	Code contributions, suggestions, support and ideas by
59			Sebastian Utz <su@tunemedia.de> and Jan Hoffmann <jan@romain-volk.com>.
60
61	joko	1.4	You may distribute it under the terms of either the GNU General Public
62			License or the Artistic License, as specified in the Perl README file.
63	joko	1.1
64			SUPPORT / WARRANTY
65	joko	1.4	Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.
66
67	joko	1.5	Please feel free to contact me at <andreas.motl@ilo.de>.
68
69	joko	1.4	EXAMPLE OUTPUT CHUNKS
70	joko	1.5	Synchronizing orm objects against entries in a .csv file - 17 items modified
71	joko	1.4
72			notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping;
73			notice: OEF::Script::AbstractFeeder:
74			- Loading data of type [n/a] and
75			filtered by * from the storage named base
76			to the storage named backend - filtered by *.
77			- Will attempt to convert the data to Country.
78			info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
79			:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
80			:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c
81			:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
82			info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 180 / total: 197 / attempt_modify: 17 / ok: 17
83
84	joko	1.5	Synchronizing orm objects against entries in a .csv file - in-sync
85	joko	1.4
86			notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping;
87			notice: OEF::Script::AbstractFeeder:
88			- Loading data of type [n/a] and
89			filtered by * from the storage named base
90			to the storage named backend - filtered by *.
91			- Will attempt to convert the data to Country.
92			info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
93			:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
94			:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
95			:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
96			info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 197 / total: 197
97	joko	1.1
98			BUGS
99			When in "import" mode for windows file - DBD::AutoCSV may hang.
100	joko	1.3	Hint: Maybe the source node contains an ident-, but no checksum-column?
101	joko	1.1
102			USER LEVEL ERRORS
103			Mapping
104
105			- - - - - - - - - - - - - - - - - - - - - - - - - -
106			info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: BizWorks::Process::Setup->startSync: Can't access mapping for node "Currency" - please check BizWorks::ResourceMapping.
107			- - - - - - - - - - - - - - - - - - - - - - - - - -
108			You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match
109			the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how
110			to access the otherwise generic nodes.
111			- - - - - - - - - - - - - - - - - - - - - - - - - -
112
113			DBD::AutoCSV's rulebase
114
115			- - - - - - - - - - - - - - - - - - - - - - - - - -
116			info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )
117			info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
118
119			Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p
120			m line 165, <GEN9> line 1.
121			called from C:/home/amo/develop/netfrag.org/nfo/perl/libs/Data/Storage/Handler/DBI.pm at 123.
122
123			DBI-Error: DBD::AutoCSV::st fetchrow_hashref failed: Attempt to fetch row from a Non-SELECT statement
124			notice: Data::Transfer::Sync->syncNodes: No nodes to synchronize.
125			- - - - - - - - - - - - - - - - - - - - - - - - - -
126			DBD::AutoCSV contains a rulebase which is spooled down while attempting to guess the style of the csv file regarding
127			parameters like newline (eol), column-seperation-character (sep_char), quoting character (quote_char).
128			If this spool runs out of entries and no style could be resolved, DBD::CSV dies causing this "Execution ERROR" which
129			results in a "DBI-Error" afterwards.
130			- - - - - - - - - - - - - - - - - - - - - - - - - -
131
132			Check structure of source node
133
134			- - - - - - - - - - - - - - - - - - - - - - - - - -
135			info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
136			critical: Data::Transfer::Sync->syncNodes: Can not synchronize: No ident found in source node, maybe try to "import" this node first.
137			- - - - - - - - - - - - - - - - - - - - - - - - - -
138			If lowlevel detection succeeds, but no other required informations are found, this message is issued.
139			"Other informations" might be:
140			- column-header-row completely missing
141			- ident column is empty
142			- - - - - - - - - - - - - - - - - - - - - - - - - -
143
144			Modify structure of source node
145
146			- - - - - - - - - - - - - - - - - - - - - - - - - -
147			info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
148			info: Data::Transfer::Sync->_prepareNode_MetaProperties( descent source )
149			warning: Data::Transfer::Sync->_prepareNode_MetaProperties: node is lacking meta properties - will try to alter...
150			SQL ERROR: Command 'ALTER' not recognized or not supported!
151
152			SQL ERROR: Command 'ALTER' not recognized or not supported!
153			- - - - - - - - - - - - - - - - - - - - - - - - - -
154			The Engine found a node which structure does not match the required. It tries to alter this automatically - only when doing "import" -
155			but the DBD driver (in this case DBD::CSV) gets in the way croaking not to be able to do this.
156			This could also appear if your database connection has insufficient rights to modify the database structure.
157			DBD::CSV croaks because it doesn't implement the ALTER command, so please edit your columns manually.
158			Hint: Add columns with the names of your "ident" and "checksum" property specifications.
159			- - - - - - - - - - - - - - - - - - - - - - - - - -
160
161			Load source node by ident
162
163			- - - - - - - - - - - - - - - - - - - - - - - - - -
164			info: Data::Transfer::Sync->_prepareNode_DummyIdent( descent source )
165			pcritical: Data::Transfer::Sync->_modifyNode failed: "source" node is empty.
166			- - - - - - - - - - - - - - - - - - - - - - - - - -
167			The source node could not be loaded. Maybe the ident is missing. Please check manually.
168			Hint: Like above, the ident and/or checksum columns may be missing....
169			- - - - - - - - - - - - - - - - - - - - - - - - - -
170
171			TODO
172	joko	1.4	o sub _resolveIdentProvider
173			x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node
174			x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm
175			o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm
176			x introduce Sync/Compare/MyComparisonImplementation.pm
177			o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing
178	joko	1.1	- this implies everything done is _really_ split up into generic actions - how else would we defer them???
179			- example uses:
180			- fetch whole checksum list from node
181			- remember source ident retransmits
182			- remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations
183	joko	1.4	o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema)
184			o expand statistics / keep track of:
185	joko	1.1	- touched/untouched nodes
186	joko	1.4	o full sync
187	joko	1.1	- just do a push and a pull for now but use stats for touched nodes in between to speed up things
188	joko	1.4	o introduce some new metadata flags for a descent
189	joko	1.1	- isNewNodePropagator
190	joko	1.4	x isWriteProtected
191	joko	1.3
MailToCvsAdmin">MailToCvsAdmin	ViewVC Help
Powered by ViewVC 1.1.26