Data/Transfer/Sync.README.html

<HTML>
<HEAD>
<TITLE>Data::Storage::Sync</TITLE>
<LINK REV="made" HREF="mailto:">
</HEAD>

<BODY>

<A NAME="__index__"></A>
<!-- INDEX BEGIN -->

<UL>

        <LI><A HREF="#news">NEWS</A></LI>
        <UL>

                <UL>

                        <UL>

                                <LI><A HREF="#20030209  backward compatibility with custom application">2003-02-09  -  backward compatibility with custom application</A></LI>
                                <LI><A HREF="#20030118  start integration with torus">2003-01-18  -  start integration with Torus</A></LI>
                                <LI><A HREF="#20021010  initial commit">2002-10-10  -  initial commit</A></LI>
                        </UL>

                </UL>

        </UL>

        <LI><A HREF="#description">DESCRIPTION</A></LI>
        <LI><A HREF="#requirements">REQUIREMENTS</A></LI>
        <UL>

                <UL>

                        <LI><A HREF="#dependencies">dependencies</A></LI>
                        <LI><A HREF="#additional modules required for 'outlook2ldap'">additional modules required for 'outlook2ldap'</A></LI>
                        <LI><A HREF="#additional modules required for 'db2db'">additional modules required for 'db2db'</A></LI>
                </UL>

        </UL>

        <LI><A HREF="#authors / copyright / license">AUTHORS / COPYRIGHT / LICENSE</A></LI>
        <LI><A HREF="#support / warranty">SUPPORT / WARRANTY</A></LI>
        <LI><A HREF="#example output chunks">EXAMPLE OUTPUT CHUNKS</A></LI>
        <UL>

                <LI><A HREF="#synchronizing orm objects against entries in a .csv file  17 items modified">Synchronizing orm objects against entries in a .csv file - 17 items modified</A></LI>
                <LI><A HREF="#synchronizing orm objects against entries in a .csv file  insync">Synchronizing orm objects against entries in a .csv file - in-sync</A></LI>
        </UL>

        <LI><A HREF="#bugs">BUGS</A></LI>
        <LI><A HREF="#user level errors">USER LEVEL ERRORS</A></LI>
        <UL>

                <UL>

                        <UL>

                                <LI><A HREF="#mapping">Mapping</A></LI>
                                <LI><A HREF="#dbd::autocsv's rulebase">DBD::AutoCSV's rulebase</A></LI>
                                <LI><A HREF="#check structure of source node">Check structure of source node</A></LI>
                                <LI><A HREF="#modify structure of source node">Modify structure of source node</A></LI>
                                <LI><A HREF="#load source node by ident">Load source node by ident</A></LI>
                        </UL>

                </UL>

        </UL>

        <LI><A HREF="#todo">TODO</A></LI>
</UL>
<!-- INDEX END -->

<HR>
<P>
<HR>
<H1><A NAME="news">NEWS</A></H1>
<P>
<H4><A NAME="20030209  backward compatibility with custom application">2003-02-09  -  backward compatibility with custom application</A></H4>
<PRE>
  - Tangram &lt;-&gt; .csv
  - Tangram &lt;-&gt; mysql
  - Tangram &lt;-&gt; perl-code</PRE>
<P>
<H4><A NAME="20030118  start integration with torus">2003-01-18  -  start integration with Torus</A></H4>
<PRE>
  - MAPI &lt;-&gt; LDAP
  - please visit <A HREF="http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/">http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/</A></PRE>
<P>
<H4><A NAME="20021010  initial commit">2002-10-10  -  initial commit</A></H4>
<PRE>
  - Tangram &lt;-&gt; mysql</PRE>
<P>
<HR>
<H1><A NAME="description">DESCRIPTION</A></H1>
<PRE>
  Data::Transfer::Sync is a module providing a generic synchronization process
  across arbitrary/multiple storages based on a ident/checksum mechanism.
  It sits on top of Data::Storage.</PRE>
<P>
<HR>
<H1><A NAME="requirements">REQUIREMENTS</A></H1>
<P>
<H3><A NAME="dependencies">dependencies</A></H3>
<PRE>
  from CPAN or (for win32) a ppm-repository:
    App::Config (CPAN only)
    Tie::IxHash
    Log::Dispatch
    Log::Dispatch::Config
    mixin
    Hash::Merge
    Tie::SecureHash
    Date::Manip</PRE>
<PRE>
  from nfo/perl/libs:
    Data::Compare
    Data::Storage
    Data::Transform
    DesignPattern:: (to be renamed into Class::)
    ... and all their dependencies.</PRE>
<P>
<H3><A NAME="additional modules required for 'outlook2ldap'">additional modules required for 'outlook2ldap'</A></H3>
<PRE>
    Win32::OLE
    Net::LDAP</PRE>
<P>
<H3><A NAME="additional modules required for 'db2db'">additional modules required for 'db2db'</A></H3>
<PRE>
    DBI
    DBD::mysql
    DBD::CSV
    Tangram</PRE>
<P>
<HR>
<H1><A NAME="authors / copyright / license">AUTHORS / COPYRIGHT / LICENSE</A></H1>
<PRE>
  The Data::Transfer::Sync module is Copyright (c) 2002 Andreas Motl.
  All rights reserved.</PRE>
<PRE>
  Code contributions, suggestions, support and ideas by
  Sebastian Utz &lt;su@tunemedia.de&gt; and Jan Hoffmann &lt;jan@romain-volk.com&gt;.
</PRE>
<PRE>

  You may distribute it under the terms of either the GNU General Public
  License or the Artistic License, as specified in the Perl README file.</PRE>
<P>
<HR>
<H1><A NAME="support / warranty">SUPPORT / WARRANTY</A></H1>
<PRE>
  Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.</PRE>
<PRE>
  Please feel free to contact me at &lt;andreas.motl@ilo.de&gt;.</PRE>
<P>
<HR>
<H1><A NAME="example output chunks">EXAMPLE OUTPUT CHUNKS</A></H1>
<P>
<H2><A NAME="synchronizing orm objects against entries in a .csv file  17 items modified">Synchronizing orm objects against entries in a .csv file - 17 items modified</A></H2>
<PRE>
  notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping;
  notice: OEF::Script::AbstractFeeder:
      - Loading data of type [n/a] and
        filtered by * from the storage named base
        to the storage named backend - filtered by *.
      - Will attempt to convert the data to Country.
  info: Data::Transfer::Sync::API-&gt;syncNodes: source=backend/* [Country] &lt;- target=base/* []
  :c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
  :c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c
  :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
  info: Data::Transfer::Sync::Core-&gt;_startsync: statistics: in_sync: 180  /  total: 197  /  attempt_modify: 17  /  ok: 17</PRE>
<P>
<H2><A NAME="synchronizing orm objects against entries in a .csv file  insync">Synchronizing orm objects against entries in a .csv file - in-sync</A></H2>
<PRE>

  notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping;
  notice: OEF::Script::AbstractFeeder:
      - Loading data of type [n/a] and
        filtered by * from the storage named base
        to the storage named backend - filtered by *.
      - Will attempt to convert the data to Country.
  info: Data::Transfer::Sync::API-&gt;syncNodes: source=backend/* [Country] &lt;- target=base/* []
  :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
  :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
  :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
  info: Data::Transfer::Sync::Core-&gt;_startsync: statistics: in_sync: 197  /  total: 197</PRE>
<P>
<HR>
<H1><A NAME="bugs">BUGS</A></H1>
<PRE>
  When in &quot;import&quot; mode for windows file - DBD::AutoCSV may hang.
  Hint: Maybe the source node contains an ident-, but no checksum-column?</PRE>
<P>
<HR>
<H1><A NAME="user level errors">USER LEVEL ERRORS</A></H1>
<P>
<H4><A NAME="mapping">Mapping</A></H4>
<PRE>
  - - - - - - - - - - - - - - - - - - - - - - - - - -
  info: BizWorks::Process::Setup-&gt;syncResource( source_node Currency mode PULL erase 0 import 0 )critical: BizWorks::Process::Setup-&gt;startSync: Can't access mapping for node &quot;Currency&quot; - please check BizWorks::ResourceMapping.
  - - - - - - - - - - - - - - - - - - - - - - - - - -
  You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match
  the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how 
  to access the otherwise generic nodes.
  - - - - - - - - - - - - - - - - - - - - - - - - - -</PRE>
<P>
<H4><A NAME="dbd::autocsv's rulebase">DBD::AutoCSV's rulebase</A></H4>
<PRE>

  - - - - - - - - - - - - - - - - - - - - - - - - - -
  info: BizWorks::Process::Setup-&gt;syncResource( source_node Currency mode PULL erase 0 import 0 )
  info: Data::Transfer::Sync-&gt;syncNodes: source=L/Currency &lt;- target=R/currencies.csv</PRE>
<PRE>

  Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p
  m line 165, &lt;GEN9&gt; line 1.
   called from C:/home/amo/develop/netfrag.org/nfo/perl/libs/Data/Storage/Handler/DBI.pm at 123.</PRE>
<PRE>

  DBI-Error: DBD::AutoCSV::st fetchrow_hashref failed: Attempt to fetch row from a Non-SELECT statement
  notice: Data::Transfer::Sync-&gt;syncNodes: No nodes to synchronize.
  - - - - - - - - - - - - - - - - - - - - - - - - - -
  DBD::AutoCSV contains a rulebase which is spooled down while attempting to guess the style of the csv file regarding
  parameters like newline (eol), column-seperation-character (sep_char), quoting character (quote_char).
  If this spool runs out of entries and no style could be resolved, DBD::CSV dies causing this &quot;Execution ERROR&quot; which
  results in a &quot;DBI-Error&quot; afterwards.
  - - - - - - - - - - - - - - - - - - - - - - - - - -</PRE>
<P>
<H4><A NAME="check structure of source node">Check structure of source node</A></H4>
<PRE>

  - - - - - - - - - - - - - - - - - - - - - - - - - -
  info: Data::Transfer::Sync-&gt;syncNodes: source=L/Currency &lt;- target=R/currencies.csv
  critical: Data::Transfer::Sync-&gt;syncNodes: Can not synchronize: No ident found in source node, maybe try to &quot;import&quot; this node first.
  - - - - - - - - - - - - - - - - - - - - - - - - - -
  If lowlevel detection succeeds, but no other required informations are found, this message is issued.
  &quot;Other informations&quot; might be:
    - column-header-row completely missing
    - ident column is empty
  - - - - - - - - - - - - - - - - - - - - - - - - - -</PRE>
<P>
<H4><A NAME="modify structure of source node">Modify structure of source node</A></H4>
<PRE>

  - - - - - - - - - - - - - - - - - - - - - - - - - -
  info: Data::Transfer::Sync-&gt;syncNodes: source=L/Currency &lt;- target=R/currencies.csv
  info: Data::Transfer::Sync-&gt;_prepareNode_MetaProperties( descent source )
  warning: Data::Transfer::Sync-&gt;_prepareNode_MetaProperties: node is lacking meta properties - will try to alter...
  SQL ERROR: Command 'ALTER' not recognized or not supported!</PRE>
<PRE>

  SQL ERROR: Command 'ALTER' not recognized or not supported!
  - - - - - - - - - - - - - - - - - - - - - - - - - -
  The Engine found a node which structure does not match the required. It tries to alter this automatically - only when doing &quot;import&quot; - 
  but the DBD driver (in this case DBD::CSV) gets in the way croaking not to be able to do this.
  This could also appear if your database connection has insufficient rights to modify the database structure.
  DBD::CSV croaks because it doesn't implement the ALTER command, so please edit your columns manually.
  Hint: Add columns with the names of your &quot;ident&quot; and &quot;checksum&quot; property specifications.
  - - - - - - - - - - - - - - - - - - - - - - - - - -</PRE>
<P>
<H4><A NAME="load source node by ident">Load source node by ident</A></H4>
<PRE>
  - - - - - - - - - - - - - - - - - - - - - - - - - -
  info: Data::Transfer::Sync-&gt;_prepareNode_DummyIdent( descent source )
  pcritical: Data::Transfer::Sync-&gt;_modifyNode failed: &quot;source&quot; node is empty.
  - - - - - - - - - - - - - - - - - - - - - - - - - -
  The source node could not be loaded. Maybe the ident is missing. Please check manually.
  Hint: Like above, the ident and/or checksum columns may be missing....
  - - - - - - - - - - - - - - - - - - - - - - - - - -</PRE>
<P>
<HR>
<H1><A NAME="todo">TODO</A></H1>
<PRE>
  o sub _resolveIdentProvider
  x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node
  x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm
  o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm
  x introduce Sync/Compare/MyComparisonImplementation.pm
  o some generic deferring method - e.g. &quot;$self-&gt;defer(action)&quot; - to be able to accumulate a bunch of actions for later processing
     - this implies everything done is _really_ split up into generic actions - how else would we defer them???
     - example uses:
        - fetch whole checksum list from node
        - remember source ident retransmits
     - remember: this is convenient - and maybe / of course faster - but we'll loose &quot;per-node-atomic&quot; operations
  o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema)
  o expand statistics / keep track of:
    - touched/untouched nodes
  o full sync
    - just do a push and a pull for now but use stats for touched nodes in between to speed up things
  o introduce some new metadata flags for a descent
    - isNewNodePropagator
    x isWriteProtected</PRE>

</BODY>

</HTML>
1	joko	1.1	<HTML>
2			<HEAD>
3	joko	1.2	<TITLE>Data::Storage::Sync</TITLE>
4	joko	1.1	<LINK REV="made" HREF="mailto:">
5			</HEAD>
6
7			<BODY>
8
9			<A NAME="__index__"></A>
10			<!-- INDEX BEGIN -->
11
12			<UL>
13
14	joko	1.2	<LI><A HREF="#news">NEWS</A></LI>
15			<UL>
16
17			<UL>
18
19			<UL>
20
21	joko	1.3	<LI><A HREF="#20030209 backward compatibility with custom application">2003-02-09 - backward compatibility with custom application</A></LI>
22			<LI><A HREF="#20030118 start integration with torus">2003-01-18 - start integration with Torus</A></LI>
23			<LI><A HREF="#20021010 initial commit">2002-10-10 - initial commit</A></LI>
24	joko	1.2	</UL>
25
26			</UL>
27
28			</UL>
29
30	joko	1.1	<LI><A HREF="#description">DESCRIPTION</A></LI>
31			<LI><A HREF="#requirements">REQUIREMENTS</A></LI>
32	joko	1.4	<UL>
33
34			<UL>
35
36			<LI><A HREF="#dependencies">dependencies</A></LI>
37			<LI><A HREF="#additional modules required for 'outlook2ldap'">additional modules required for 'outlook2ldap'</A></LI>
38			<LI><A HREF="#additional modules required for 'db2db'">additional modules required for 'db2db'</A></LI>
39			</UL>
40
41			</UL>
42
43	joko	1.3	<LI><A HREF="#authors / copyright / license">AUTHORS / COPYRIGHT / LICENSE</A></LI>
44	joko	1.1	<LI><A HREF="#support / warranty">SUPPORT / WARRANTY</A></LI>
45	joko	1.3	<LI><A HREF="#example output chunks">EXAMPLE OUTPUT CHUNKS</A></LI>
46			<UL>
47
48	joko	1.4	<LI><A HREF="#synchronizing orm objects against entries in a .csv file 17 items modified">Synchronizing orm objects against entries in a .csv file - 17 items modified</A></LI>
49			<LI><A HREF="#synchronizing orm objects against entries in a .csv file insync">Synchronizing orm objects against entries in a .csv file - in-sync</A></LI>
50	joko	1.3	</UL>
51
52	joko	1.1	<LI><A HREF="#bugs">BUGS</A></LI>
53			<LI><A HREF="#user level errors">USER LEVEL ERRORS</A></LI>
54			<UL>
55
56			<UL>
57
58			<UL>
59
60			<LI><A HREF="#mapping">Mapping</A></LI>
61			<LI><A HREF="#dbd::autocsv's rulebase">DBD::AutoCSV's rulebase</A></LI>
62			<LI><A HREF="#check structure of source node">Check structure of source node</A></LI>
63			<LI><A HREF="#modify structure of source node">Modify structure of source node</A></LI>
64			<LI><A HREF="#load source node by ident">Load source node by ident</A></LI>
65			</UL>
66
67			</UL>
68
69			</UL>
70
71			<LI><A HREF="#todo">TODO</A></LI>
72			</UL>
73			<!-- INDEX END -->
74
75			<HR>
76			<P>
77			<HR>
78	joko	1.2	<H1><A NAME="news">NEWS</A></H1>
79			<P>
80	joko	1.3	<H4><A NAME="20030209 backward compatibility with custom application">2003-02-09 - backward compatibility with custom application</A></H4>
81	joko	1.2	<PRE>
82	joko	1.3	- Tangram <-> .csv
83			- Tangram <-> mysql
84			- Tangram <-> perl-code</PRE>
85	joko	1.2	<P>
86	joko	1.3	<H4><A NAME="20030118 start integration with torus">2003-01-18 - start integration with Torus</A></H4>
87			<PRE>
88			- MAPI <-> LDAP
89	joko	1.4	- please visit <A HREF="http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/">http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/</A></PRE>
90	joko	1.3	<P>
91			<H4><A NAME="20021010 initial commit">2002-10-10 - initial commit</A></H4>
92			<PRE>
93			- Tangram <-> mysql</PRE>
94	joko	1.2	<P>
95			<HR>
96	joko	1.1	<H1><A NAME="description">DESCRIPTION</A></H1>
97	joko	1.3	<PRE>
98			Data::Transfer::Sync is a module providing a generic synchronization process
99			across arbitrary/multiple storages based on a ident/checksum mechanism.
100			It sits on top of Data::Storage.</PRE>
101	joko	1.1	<P>
102			<HR>
103			<H1><A NAME="requirements">REQUIREMENTS</A></H1>
104	joko	1.4	<P>
105			<H3><A NAME="dependencies">dependencies</A></H3>
106			<PRE>
107			from CPAN or (for win32) a ppm-repository:
108			App::Config (CPAN only)
109			Tie::IxHash
110			Log::Dispatch
111			Log::Dispatch::Config
112			mixin
113			Hash::Merge
114			Tie::SecureHash
115			Date::Manip</PRE>
116	joko	1.1	<PRE>
117	joko	1.4	from nfo/perl/libs:
118	joko	1.3	Data::Compare
119	joko	1.1	Data::Storage
120			Data::Transform
121	joko	1.4	DesignPattern:: (to be renamed into Class::)
122	joko	1.3	... and all their dependencies.</PRE>
123	joko	1.1	<P>
124	joko	1.4	<H3><A NAME="additional modules required for 'outlook2ldap'">additional modules required for 'outlook2ldap'</A></H3>
125			<PRE>
126			Win32::OLE
127			Net::LDAP</PRE>
128			<P>
129			<H3><A NAME="additional modules required for 'db2db'">additional modules required for 'db2db'</A></H3>
130			<PRE>
131			DBI
132			DBD::mysql
133			DBD::CSV
134			Tangram</PRE>
135			<P>
136	joko	1.1	<HR>
137	joko	1.3	<H1><A NAME="authors / copyright / license">AUTHORS / COPYRIGHT / LICENSE</A></H1>
138			<PRE>
139			The Data::Transfer::Sync module is Copyright (c) 2002 Andreas Motl.
140			All rights reserved.</PRE>
141			<PRE>
142	joko	1.4	Code contributions, suggestions, support and ideas by
143			Sebastian Utz <su@tunemedia.de> and Jan Hoffmann <jan@romain-volk.com>.
144			</PRE>
145	joko	1.3	<PRE>
146	joko	1.4
147	joko	1.3	You may distribute it under the terms of either the GNU General Public
148			License or the Artistic License, as specified in the Perl README file.</PRE>
149	joko	1.1	<P>
150			<HR>
151			<H1><A NAME="support / warranty">SUPPORT / WARRANTY</A></H1>
152	joko	1.3	<PRE>
153			Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.</PRE>
154	joko	1.4	<PRE>
155			Please feel free to contact me at <andreas.motl@ilo.de>.</PRE>
156	joko	1.3	<P>
157			<HR>
158			<H1><A NAME="example output chunks">EXAMPLE OUTPUT CHUNKS</A></H1>
159			<P>
160	joko	1.4	<H2><A NAME="synchronizing orm objects against entries in a .csv file 17 items modified">Synchronizing orm objects against entries in a .csv file - 17 items modified</A></H2>
161	joko	1.3	<PRE>
162			notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping;
163			notice: OEF::Script::AbstractFeeder:
164			- Loading data of type [n/a] and
165			filtered by * from the storage named base
166			to the storage named backend - filtered by *.
167			- Will attempt to convert the data to Country.
168			info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
169			:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
170			:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c
171			:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
172			info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 180 / total: 197 / attempt_modify: 17 / ok: 17</PRE>
173			<P>
174	joko	1.4	<H2><A NAME="synchronizing orm objects against entries in a .csv file insync">Synchronizing orm objects against entries in a .csv file - in-sync</A></H2>
175	joko	1.3	<PRE>
176
177			notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping;
178			notice: OEF::Script::AbstractFeeder:
179			- Loading data of type [n/a] and
180			filtered by * from the storage named base
181			to the storage named backend - filtered by *.
182			- Will attempt to convert the data to Country.
183			info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* []
184			:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
185			:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
186			:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
187			info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 197 / total: 197</PRE>
188	joko	1.1	<P>
189			<HR>
190			<H1><A NAME="bugs">BUGS</A></H1>
191			<PRE>
192			When in "import" mode for windows file - DBD::AutoCSV may hang.
193	joko	1.2	Hint: Maybe the source node contains an ident-, but no checksum-column?</PRE>
194	joko	1.1	<P>
195			<HR>
196			<H1><A NAME="user level errors">USER LEVEL ERRORS</A></H1>
197			<P>
198			<H4><A NAME="mapping">Mapping</A></H4>
199			<PRE>
200			- - - - - - - - - - - - - - - - - - - - - - - - - -
201			info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: BizWorks::Process::Setup->startSync: Can't access mapping for node "Currency" - please check BizWorks::ResourceMapping.
202			- - - - - - - - - - - - - - - - - - - - - - - - - -
203			You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match
204			the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how
205			to access the otherwise generic nodes.
206			- - - - - - - - - - - - - - - - - - - - - - - - - -</PRE>
207			<P>
208			<H4><A NAME="dbd::autocsv's rulebase">DBD::AutoCSV's rulebase</A></H4>
209			<PRE>
210
211			- - - - - - - - - - - - - - - - - - - - - - - - - -
212			info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )
213			info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv</PRE>
214			<PRE>
215
216			Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p
217			m line 165, <GEN9> line 1.
218			called from C:/home/amo/develop/netfrag.org/nfo/perl/libs/Data/Storage/Handler/DBI.pm at 123.</PRE>
219			<PRE>
220
221			DBI-Error: DBD::AutoCSV::st fetchrow_hashref failed: Attempt to fetch row from a Non-SELECT statement
222			notice: Data::Transfer::Sync->syncNodes: No nodes to synchronize.
223			- - - - - - - - - - - - - - - - - - - - - - - - - -
224			DBD::AutoCSV contains a rulebase which is spooled down while attempting to guess the style of the csv file regarding
225			parameters like newline (eol), column-seperation-character (sep_char), quoting character (quote_char).
226			If this spool runs out of entries and no style could be resolved, DBD::CSV dies causing this "Execution ERROR" which
227			results in a "DBI-Error" afterwards.
228			- - - - - - - - - - - - - - - - - - - - - - - - - -</PRE>
229			<P>
230			<H4><A NAME="check structure of source node">Check structure of source node</A></H4>
231			<PRE>
232
233			- - - - - - - - - - - - - - - - - - - - - - - - - -
234			info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
235			critical: Data::Transfer::Sync->syncNodes: Can not synchronize: No ident found in source node, maybe try to "import" this node first.
236			- - - - - - - - - - - - - - - - - - - - - - - - - -
237			If lowlevel detection succeeds, but no other required informations are found, this message is issued.
238			"Other informations" might be:
239			- column-header-row completely missing
240			- ident column is empty
241			- - - - - - - - - - - - - - - - - - - - - - - - - -</PRE>
242			<P>
243			<H4><A NAME="modify structure of source node">Modify structure of source node</A></H4>
244			<PRE>
245
246			- - - - - - - - - - - - - - - - - - - - - - - - - -
247			info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv
248			info: Data::Transfer::Sync->_prepareNode_MetaProperties( descent source )
249			warning: Data::Transfer::Sync->_prepareNode_MetaProperties: node is lacking meta properties - will try to alter...
250			SQL ERROR: Command 'ALTER' not recognized or not supported!</PRE>
251			<PRE>
252
253			SQL ERROR: Command 'ALTER' not recognized or not supported!
254			- - - - - - - - - - - - - - - - - - - - - - - - - -
255			The Engine found a node which structure does not match the required. It tries to alter this automatically - only when doing "import" -
256			but the DBD driver (in this case DBD::CSV) gets in the way croaking not to be able to do this.
257			This could also appear if your database connection has insufficient rights to modify the database structure.
258			DBD::CSV croaks because it doesn't implement the ALTER command, so please edit your columns manually.
259			Hint: Add columns with the names of your "ident" and "checksum" property specifications.
260			- - - - - - - - - - - - - - - - - - - - - - - - - -</PRE>
261			<P>
262			<H4><A NAME="load source node by ident">Load source node by ident</A></H4>
263			<PRE>
264			- - - - - - - - - - - - - - - - - - - - - - - - - -
265			info: Data::Transfer::Sync->_prepareNode_DummyIdent( descent source )
266			pcritical: Data::Transfer::Sync->_modifyNode failed: "source" node is empty.
267			- - - - - - - - - - - - - - - - - - - - - - - - - -
268			The source node could not be loaded. Maybe the ident is missing. Please check manually.
269			Hint: Like above, the ident and/or checksum columns may be missing....
270			- - - - - - - - - - - - - - - - - - - - - - - - - -</PRE>
271			<P>
272			<HR>
273			<H1><A NAME="todo">TODO</A></H1>
274			<PRE>
275	joko	1.3	o sub _resolveIdentProvider
276			x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node
277			x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm
278			o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm
279			x introduce Sync/Compare/MyComparisonImplementation.pm
280			o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing
281	joko	1.1	- this implies everything done is _really_ split up into generic actions - how else would we defer them???
282			- example uses:
283			- fetch whole checksum list from node
284			- remember source ident retransmits
285			- remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations
286	joko	1.3	o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema)
287			o expand statistics / keep track of:
288	joko	1.1	- touched/untouched nodes
289	joko	1.3	o full sync
290	joko	1.1	- just do a push and a pull for now but use stats for touched nodes in between to speed up things
291	joko	1.3	o introduce some new metadata flags for a descent
292	joko	1.1	- isNewNodePropagator
293	joko	1.3	x isWriteProtected</PRE>
294	joko	1.1
295			</BODY>
296
297			</HTML>
MailToCvsAdmin">MailToCvsAdmin	ViewVC Help
Powered by ViewVC 1.1.26