libs/Data/README.text

NAME
    Data::Storage - Interface for accessing various Storage implementations
    for Perl in an independent way

AIMS
      - should encapsulate Tangram, DBI, DBD::CSV and LWP:: to access them in an unordinary (more convenient) way ;)
      - introduce a generic layered structure, refactor *SUBLAYER*-stuff, make (e.g.) this possible:
        Perl Data::Storage[DBD::CSV]  ->  Perl LWP::  ->  Internet HTTP/FTP/*  ->  Host Daemon  ->  csv-file
      - provide generic synchronization mechanisms across arbitrary/multiple storages based on ident/checksum
        maybe it's possible to have schema-, structural- and semantical modifications synchronized???

SYNOPSIS
  BASIC ACCESS

  ADVANCED ACCESS

      ... via inheritance:
  
        use Data::Storage;
        my $proxyObj = new HttpProxy;
        $proxyObj->{url} = $url;
        $proxyObj->{payload} = $content;
        $self->{storage}->insert($proxyObj);
    
        use Data::Storage;
        my $proxyObj = HttpProxy->new(
          url => $url,
          payload => $content,
        );
        $self->{storage}->insert($proxyObj);

  SYNCHRONIZATION

      my $nodemapping = {
        'LangText' => 'langtexts.csv',
        'Currency' => 'currencies.csv',
        'Country'  => 'countries.csv',
      };

      my $propmapping = {
        'LangText' => [
          [ 'source:lcountrykey'  =>  'target:country' ],
          [ 'source:lkey'         =>  'target:key' ],
          [ 'source:lvalue'       =>  'target:text' ],
        ],
        'Currency' => [
          [ 'source:ckey'         =>  'target:key' ],
          [ 'source:cname'        =>  'target:text' ],
        ],
        'Country' => [
          [ 'source:ckey'         =>  'target:key' ],
          [ 'source:cname'        =>  'target:text' ],
        ],
      };

      sub syncResource {

        my $self = shift;
        my $node_source = shift;
        my $mode = shift;
        my $opts = shift;
    
        $mode ||= '';
        $opts->{erase} ||= 0;
    
        $logger->info( __PACKAGE__ . "->syncResource( node_source $node_source mode $mode erase $opts->{erase} )");
  
        # resolve metadata for syncing requested resource
        my $node_target = $nodemapping->{$node_source};
        my $mapping = $propmapping->{$node_source};
    
        if (!$node_target || !$mapping) {
          # loggger.... "no target, sorry!"
          print "error while resolving resource metadata", "\n";
          return;
        }
    
        if ($opts->{erase}) {
          $self->_erase_all($node_source);
        }
  
        # create new sync object
        my $sync = Data::Transfer::Sync->new( 
          storages => {
            L => $self->{bizWorks}->{backend},
            R => $self->{bizWorks}->{resources},
          },
          id_authorities        =>  [qw( L ) ],
          checksum_authorities  =>  [qw( L ) ],
          write_protected       =>  [qw( R ) ],
          verbose               =>  1,
        );
    
        # sync
        # todo: filter!?
        $sync->syncNodes( {
          direction       =>  $mode,                 # | +PUSH | +PULL | -FULL | +IMPORT | -EXPORT
          method          =>  'checksum',            # | -timestamp | -manual
          source          =>  "L:$node_source",
          source_ident    =>  'storage_method:id',
          source_exclude  =>  [qw( id cs )],
          target          =>  "R:$node_target",
          target_ident    =>  'property:oid',
          mapping         =>  $mapping,
        } );

      }

  NOTE

    This module heavily relies on DBI and Tangram, but adds a lot of
    additional bugs and quirks. Please look at their documentation and/or
    this code for additional information.

REQUIREMENTS
      For full functionality:
        DBI              from CPAN
        DBD::mysql       from CPAN
        Tangram 2.04     from CPAN         (hmmm, 2.04 won't do in some cases)
        Tangram 2.05     from http://...   (2.05 seems okay but there are also additional patches from our side)
        Class::Tangram   from CPAN
        DBD::CSV         from CPAN
        MySQL::Diff      from http://adamspiers.org/computing/mysqldiff/
        ... and all their dependencies

DESCRIPTION
    Data::Storage is a module for accessing various "data structures" stored
    inside various "data containers". It sits on top of DBI and/or Tangram.

AUTHORS / COPYRIGHT
    The Data::Storage module is Copyright (c) 2002 Andreas Motl. All rights
    reserved.

    You may distribute it under the terms of either the GNU General Public
    License or the Artistic License, as specified in the Perl README file.

ACKNOWLEDGEMENTS
    Larry Wall for Perl, Tim Bunce for DBI, Jean-Louis Leroy for Tangram and
    Set::Object, Sam Vilain for Class::Tangram, Jochen Wiedmann and Jeff
    Zucker for DBD::CSV and related, Adam Spiers for MySQL::Diff and all
    contributors.

SUPPORT / WARRANTY
    Data::Storage is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.

TODO
  BUGS

    "DBI-Error [Tangram]: DBD::mysql::st execute failed: Unknown column
    't1.requestdump' in 'field list'"

      ... occours when operating on object-attributes not introduced yet:
      this should be detected and appended/replaced through:
      "Schema-Error detected, maybe (just) an inconsistency. 
      Please check if your declaration in schema-module "a" matches structure in database "b" or try to run"
      db_setup.pl --dbkey=import --action=deploy

    Compare schema (structure diff) with database ...

      ... when issuing "db_setup.pl --dbkey=import --action=deploy"
      on a database with an already deployed schema, use an additional "--update" then
      to lift the schema inside the database to the current declared schema.
      You will have to approve removals and changes on field-level while
      new objects and new fields are introduced silently without any interaction needed.
      In future versions there may be additional options to control silent processing of
      removals and changes.
      See this CRUD-table applying to the actions occouring on Classes and Class variables when deploying schemas,
      don't mix this up with CRUD-actions on Objects, these are already handled by (e.g.) Tangram itself.
      Classes:
        C create    ->  yes, handled automatically
        R retrieve  ->  no, not subject of this aspect since it is about deployment only
        U update    ->  yes, automatically for Class meta-attributes, yes/no for Class variables (look at the rules down here)
        D delete    ->  yes, just by user-interaction
      Class variables:
        C create    ->  yes, handled automatically
        R retrieve  ->  no, not subject of this aspect since it is about deployment only
        U update    ->  yes, just by user-interaction; maybe automatically if it can be determined that data wouldn't be lost
        D delete    ->  yes, just by user-interaction
  
      It's all about not to be able to loose data simply while this is in pre-alpha stage.
      And loosing data by being able to modify and redeploy schemas easily is definitely quite easy.
  
      As we can see, creations of Classes and new Class variables is handled 
      automatically and this is believed to be the most common case under normal circumstances.

  FEATURES

      - Get this stuff together with UML (Unified Modeling Language) and/or standards from ODMG.
      - Make it possible to load/save schemas in XMI (XML Metadata Interchange), 
        which seems to be most commonly used today, perhaps handle objects with OIFML.
        Integrate/bundle this with a web-/html-based UML modeling tool or 
        some other interesting stuff like the "Co-operative UML Editor" from Uni Darmstadt. (web-/java-based)
      - Enable Round Trip Engineering. Keep code and diagrams in sync. Don't annoy/bother the programmers.
      - Add support for some more handlers/locators to be able to 
         access the following standards/protocols/interfaces/programs/apis transparently:
        +  DBD::CSV (via Data::Storage::Handler::DBI)
       (-) Text::CSV, XML::CSV, XML::Excel
        -  MAPI
        -  LDAP
        -  DAV (look at PerlDAV: http://www.webdav.org/perldav/)
        -  Mbox (use formail for seperating/splitting entries/nodes)
        -  Cyrus (cyrdeliver - what about cyrretrieve (export)???)
        -  use File::DiffTree, use File::Compare
        -  Hibernate
        -  "Win32::UserAccountDb"
        -  "*nix::UserAccountDb"
        -  .wab - files (Windows Address Book)
        -  .pst - files (Outlook Post Storage?)
        -  XML (e.g. via XML::Simple?)
      - Move to t3, look at InCASE

   LINKS / REFERENCES

      Specs:
        UML 1.3 Spec: http://cgi.omg.org/cgi-bin/doc?ad/99-06-08.pdf
        XMI 1.1 Spec: http://cgi.omg.org/cgi-bin/doc?ad/99-10-02.pdf
        XMI 2.0 Spec: http://cgi.omg.org/docs/ad/01-06-12.pdf
        ODMG: http://odmg.org/
        OIFML: http://odmg.org/library/readingroom/oifml.pdf

      CASE Tools:
        Rational Rose (commercial): http://www.rational.com/products/rose/
        Together (commercial): http://www.oi.com/products/controlcenter/index.jsp
        InCASE - Tangram-based Universal Object Editor 
        Sybase PowerDesigner: http://www.sybase.com/powerdesigner
  
      UML Editors:
        Fujaba (free, university): http://www.fujaba.de/
        ArgoUML (free): http://argouml.tigris.org/
        Poseidon (commercial): http://www.gentleware.com/products/poseidonDE.php3
        Co-operative UML Editor (research): http://www.darmstadt.gmd.de/concert/activities/internal/umledit.html
        Metamill (commercial): http://www.metamill.com/
        Violet (university, research, education): http://www.horstmann.com/violet/
        PyUt (free): http://pyut.sourceforge.net/
        (Dia (free): http://www.lysator.liu.se/~alla/dia/)
        UMLet (free, university): http://www.swt.tuwien.ac.at/umlet/index.html
        Voodoo (free): http://voodoo.sourceforge.net/
        Umbrello UML Modeller: http://uml.sourceforge.net/

      UML Tools:
        http://www.objectsbydesign.com/tools/umltools_byPrice.html

      Further readings:
        http://www.google.com/search?q=web+based+uml+editor&hl=en&lr=&ie=UTF-8&oe=UTF-8&start=10&sa=N
        http://www.fernuni-hagen.de/DVT/Aktuelles/01FHHeidelberg.pdf
        http://www.enhyper.com/src/documentation/
        http://cis.cs.tu-berlin.de/Dokumente/Diplomarbeiten/2001/skinner.pdf
        http://citeseer.nj.nec.com/vilain00diagrammatic.html
        http://archive.devx.com/uml/articles/Smith01/Smith01-3.asp

1	joko	1.1	NAME
2			Data::Storage - Interface for accessing various Storage implementations
3			for Perl in an independent way
4
5	joko	1.4	AIMS
6			- should encapsulate Tangram, DBI, DBD::CSV and LWP:: to access them in an unordinary (more convenient) way ;)
7			- introduce a generic layered structure, refactor SUBLAYER-stuff, make (e.g.) this possible:
8			Perl Data::Storage[DBD::CSV] -> Perl LWP:: -> Internet HTTP/FTP/* -> Host Daemon -> csv-file
9			- provide generic synchronization mechanisms across arbitrary/multiple storages based on ident/checksum
10			maybe it's possible to have schema-, structural- and semantical modifications synchronized???
11
12	joko	1.1	SYNOPSIS
13	joko	1.4	BASIC ACCESS
14
15			ADVANCED ACCESS
16	joko	1.1
17			... via inheritance:
18
19			use Data::Storage;
20			my $proxyObj = new HttpProxy;
21			$proxyObj->{url} = $url;
22			$proxyObj->{payload} = $content;
23			$self->{storage}->insert($proxyObj);
24
25			use Data::Storage;
26			my $proxyObj = HttpProxy->new(
27			url => $url,
28			payload => $content,
29			);
30			$self->{storage}->insert($proxyObj);
31
32	joko	1.4	SYNCHRONIZATION
33
34			my $nodemapping = {
35			'LangText' => 'langtexts.csv',
36			'Currency' => 'currencies.csv',
37			'Country' => 'countries.csv',
38			};
39
40			my $propmapping = {
41			'LangText' => [
42			[ 'source:lcountrykey' => 'target:country' ],
43			[ 'source:lkey' => 'target:key' ],
44			[ 'source:lvalue' => 'target:text' ],
45			],
46			'Currency' => [
47			[ 'source:ckey' => 'target:key' ],
48			[ 'source:cname' => 'target:text' ],
49			],
50			'Country' => [
51			[ 'source:ckey' => 'target:key' ],
52			[ 'source:cname' => 'target:text' ],
53			],
54			};
55
56			sub syncResource {
57
58			my $self = shift;
59			my $node_source = shift;
60			my $mode = shift;
61			my $opts = shift;
62
63			$mode \|\|= '';
64			$opts->{erase} \|\|= 0;
65
66			$logger->info( __PACKAGE__ . "->syncResource( node_source $node_source mode $mode erase $opts->{erase} )");
67
68			# resolve metadata for syncing requested resource
69			my $node_target = $nodemapping->{$node_source};
70			my $mapping = $propmapping->{$node_source};
71
72			if (!$node_target \|\| !$mapping) {
73			# loggger.... "no target, sorry!"
74			print "error while resolving resource metadata", "\n";
75			return;
76			}
77
78			if ($opts->{erase}) {
79			$self->_erase_all($node_source);
80			}
81
82			# create new sync object
83			my $sync = Data::Transfer::Sync->new(
84			storages => {
85			L => $self->{bizWorks}->{backend},
86			R => $self->{bizWorks}->{resources},
87			},
88			id_authorities => [qw( L ) ],
89			checksum_authorities => [qw( L ) ],
90			write_protected => [qw( R ) ],
91			verbose => 1,
92			);
93
94			# sync
95			# todo: filter!?
96			$sync->syncNodes( {
97			direction => $mode, # \| +PUSH \| +PULL \| -FULL \| +IMPORT \| -EXPORT
98			method => 'checksum', # \| -timestamp \| -manual
99			source => "L:$node_source",
100			source_ident => 'storage_method:id',
101			source_exclude => [qw( id cs )],
102			target => "R:$node_target",
103			target_ident => 'property:oid',
104			mapping => $mapping,
105			} );
106
107			}
108
109	joko	1.1	NOTE
110
111			This module heavily relies on DBI and Tangram, but adds a lot of
112	joko	1.3	additional bugs and quirks. Please look at their documentation and/or
113			this code for additional information.
114
115			REQUIREMENTS
116	joko	1.4	For full functionality:
117			DBI from CPAN
118			DBD::mysql from CPAN
119			Tangram 2.04 from CPAN (hmmm, 2.04 won't do in some cases)
120			Tangram 2.05 from http://... (2.05 seems okay but there are also additional patches from our side)
121			Class::Tangram from CPAN
122			DBD::CSV from CPAN
123			MySQL::Diff from http://adamspiers.org/computing/mysqldiff/
124			... and all their dependencies
125	joko	1.1
126			DESCRIPTION
127	joko	1.4	Data::Storage is a module for accessing various "data structures" stored
128	joko	1.1	inside various "data containers". It sits on top of DBI and/or Tangram.
129
130			AUTHORS / COPYRIGHT
131			The Data::Storage module is Copyright (c) 2002 Andreas Motl. All rights
132			reserved.
133
134			You may distribute it under the terms of either the GNU General Public
135			License or the Artistic License, as specified in the Perl README file.
136
137			ACKNOWLEDGEMENTS
138	joko	1.3	Larry Wall for Perl, Tim Bunce for DBI, Jean-Louis Leroy for Tangram and
139	joko	1.4	Set::Object, Sam Vilain for Class::Tangram, Jochen Wiedmann and Jeff
140			Zucker for DBD::CSV and related, Adam Spiers for MySQL::Diff and all
141			contributors.
142	joko	1.1
143			SUPPORT / WARRANTY
144			Data::Storage is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.
145
146			TODO
147	joko	1.4	BUGS
148	joko	1.1
149	joko	1.4	"DBI-Error [Tangram]: DBD::mysql::st execute failed: Unknown column
150			't1.requestdump' in 'field list'"
151	joko	1.1
152	joko	1.4	... occours when operating on object-attributes not introduced yet:
153			this should be detected and appended/replaced through:
154			"Schema-Error detected, maybe (just) an inconsistency.
155			Please check if your declaration in schema-module "a" matches structure in database "b" or try to run"
156			db_setup.pl --dbkey=import --action=deploy
157	joko	1.1
158	joko	1.4	Compare schema (structure diff) with database ...
159	joko	1.1
160			... when issuing "db_setup.pl --dbkey=import --action=deploy"
161			on a database with an already deployed schema, use an additional "--update" then
162			to lift the schema inside the database to the current declared schema.
163			You will have to approve removals and changes on field-level while
164			new objects and new fields are introduced silently without any interaction needed.
165			In future versions there may be additional options to control silent processing of
166			removals and changes.
167			See this CRUD-table applying to the actions occouring on Classes and Class variables when deploying schemas,
168			don't mix this up with CRUD-actions on Objects, these are already handled by (e.g.) Tangram itself.
169			Classes:
170			C create -> yes, handled automatically
171			R retrieve -> no, not subject of this aspect since it is about deployment only
172			U update -> yes, automatically for Class meta-attributes, yes/no for Class variables (look at the rules down here)
173			D delete -> yes, just by user-interaction
174			Class variables:
175			C create -> yes, handled automatically
176			R retrieve -> no, not subject of this aspect since it is about deployment only
177			U update -> yes, just by user-interaction; maybe automatically if it can be determined that data wouldn't be lost
178			D delete -> yes, just by user-interaction
179	joko	1.2
180			It's all about not to be able to loose data simply while this is in pre-alpha stage.
181			And loosing data by being able to modify and redeploy schemas easily is definitely quite easy.
182
183			As we can see, creations of Classes and new Class variables is handled
184			automatically and this is believed to be the most common case under normal circumstances.
185	joko	1.1
186	joko	1.4	FEATURES
187	joko	1.1
188	joko	1.2	- Get this stuff together with UML (Unified Modeling Language) and/or standards from ODMG.
189			- Make it possible to load/save schemas in XMI (XML Metadata Interchange),
190			which seems to be most commonly used today, perhaps handle objects with OIFML.
191			Integrate/bundle this with a web-/html-based UML modeling tool or
192			some other interesting stuff like the "Co-operative UML Editor" from Uni Darmstadt. (web-/java-based)
193			- Enable Round Trip Engineering. Keep code and diagrams in sync. Don't annoy/bother the programmers.
194	joko	1.4	- Add support for some more handlers/locators to be able to
195			access the following standards/protocols/interfaces/programs/apis transparently:
196			+ DBD::CSV (via Data::Storage::Handler::DBI)
197			(-) Text::CSV, XML::CSV, XML::Excel
198			- MAPI
199			- LDAP
200			- DAV (look at PerlDAV: http://www.webdav.org/perldav/)
201			- Mbox (use formail for seperating/splitting entries/nodes)
202			- Cyrus (cyrdeliver - what about cyrretrieve (export)???)
203			- use File::DiffTree, use File::Compare
204			- Hibernate
205			- "Win32::UserAccountDb"
206			- "*nix::UserAccountDb"
207			- .wab - files (Windows Address Book)
208			- .pst - files (Outlook Post Storage?)
209			- XML (e.g. via XML::Simple?)
210			- Move to t3, look at InCASE
211	joko	1.1
212	joko	1.4	LINKS / REFERENCES
213	joko	1.1
214	joko	1.2	Specs:
215	joko	1.1	UML 1.3 Spec: http://cgi.omg.org/cgi-bin/doc?ad/99-06-08.pdf
216			XMI 1.1 Spec: http://cgi.omg.org/cgi-bin/doc?ad/99-10-02.pdf
217			XMI 2.0 Spec: http://cgi.omg.org/docs/ad/01-06-12.pdf
218			ODMG: http://odmg.org/
219			OIFML: http://odmg.org/library/readingroom/oifml.pdf
220
221	joko	1.2	CASE Tools:
222			Rational Rose (commercial): http://www.rational.com/products/rose/
223			Together (commercial): http://www.oi.com/products/controlcenter/index.jsp
224			InCASE - Tangram-based Universal Object Editor
225			Sybase PowerDesigner: http://www.sybase.com/powerdesigner
226
227			UML Editors:
228			Fujaba (free, university): http://www.fujaba.de/
229			ArgoUML (free): http://argouml.tigris.org/
230			Poseidon (commercial): http://www.gentleware.com/products/poseidonDE.php3
231			Co-operative UML Editor (research): http://www.darmstadt.gmd.de/concert/activities/internal/umledit.html
232			Metamill (commercial): http://www.metamill.com/
233			Violet (university, research, education): http://www.horstmann.com/violet/
234			PyUt (free): http://pyut.sourceforge.net/
235			(Dia (free): http://www.lysator.liu.se/~alla/dia/)
236			UMLet (free, university): http://www.swt.tuwien.ac.at/umlet/index.html
237			Voodoo (free): http://voodoo.sourceforge.net/
238	joko	1.3	Umbrello UML Modeller: http://uml.sourceforge.net/
239	joko	1.2
240			UML Tools:
241			http://www.objectsbydesign.com/tools/umltools_byPrice.html
242
243			Further readings:
244	joko	1.1	http://www.google.com/search?q=web+based+uml+editor&hl=en&lr=&ie=UTF-8&oe=UTF-8&start=10&sa=N
245			http://www.fernuni-hagen.de/DVT/Aktuelles/01FHHeidelberg.pdf
246			http://www.enhyper.com/src/documentation/
247			http://cis.cs.tu-berlin.de/Dokumente/Diplomarbeiten/2001/skinner.pdf
248			http://citeseer.nj.nec.com/vilain00diagrammatic.html
249			http://archive.devx.com/uml/articles/Smith01/Smith01-3.asp
250
MailToCvsAdmin">MailToCvsAdmin	ViewVC Help
Powered by ViewVC 1.1.26