libs/Data/Storage.pod

=pod


=head1 NAME

  Data::Storage - Interface for accessing various Storage implementations for Perl in an independent way


=head1 AIMS

  - should encapsulate Tangram, DBI, DBD::CSV and LWP:: to access them in an unordinary (more convenient) way ;)
  - introduce a generic layered structure, refactor *SUBLAYER*-stuff, make (e.g.) this possible:
    Perl Data::Storage[DBD::CSV]  ->  Perl LWP::  ->  Internet HTTP/FTP/*  ->  Host Daemon  ->  csv-file
  - provide generic synchronization mechanisms across arbitrary/multiple storages based on ident/checksum
    maybe it's possible to have schema-, structural- and semantical modifications synchronized???


=head1 SYNOPSIS

=head2 BASIC ACCESS

=head2 ADVANCED ACCESS

  ... via inheritance:
  
    use Data::Storage;
    my $proxyObj = new HttpProxy;
    $proxyObj->{url} = $url;
    $proxyObj->{payload} = $content;
    $self->{storage}->insert($proxyObj);
    
    use Data::Storage;
    my $proxyObj = HttpProxy->new(
      url => $url,
      payload => $content,
    );
    $self->{storage}->insert($proxyObj);


=head2 SYNCHRONIZATION

  my $nodemapping = {
    'LangText' => 'langtexts.csv',
    'Currency' => 'currencies.csv',
    'Country'  => 'countries.csv',
  };

  my $propmapping = {
    'LangText' => [
      [ 'source:lcountrykey'  =>  'target:country' ],
      [ 'source:lkey'         =>  'target:key' ],
      [ 'source:lvalue'       =>  'target:text' ],
    ],
    'Currency' => [
      [ 'source:ckey'         =>  'target:key' ],
      [ 'source:cname'        =>  'target:text' ],
    ],
    'Country' => [
      [ 'source:ckey'         =>  'target:key' ],
      [ 'source:cname'        =>  'target:text' ],
    ],
  };

  s ub syncResource {

    my $self = shift;
    my $node_source = shift;
    my $mode = shift;
    my $opts = shift;
    
    $mode ||= '';
    $opts->{erase} ||= 0;
    
    $logger->info( __PACKAGE__ . "->syncResource( node_source $node_source mode $mode erase $opts->{erase} )");
  
    # resolve metadata for syncing requested resource
    my $node_target = $nodemapping->{$node_source};
    my $mapping = $propmapping->{$node_source};
    
    if (!$node_target || !$mapping) {
      # loggger.... "no target, sorry!"
      print "error while resolving resource metadata", "\n";
      return;
    }
    
    if ($opts->{erase}) {
      $self->_erase_all($node_source);
    }
  
    # create new sync object
    my $sync = Data::Transfer::Sync->new( 
      storages => {
        L => $self->{bizWorks}->{backend},
        R => $self->{bizWorks}->{resources},
      },
      id_authorities        =>  [qw( L ) ],
      checksum_authorities  =>  [qw( L ) ],
      write_protected       =>  [qw( R ) ],
      verbose               =>  1,
    );
    
    # sync
    # todo: filter!?
    $sync->syncNodes( {
      direction       =>  $mode,                 # | +PUSH | +PULL | -FULL | +IMPORT | -EXPORT
      method          =>  'checksum',            # | -timestamp | -manual
      source          =>  "L:$node_source",
      source_ident    =>  'storage_method:id',
      source_exclude  =>  [qw( id cs )],
      target          =>  "R:$node_target",
      target_ident    =>  'property:oid',
      mapping         =>  $mapping,
    } );

  }


=head2 NOTE

  This module heavily relies on DBI and Tangram, but adds a lot of additional bugs and quirks. 
  Please look at their documentation and/or this code for additional information.


=head1 REQUIREMENTS

  For full functionality:
    DBI              from CPAN
    DBD::mysql       from CPAN
    Tangram 2.04     from CPAN         (hmmm, 2.04 won't do in some cases)
    Tangram 2.05     from http://...   (2.05 seems okay but there are also additional patches from our side)
    Class::Tangram   from CPAN
    DBD::CSV         from CPAN
    MySQL::Diff      from http://adamspiers.org/computing/mysqldiff/
    ... and all their dependencies


=head1 DESCRIPTION

=head2 Data::Storage

  Data::Storage is a module for accessing various "data structures / kinds of structured data" stored inside
  various "data containers".
  We tried to use the AdapterPattern to implement a wrapper-layer around known CPAN modules.
  (e.g. DBI, Tangram, XML::Simple)
  References:
  - http://c2.com/cgi/wiki?AdapterPattern
  - http://home.earthlink.net/~huston2/dp/adapter.html

=head2 Why?

  You will get a better code-structure (not bad for later maintenance) in growing Perl code projects,
  especially when using multiple database connections at the same time.
  You will be able to switch between different _kinds_ of implementations used for storing data.
  Your code will use the very same API to access these storage layers.
      ... implementation has to be changed for now
  Maybe you will be able to switch "on-the-fly" without changing any bits in code in the future.... 
      ... but that's not the focus

=head2 What else?

  Having this, we were able to do implement a generic data synchronization module more easy,
  please look at Data::Transfer.


=head1 AUTHORS / COPYRIGHT

  The Data::Storage module is Copyright (c) 2002 Andreas Motl.
  All rights reserved.
  You may distribute it under the terms of either the GNU General Public
  License or the Artistic License, as specified in the Perl README file.


=head1 ACKNOWLEDGEMENTS

  Larry Wall for Perl, Tim Bunce for DBI, Jean-Louis Leroy for Tangram and Set::Object, 
  Sam Vilain for Class::Tangram, Jochen Wiedmann and Jeff Zucker for DBD::CSV & Co.,
  Adam Spiers for MySQL::Diff and all contributors.


=head1 SUPPORT / WARRANTY

  Data::Storage is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.


=head1 TODO


=head2 BUGS

"DBI-Error [Tangram]: DBD::mysql::st execute failed: Unknown column 't1.requestdump' in 'field list'"

  ... occours when operating on object-attributes not introduced yet:
  this should be detected and appended/replaced through:
  "Schema-Error detected, maybe (just) an inconsistency. 
  Please check if your declaration in schema-module "a" matches structure in database "b" or try to run"
  db_setup.pl --dbkey=import --action=deploy


Compare schema (structure diff) with database ... 

  ... when issuing "db_setup.pl --dbkey=import --action=deploy"
  on a database with an already deployed schema, use an additional "--update" then
  to lift the schema inside the database to the current declared schema.
  You will have to approve removals and changes on field-level while
  new objects and new fields are introduced silently without any interaction needed.
  In future versions there may be additional options to control silent processing of
  removals and changes.
  See this CRUD-table applying to the actions occouring on Classes and Class variables when deploying schemas,
  don't mix this up with CRUD-actions on Objects, these are already handled by (e.g.) Tangram itself.
  Classes:
    C create    ->  yes, handled automatically
    R retrieve  ->  no, not subject of this aspect since it is about deployment only
    U update    ->  yes, automatically for Class meta-attributes, yes/no for Class variables (look at the rules down here)
    D delete    ->  yes, just by user-interaction
  Class variables:
    C create    ->  yes, handled automatically
    R retrieve  ->  no, not subject of this aspect since it is about deployment only
    U update    ->  yes, just by user-interaction; maybe automatically if it can be determined that data wouldn't be lost
    D delete    ->  yes, just by user-interaction
  
  It's all about not to be able to loose data simply while this is in pre-alpha stage.
  And loosing data by being able to modify and redeploy schemas easily is definitely quite easy.
  
  As we can see, creations of Classes and new Class variables is handled 
  automatically and this is believed to be the most common case under normal circumstances.


=head2 FEATURES

  - Get this stuff together with UML (Unified Modeling Language) and/or standards from ODMG.
  - Make it possible to load/save schemas in XMI (XML Metadata Interchange), 
    which seems to be most commonly used today, perhaps handle objects with OIFML.
    Integrate/bundle this with a web-/html-based UML modeling tool or 
    some other interesting stuff like the "Co-operative UML Editor" from Uni Darmstadt. (web-/java-based)
  - Enable Round Trip Engineering. Keep code and diagrams in sync. Don't annoy/bother the programmers.
  - Add support for some more handlers/locators to be able to 
     access the following standards/protocols/interfaces/programs/apis transparently:
    +  DBD::CSV (via Data::Storage::Handler::DBI)
   (-) Text::CSV, XML::CSV, XML::Excel
    -  MAPI
    -  LDAP
    -  DAV (look at PerlDAV: http://www.webdav.org/perldav/)
    -  Mbox (use formail for seperating/splitting entries/nodes)
    -  Cyrus (cyrdeliver - what about cyrretrieve (export)???)
    -  use File::DiffTree, use File::Compare
    -  Hibernate
    -  "Win32::UserAccountDb"
    -  "*nix::UserAccountDb"
    -  .wab - files (Windows Address Book)
    -  .pst - files (Outlook Post Storage?)
    -  XML (e.g. via XML::Simple?)
  - Move to t3, look at InCASE
  - some kind of security layer for methods/objects
    - acls (stored via tangram/ldap?) for functions, methods and objects (entity- & data!?)
    - where are the hooks needed then?
      - is Data::Storage & Co. okay, or do we have to touch the innards of DBI and/or Tangram?
      - an attempt to start could be: 
         - 'sub getACLByObjectId($id, $context)'
         - 'sub getACLByMethodname($id, $context)'
         - 'sub getACLByName($id, $context)'
            ( would require a kinda registry to look up these very names pointing to arbitrary locations (code, data, ...) )
  - add more hooks and various levels
  - better integrate introduced 'getObjectByGuid'-mechanism from Data::Storage::Handler::Tangram


=head3 LINKS / REFERENCES

  Specs:
    UML 1.3 Spec: http://cgi.omg.org/cgi-bin/doc?ad/99-06-08.pdf
    XMI 1.1 Spec: http://cgi.omg.org/cgi-bin/doc?ad/99-10-02.pdf
    XMI 2.0 Spec: http://cgi.omg.org/docs/ad/01-06-12.pdf
    ODMG: http://odmg.org/
    OIFML: http://odmg.org/library/readingroom/oifml.pdf

  CASE Tools:
    Rational Rose (commercial): http://www.rational.com/products/rose/
    Together (commercial): http://www.oi.com/products/controlcenter/index.jsp
    InCASE - Tangram-based Universal Object Editor 
    Sybase PowerDesigner: http://www.sybase.com/powerdesigner
  
  UML Editors:
    Fujaba (free, university): http://www.fujaba.de/
    ArgoUML (free): http://argouml.tigris.org/
    Poseidon (commercial): http://www.gentleware.com/products/poseidonDE.php3
    Co-operative UML Editor (research): http://www.darmstadt.gmd.de/concert/activities/internal/umledit.html
    Metamill (commercial): http://www.metamill.com/
    Violet (university, research, education): http://www.horstmann.com/violet/
    PyUt (free): http://pyut.sourceforge.net/
    (Dia (free): http://www.lysator.liu.se/~alla/dia/)
    UMLet (free, university): http://www.swt.tuwien.ac.at/umlet/index.html
    Voodoo (free): http://voodoo.sourceforge.net/
    Umbrello UML Modeller: http://uml.sourceforge.net/

  UML Tools:
    http://www.objectsbydesign.com/tools/umltools_byPrice.html

  Further readings:
    http://www.google.com/search?q=web+based+uml+editor&hl=en&lr=&ie=UTF-8&oe=UTF-8&start=10&sa=N
    http://www.fernuni-hagen.de/DVT/Aktuelles/01FHHeidelberg.pdf
    http://www.enhyper.com/src/documentation/
    http://cis.cs.tu-berlin.de/Dokumente/Diplomarbeiten/2001/skinner.pdf
    http://citeseer.nj.nec.com/vilain00diagrammatic.html
    http://archive.devx.com/uml/articles/Smith01/Smith01-3.asp


=cut

1	joko	1.1	=pod
2
3
4			=head1 NAME
5
6			Data::Storage - Interface for accessing various Storage implementations for Perl in an independent way
7
8
9			=head1 AIMS
10
11			- should encapsulate Tangram, DBI, DBD::CSV and LWP:: to access them in an unordinary (more convenient) way ;)
12			- introduce a generic layered structure, refactor SUBLAYER-stuff, make (e.g.) this possible:
13			Perl Data::Storage[DBD::CSV] -> Perl LWP:: -> Internet HTTP/FTP/* -> Host Daemon -> csv-file
14			- provide generic synchronization mechanisms across arbitrary/multiple storages based on ident/checksum
15			maybe it's possible to have schema-, structural- and semantical modifications synchronized???
16
17
18			=head1 SYNOPSIS
19
20			=head2 BASIC ACCESS
21
22			=head2 ADVANCED ACCESS
23
24			... via inheritance:
25
26			use Data::Storage;
27			my $proxyObj = new HttpProxy;
28			$proxyObj->{url} = $url;
29			$proxyObj->{payload} = $content;
30			$self->{storage}->insert($proxyObj);
31
32			use Data::Storage;
33			my $proxyObj = HttpProxy->new(
34			url => $url,
35			payload => $content,
36			);
37			$self->{storage}->insert($proxyObj);
38
39
40			=head2 SYNCHRONIZATION
41
42			my $nodemapping = {
43			'LangText' => 'langtexts.csv',
44			'Currency' => 'currencies.csv',
45			'Country' => 'countries.csv',
46			};
47
48			my $propmapping = {
49			'LangText' => [
50			[ 'source:lcountrykey' => 'target:country' ],
51			[ 'source:lkey' => 'target:key' ],
52			[ 'source:lvalue' => 'target:text' ],
53			],
54			'Currency' => [
55			[ 'source:ckey' => 'target:key' ],
56			[ 'source:cname' => 'target:text' ],
57			],
58			'Country' => [
59			[ 'source:ckey' => 'target:key' ],
60			[ 'source:cname' => 'target:text' ],
61			],
62			};
63
64			s ub syncResource {
65
66			my $self = shift;
67			my $node_source = shift;
68			my $mode = shift;
69			my $opts = shift;
70
71			$mode \|\|= '';
72			$opts->{erase} \|\|= 0;
73
74			$logger->info( __PACKAGE__ . "->syncResource( node_source $node_source mode $mode erase $opts->{erase} )");
75
76			# resolve metadata for syncing requested resource
77			my $node_target = $nodemapping->{$node_source};
78			my $mapping = $propmapping->{$node_source};
79
80			if (!$node_target \|\| !$mapping) {
81			# loggger.... "no target, sorry!"
82			print "error while resolving resource metadata", "\n";
83			return;
84			}
85
86			if ($opts->{erase}) {
87			$self->_erase_all($node_source);
88			}
89
90			# create new sync object
91			my $sync = Data::Transfer::Sync->new(
92			storages => {
93			L => $self->{bizWorks}->{backend},
94			R => $self->{bizWorks}->{resources},
95			},
96			id_authorities => [qw( L ) ],
97			checksum_authorities => [qw( L ) ],
98			write_protected => [qw( R ) ],
99			verbose => 1,
100			);
101
102			# sync
103			# todo: filter!?
104			$sync->syncNodes( {
105			direction => $mode, # \| +PUSH \| +PULL \| -FULL \| +IMPORT \| -EXPORT
106			method => 'checksum', # \| -timestamp \| -manual
107			source => "L:$node_source",
108			source_ident => 'storage_method:id',
109			source_exclude => [qw( id cs )],
110			target => "R:$node_target",
111			target_ident => 'property:oid',
112			mapping => $mapping,
113			} );
114
115			}
116
117
118			=head2 NOTE
119
120			This module heavily relies on DBI and Tangram, but adds a lot of additional bugs and quirks.
121			Please look at their documentation and/or this code for additional information.
122
123
124			=head1 REQUIREMENTS
125
126			For full functionality:
127			DBI from CPAN
128			DBD::mysql from CPAN
129			Tangram 2.04 from CPAN (hmmm, 2.04 won't do in some cases)
130			Tangram 2.05 from http://... (2.05 seems okay but there are also additional patches from our side)
131			Class::Tangram from CPAN
132			DBD::CSV from CPAN
133			MySQL::Diff from http://adamspiers.org/computing/mysqldiff/
134			... and all their dependencies
135
136
137			=head1 DESCRIPTION
138
139			=head2 Data::Storage
140
141			Data::Storage is a module for accessing various "data structures / kinds of structured data" stored inside
142			various "data containers".
143			We tried to use the AdapterPattern to implement a wrapper-layer around known CPAN modules.
144			(e.g. DBI, Tangram, XML::Simple)
145			References:
146			- http://c2.com/cgi/wiki?AdapterPattern
147			- http://home.earthlink.net/~huston2/dp/adapter.html
148
149			=head2 Why?
150
151			You will get a better code-structure (not bad for later maintenance) in growing Perl code projects,
152			especially when using multiple database connections at the same time.
153			You will be able to switch between different _kinds_ of implementations used for storing data.
154			Your code will use the very same API to access these storage layers.
155			... implementation has to be changed for now
156			Maybe you will be able to switch "on-the-fly" without changing any bits in code in the future....
157			... but that's not the focus
158
159			=head2 What else?
160
161			Having this, we were able to do implement a generic data synchronization module more easy,
162			please look at Data::Transfer.
163
164
165			=head1 AUTHORS / COPYRIGHT
166
167			The Data::Storage module is Copyright (c) 2002 Andreas Motl.
168			All rights reserved.
169			You may distribute it under the terms of either the GNU General Public
170			License or the Artistic License, as specified in the Perl README file.
171
172
173			=head1 ACKNOWLEDGEMENTS
174
175			Larry Wall for Perl, Tim Bunce for DBI, Jean-Louis Leroy for Tangram and Set::Object,
176			Sam Vilain for Class::Tangram, Jochen Wiedmann and Jeff Zucker for DBD::CSV & Co.,
177			Adam Spiers for MySQL::Diff and all contributors.
178
179
180			=head1 SUPPORT / WARRANTY
181
182			Data::Storage is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.
183
184
185			=head1 TODO
186
187
188			=head2 BUGS
189
190			"DBI-Error [Tangram]: DBD::mysql::st execute failed: Unknown column 't1.requestdump' in 'field list'"
191
192			... occours when operating on object-attributes not introduced yet:
193			this should be detected and appended/replaced through:
194			"Schema-Error detected, maybe (just) an inconsistency.
195			Please check if your declaration in schema-module "a" matches structure in database "b" or try to run"
196			db_setup.pl --dbkey=import --action=deploy
197
198
199			Compare schema (structure diff) with database ...
200
201			... when issuing "db_setup.pl --dbkey=import --action=deploy"
202			on a database with an already deployed schema, use an additional "--update" then
203			to lift the schema inside the database to the current declared schema.
204			You will have to approve removals and changes on field-level while
205			new objects and new fields are introduced silently without any interaction needed.
206			In future versions there may be additional options to control silent processing of
207			removals and changes.
208			See this CRUD-table applying to the actions occouring on Classes and Class variables when deploying schemas,
209			don't mix this up with CRUD-actions on Objects, these are already handled by (e.g.) Tangram itself.
210			Classes:
211			C create -> yes, handled automatically
212			R retrieve -> no, not subject of this aspect since it is about deployment only
213			U update -> yes, automatically for Class meta-attributes, yes/no for Class variables (look at the rules down here)
214			D delete -> yes, just by user-interaction
215			Class variables:
216			C create -> yes, handled automatically
217			R retrieve -> no, not subject of this aspect since it is about deployment only
218			U update -> yes, just by user-interaction; maybe automatically if it can be determined that data wouldn't be lost
219			D delete -> yes, just by user-interaction
220
221			It's all about not to be able to loose data simply while this is in pre-alpha stage.
222			And loosing data by being able to modify and redeploy schemas easily is definitely quite easy.
223
224			As we can see, creations of Classes and new Class variables is handled
225			automatically and this is believed to be the most common case under normal circumstances.
226
227
228			=head2 FEATURES
229
230			- Get this stuff together with UML (Unified Modeling Language) and/or standards from ODMG.
231			- Make it possible to load/save schemas in XMI (XML Metadata Interchange),
232			which seems to be most commonly used today, perhaps handle objects with OIFML.
233			Integrate/bundle this with a web-/html-based UML modeling tool or
234			some other interesting stuff like the "Co-operative UML Editor" from Uni Darmstadt. (web-/java-based)
235			- Enable Round Trip Engineering. Keep code and diagrams in sync. Don't annoy/bother the programmers.
236			- Add support for some more handlers/locators to be able to
237			access the following standards/protocols/interfaces/programs/apis transparently:
238			+ DBD::CSV (via Data::Storage::Handler::DBI)
239			(-) Text::CSV, XML::CSV, XML::Excel
240			- MAPI
241			- LDAP
242			- DAV (look at PerlDAV: http://www.webdav.org/perldav/)
243			- Mbox (use formail for seperating/splitting entries/nodes)
244			- Cyrus (cyrdeliver - what about cyrretrieve (export)???)
245			- use File::DiffTree, use File::Compare
246			- Hibernate
247			- "Win32::UserAccountDb"
248			- "*nix::UserAccountDb"
249			- .wab - files (Windows Address Book)
250			- .pst - files (Outlook Post Storage?)
251			- XML (e.g. via XML::Simple?)
252			- Move to t3, look at InCASE
253			- some kind of security layer for methods/objects
254			- acls (stored via tangram/ldap?) for functions, methods and objects (entity- & data!?)
255			- where are the hooks needed then?
256			- is Data::Storage & Co. okay, or do we have to touch the innards of DBI and/or Tangram?
257			- an attempt to start could be:
258			- 'sub getACLByObjectId($id, $context)'
259			- 'sub getACLByMethodname($id, $context)'
260			- 'sub getACLByName($id, $context)'
261			( would require a kinda registry to look up these very names pointing to arbitrary locations (code, data, ...) )
262			- add more hooks and various levels
263			- better integrate introduced 'getObjectByGuid'-mechanism from Data::Storage::Handler::Tangram
264
265
266			=head3 LINKS / REFERENCES
267
268			Specs:
269			UML 1.3 Spec: http://cgi.omg.org/cgi-bin/doc?ad/99-06-08.pdf
270			XMI 1.1 Spec: http://cgi.omg.org/cgi-bin/doc?ad/99-10-02.pdf
271			XMI 2.0 Spec: http://cgi.omg.org/docs/ad/01-06-12.pdf
272			ODMG: http://odmg.org/
273			OIFML: http://odmg.org/library/readingroom/oifml.pdf
274
275			CASE Tools:
276			Rational Rose (commercial): http://www.rational.com/products/rose/
277			Together (commercial): http://www.oi.com/products/controlcenter/index.jsp
278			InCASE - Tangram-based Universal Object Editor
279			Sybase PowerDesigner: http://www.sybase.com/powerdesigner
280
281			UML Editors:
282			Fujaba (free, university): http://www.fujaba.de/
283			ArgoUML (free): http://argouml.tigris.org/
284			Poseidon (commercial): http://www.gentleware.com/products/poseidonDE.php3
285			Co-operative UML Editor (research): http://www.darmstadt.gmd.de/concert/activities/internal/umledit.html
286			Metamill (commercial): http://www.metamill.com/
287			Violet (university, research, education): http://www.horstmann.com/violet/
288			PyUt (free): http://pyut.sourceforge.net/
289			(Dia (free): http://www.lysator.liu.se/~alla/dia/)
290			UMLet (free, university): http://www.swt.tuwien.ac.at/umlet/index.html
291			Voodoo (free): http://voodoo.sourceforge.net/
292			Umbrello UML Modeller: http://uml.sourceforge.net/
293
294			UML Tools:
295			http://www.objectsbydesign.com/tools/umltools_byPrice.html
296
297			Further readings:
298			http://www.google.com/search?q=web+based+uml+editor&hl=en&lr=&ie=UTF-8&oe=UTF-8&start=10&sa=N
299			http://www.fernuni-hagen.de/DVT/Aktuelles/01FHHeidelberg.pdf
300			http://www.enhyper.com/src/documentation/
301			http://cis.cs.tu-berlin.de/Dokumente/Diplomarbeiten/2001/skinner.pdf
302			http://citeseer.nj.nec.com/vilain00diagrammatic.html
303			http://archive.devx.com/uml/articles/Smith01/Smith01-3.asp
304
305
306			=cut
307
MailToCvsAdmin">MailToCvsAdmin	ViewVC Help
Powered by ViewVC 1.1.26