libs/Data/README.text

NAME
      Data::Storage - Interface for accessing various Storage implementations for Perl in an independent way

AIMS
      - should encapsulate Tangram, DBI, DBD::CSV and LWP:: to access them in an unordinary (more convenient) way ;)
      - introduce a generic layered structure, refactor *SUBLAYER*-stuff, make (e.g.) this possible:
        Perl Data::Storage[DBD::CSV]  ->  Perl LWP::  ->  Internet HTTP/FTP/*  ->  Host Daemon  ->  csv-file
      - provide generic synchronization mechanisms across arbitrary/multiple storages based on ident/checksum
        maybe it's possible to have schema-, structural- and semantical modifications synchronized???

SYNOPSIS
  BASIC ACCESS

  ADVANCED ACCESS

      ... via inheritance:
  
        use Data::Storage;
        my $proxyObj = new HttpProxy;
        $proxyObj->{url} = $url;
        $proxyObj->{payload} = $content;
        $self->{storage}->insert($proxyObj);
    
        use Data::Storage;
        my $proxyObj = HttpProxy->new(
          url => $url,
          payload => $content,
        );
        $self->{storage}->insert($proxyObj);

  SYNCHRONIZATION

      my $nodemapping = {
        'LangText' => 'langtexts.csv',
        'Currency' => 'currencies.csv',
        'Country'  => 'countries.csv',
      };

      my $propmapping = {
        'LangText' => [
          [ 'source:lcountrykey'  =>  'target:country' ],
          [ 'source:lkey'         =>  'target:key' ],
          [ 'source:lvalue'       =>  'target:text' ],
        ],
        'Currency' => [
          [ 'source:ckey'         =>  'target:key' ],
          [ 'source:cname'        =>  'target:text' ],
        ],
        'Country' => [
          [ 'source:ckey'         =>  'target:key' ],
          [ 'source:cname'        =>  'target:text' ],
        ],
      };

      sub syncResource {

        my $self = shift;
        my $node_source = shift;
        my $mode = shift;
        my $opts = shift;
    
        $mode ||= '';
        $opts->{erase} ||= 0;
    
        $logger->info( __PACKAGE__ . "->syncResource( node_source $node_source mode $mode erase $opts->{erase} )");
  
        # resolve metadata for syncing requested resource
        my $node_target = $nodemapping->{$node_source};
        my $mapping = $propmapping->{$node_source};
    
        if (!$node_target || !$mapping) {
          # loggger.... "no target, sorry!"
          print "error while resolving resource metadata", "\n";
          return;
        }
    
        if ($opts->{erase}) {
          $self->_erase_all($node_source);
        }
  
        # create new sync object
        my $sync = Data::Transfer::Sync->new( 
          storages => {
            L => $self->{bizWorks}->{backend},
            R => $self->{bizWorks}->{resources},
          },
          id_authorities        =>  [qw( L ) ],
          checksum_authorities  =>  [qw( L ) ],
          write_protected       =>  [qw( R ) ],
          verbose               =>  1,
        );
    
        # sync
        # todo: filter!?
        $sync->syncNodes( {
          direction       =>  $mode,                 # | +PUSH | +PULL | -FULL | +IMPORT | -EXPORT
          method          =>  'checksum',            # | -timestamp | -manual
          source          =>  "L:$node_source",
          source_ident    =>  'storage_method:id',
          source_exclude  =>  [qw( id cs )],
          target          =>  "R:$node_target",
          target_ident    =>  'property:oid',
          mapping         =>  $mapping,
        } );

      }

  NOTE

      This module heavily relies on DBI and Tangram, but adds a lot of additional bugs and quirks. 
      Please look at their documentation and/or this code for additional information.

REQUIREMENTS
      For full functionality:
        DBI              from CPAN
        DBD::mysql       from CPAN
        Tangram 2.04     from CPAN         (hmmm, 2.04 won't do in some cases)
        Tangram 2.05     from http://...   (2.05 seems okay but there are also additional patches from our side)
        Class::Tangram   from CPAN
        DBD::CSV         from CPAN
        MySQL::Diff      from http://adamspiers.org/computing/mysqldiff/
        ... and all their dependencies

DESCRIPTION
  Data::Storage

      Data::Storage is a module for accessing various "data structures / kinds of structured data" stored inside
      various "data containers".
      We tried to use the AdapterPattern (http://c2.com/cgi/wiki?AdapterPattern) to implement a wrapper-layer
      around core CPAN modules (Tangram, DBI).

  Why?

      You will get a better code-structure (not bad for later maintenance) in growing Perl code projects,
      especially when using multiple database connections at the same time.
      You will be able to switch between different _kinds_ of implementations used for storing data.
      Your code will use the very same API to access these storage layers.
          ... implementation has to be changed for now
      Maybe you will be able to switch "on-the-fly" without changing any bits in code in the future.... 
          ... but that's not the focus

  What else?

      Having this, we were able to do implement a generic data synchronization module more easy,
      please look at Data::Transfer.

AUTHORS / COPYRIGHT
      The Data::Storage module is Copyright (c) 2002 Andreas Motl.
      All rights reserved.
      You may distribute it under the terms of either the GNU General Public
      License or the Artistic License, as specified in the Perl README file.

ACKNOWLEDGEMENTS
      Larry Wall for Perl, Tim Bunce for DBI, Jean-Louis Leroy for Tangram and Set::Object, 
      Sam Vilain for Class::Tangram, Jochen Wiedmann and Jeff Zucker for DBD::CSV & Co.,
      Adam Spiers for MySQL::Diff and all contributors.

SUPPORT / WARRANTY
      Data::Storage is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.

TODO
  BUGS

    "DBI-Error [Tangram]: DBD::mysql::st execute failed: Unknown column
    't1.requestdump' in 'field list'"

      ... occours when operating on object-attributes not introduced yet:
      this should be detected and appended/replaced through:
      "Schema-Error detected, maybe (just) an inconsistency. 
      Please check if your declaration in schema-module "a" matches structure in database "b" or try to run"
      db_setup.pl --dbkey=import --action=deploy

    Compare schema (structure diff) with database ...

      ... when issuing "db_setup.pl --dbkey=import --action=deploy"
      on a database with an already deployed schema, use an additional "--update" then
      to lift the schema inside the database to the current declared schema.
      You will have to approve removals and changes on field-level while
      new objects and new fields are introduced silently without any interaction needed.
      In future versions there may be additional options to control silent processing of
      removals and changes.
      See this CRUD-table applying to the actions occouring on Classes and Class variables when deploying schemas,
      don't mix this up with CRUD-actions on Objects, these are already handled by (e.g.) Tangram itself.
      Classes:
        C create    ->  yes, handled automatically
        R retrieve  ->  no, not subject of this aspect since it is about deployment only
        U update    ->  yes, automatically for Class meta-attributes, yes/no for Class variables (look at the rules down here)
        D delete    ->  yes, just by user-interaction
      Class variables:
        C create    ->  yes, handled automatically
        R retrieve  ->  no, not subject of this aspect since it is about deployment only
        U update    ->  yes, just by user-interaction; maybe automatically if it can be determined that data wouldn't be lost
        D delete    ->  yes, just by user-interaction
  
      It's all about not to be able to loose data simply while this is in pre-alpha stage.
      And loosing data by being able to modify and redeploy schemas easily is definitely quite easy.
  
      As we can see, creations of Classes and new Class variables is handled 
      automatically and this is believed to be the most common case under normal circumstances.

  FEATURES

      - Get this stuff together with UML (Unified Modeling Language) and/or standards from ODMG.
      - Make it possible to load/save schemas in XMI (XML Metadata Interchange), 
        which seems to be most commonly used today, perhaps handle objects with OIFML.
        Integrate/bundle this with a web-/html-based UML modeling tool or 
        some other interesting stuff like the "Co-operative UML Editor" from Uni Darmstadt. (web-/java-based)
      - Enable Round Trip Engineering. Keep code and diagrams in sync. Don't annoy/bother the programmers.
      - Add support for some more handlers/locators to be able to 
         access the following standards/protocols/interfaces/programs/apis transparently:
        +  DBD::CSV (via Data::Storage::Handler::DBI)
       (-) Text::CSV, XML::CSV, XML::Excel
        -  MAPI
        -  LDAP
        -  DAV (look at PerlDAV: http://www.webdav.org/perldav/)
        -  Mbox (use formail for seperating/splitting entries/nodes)
        -  Cyrus (cyrdeliver - what about cyrretrieve (export)???)
        -  use File::DiffTree, use File::Compare
        -  Hibernate
        -  "Win32::UserAccountDb"
        -  "*nix::UserAccountDb"
        -  .wab - files (Windows Address Book)
        -  .pst - files (Outlook Post Storage?)
        -  XML (e.g. via XML::Simple?)
      - Move to t3, look at InCASE
      - some kind of security layer for methods/objects
        - acls (stored via tangram/ldap?) for functions, methods and objects (entity- & data!?)
        - where are the hooks needed then?
          - is Data::Storage & Co. okay, or do we have to touch the innards of DBI and/or Tangram?
          - an attempt to start could be: 
             - 'sub getACLByObjectId($id, $context)'
             - 'sub getACLByMethodname($id, $context)'
             - 'sub getACLByName($id, $context)'
                ( would require a kinda registry to look up these very names pointing to arbitrary locations (code, data, ...) )

   LINKS / REFERENCES

      Specs:
        UML 1.3 Spec: http://cgi.omg.org/cgi-bin/doc?ad/99-06-08.pdf
        XMI 1.1 Spec: http://cgi.omg.org/cgi-bin/doc?ad/99-10-02.pdf
        XMI 2.0 Spec: http://cgi.omg.org/docs/ad/01-06-12.pdf
        ODMG: http://odmg.org/
        OIFML: http://odmg.org/library/readingroom/oifml.pdf

      CASE Tools:
        Rational Rose (commercial): http://www.rational.com/products/rose/
        Together (commercial): http://www.oi.com/products/controlcenter/index.jsp
        InCASE - Tangram-based Universal Object Editor 
        Sybase PowerDesigner: http://www.sybase.com/powerdesigner
  
      UML Editors:
        Fujaba (free, university): http://www.fujaba.de/
        ArgoUML (free): http://argouml.tigris.org/
        Poseidon (commercial): http://www.gentleware.com/products/poseidonDE.php3
        Co-operative UML Editor (research): http://www.darmstadt.gmd.de/concert/activities/internal/umledit.html
        Metamill (commercial): http://www.metamill.com/
        Violet (university, research, education): http://www.horstmann.com/violet/
        PyUt (free): http://pyut.sourceforge.net/
        (Dia (free): http://www.lysator.liu.se/~alla/dia/)
        UMLet (free, university): http://www.swt.tuwien.ac.at/umlet/index.html
        Voodoo (free): http://voodoo.sourceforge.net/
        Umbrello UML Modeller: http://uml.sourceforge.net/

      UML Tools:
        http://www.objectsbydesign.com/tools/umltools_byPrice.html

      Further readings:
        http://www.google.com/search?q=web+based+uml+editor&hl=en&lr=&ie=UTF-8&oe=UTF-8&start=10&sa=N
        http://www.fernuni-hagen.de/DVT/Aktuelles/01FHHeidelberg.pdf
        http://www.enhyper.com/src/documentation/
        http://cis.cs.tu-berlin.de/Dokumente/Diplomarbeiten/2001/skinner.pdf
        http://citeseer.nj.nec.com/vilain00diagrammatic.html
        http://archive.devx.com/uml/articles/Smith01/Smith01-3.asp

1	joko	1.1	NAME
2	joko	1.5	Data::Storage - Interface for accessing various Storage implementations for Perl in an independent way
3	joko	1.1
4	joko	1.4	AIMS
5			- should encapsulate Tangram, DBI, DBD::CSV and LWP:: to access them in an unordinary (more convenient) way ;)
6			- introduce a generic layered structure, refactor SUBLAYER-stuff, make (e.g.) this possible:
7			Perl Data::Storage[DBD::CSV] -> Perl LWP:: -> Internet HTTP/FTP/* -> Host Daemon -> csv-file
8			- provide generic synchronization mechanisms across arbitrary/multiple storages based on ident/checksum
9			maybe it's possible to have schema-, structural- and semantical modifications synchronized???
10
11	joko	1.1	SYNOPSIS
12	joko	1.4	BASIC ACCESS
13
14			ADVANCED ACCESS
15	joko	1.1
16			... via inheritance:
17
18			use Data::Storage;
19			my $proxyObj = new HttpProxy;
20			$proxyObj->{url} = $url;
21			$proxyObj->{payload} = $content;
22			$self->{storage}->insert($proxyObj);
23
24			use Data::Storage;
25			my $proxyObj = HttpProxy->new(
26			url => $url,
27			payload => $content,
28			);
29			$self->{storage}->insert($proxyObj);
30
31	joko	1.4	SYNCHRONIZATION
32
33			my $nodemapping = {
34			'LangText' => 'langtexts.csv',
35			'Currency' => 'currencies.csv',
36			'Country' => 'countries.csv',
37			};
38
39			my $propmapping = {
40			'LangText' => [
41			[ 'source:lcountrykey' => 'target:country' ],
42			[ 'source:lkey' => 'target:key' ],
43			[ 'source:lvalue' => 'target:text' ],
44			],
45			'Currency' => [
46			[ 'source:ckey' => 'target:key' ],
47			[ 'source:cname' => 'target:text' ],
48			],
49			'Country' => [
50			[ 'source:ckey' => 'target:key' ],
51			[ 'source:cname' => 'target:text' ],
52			],
53			};
54
55			sub syncResource {
56
57			my $self = shift;
58			my $node_source = shift;
59			my $mode = shift;
60			my $opts = shift;
61
62			$mode \|\|= '';
63			$opts->{erase} \|\|= 0;
64
65			$logger->info( __PACKAGE__ . "->syncResource( node_source $node_source mode $mode erase $opts->{erase} )");
66
67			# resolve metadata for syncing requested resource
68			my $node_target = $nodemapping->{$node_source};
69			my $mapping = $propmapping->{$node_source};
70
71			if (!$node_target \|\| !$mapping) {
72			# loggger.... "no target, sorry!"
73			print "error while resolving resource metadata", "\n";
74			return;
75			}
76
77			if ($opts->{erase}) {
78			$self->_erase_all($node_source);
79			}
80
81			# create new sync object
82			my $sync = Data::Transfer::Sync->new(
83			storages => {
84			L => $self->{bizWorks}->{backend},
85			R => $self->{bizWorks}->{resources},
86			},
87			id_authorities => [qw( L ) ],
88			checksum_authorities => [qw( L ) ],
89			write_protected => [qw( R ) ],
90			verbose => 1,
91			);
92
93			# sync
94			# todo: filter!?
95			$sync->syncNodes( {
96			direction => $mode, # \| +PUSH \| +PULL \| -FULL \| +IMPORT \| -EXPORT
97			method => 'checksum', # \| -timestamp \| -manual
98			source => "L:$node_source",
99			source_ident => 'storage_method:id',
100			source_exclude => [qw( id cs )],
101			target => "R:$node_target",
102			target_ident => 'property:oid',
103			mapping => $mapping,
104			} );
105
106			}
107
108	joko	1.1	NOTE
109
110	joko	1.5	This module heavily relies on DBI and Tangram, but adds a lot of additional bugs and quirks.
111			Please look at their documentation and/or this code for additional information.
112	joko	1.3
113			REQUIREMENTS
114	joko	1.4	For full functionality:
115			DBI from CPAN
116			DBD::mysql from CPAN
117			Tangram 2.04 from CPAN (hmmm, 2.04 won't do in some cases)
118			Tangram 2.05 from http://... (2.05 seems okay but there are also additional patches from our side)
119			Class::Tangram from CPAN
120			DBD::CSV from CPAN
121			MySQL::Diff from http://adamspiers.org/computing/mysqldiff/
122			... and all their dependencies
123	joko	1.1
124			DESCRIPTION
125	joko	1.5	Data::Storage
126
127			Data::Storage is a module for accessing various "data structures / kinds of structured data" stored inside
128			various "data containers".
129			We tried to use the AdapterPattern (http://c2.com/cgi/wiki?AdapterPattern) to implement a wrapper-layer
130			around core CPAN modules (Tangram, DBI).
131
132			Why?
133
134			You will get a better code-structure (not bad for later maintenance) in growing Perl code projects,
135			especially when using multiple database connections at the same time.
136			You will be able to switch between different _kinds_ of implementations used for storing data.
137			Your code will use the very same API to access these storage layers.
138			... implementation has to be changed for now
139			Maybe you will be able to switch "on-the-fly" without changing any bits in code in the future....
140			... but that's not the focus
141
142			What else?
143
144			Having this, we were able to do implement a generic data synchronization module more easy,
145			please look at Data::Transfer.
146	joko	1.1
147			AUTHORS / COPYRIGHT
148	joko	1.5	The Data::Storage module is Copyright (c) 2002 Andreas Motl.
149			All rights reserved.
150			You may distribute it under the terms of either the GNU General Public
151			License or the Artistic License, as specified in the Perl README file.
152	joko	1.1
153			ACKNOWLEDGEMENTS
154	joko	1.5	Larry Wall for Perl, Tim Bunce for DBI, Jean-Louis Leroy for Tangram and Set::Object,
155			Sam Vilain for Class::Tangram, Jochen Wiedmann and Jeff Zucker for DBD::CSV & Co.,
156			Adam Spiers for MySQL::Diff and all contributors.
157	joko	1.1
158			SUPPORT / WARRANTY
159	joko	1.5	Data::Storage is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.
160	joko	1.1
161			TODO
162	joko	1.4	BUGS
163	joko	1.1
164	joko	1.4	"DBI-Error [Tangram]: DBD::mysql::st execute failed: Unknown column
165			't1.requestdump' in 'field list'"
166	joko	1.1
167	joko	1.4	... occours when operating on object-attributes not introduced yet:
168			this should be detected and appended/replaced through:
169			"Schema-Error detected, maybe (just) an inconsistency.
170			Please check if your declaration in schema-module "a" matches structure in database "b" or try to run"
171			db_setup.pl --dbkey=import --action=deploy
172	joko	1.1
173	joko	1.4	Compare schema (structure diff) with database ...
174	joko	1.1
175			... when issuing "db_setup.pl --dbkey=import --action=deploy"
176			on a database with an already deployed schema, use an additional "--update" then
177			to lift the schema inside the database to the current declared schema.
178			You will have to approve removals and changes on field-level while
179			new objects and new fields are introduced silently without any interaction needed.
180			In future versions there may be additional options to control silent processing of
181			removals and changes.
182			See this CRUD-table applying to the actions occouring on Classes and Class variables when deploying schemas,
183			don't mix this up with CRUD-actions on Objects, these are already handled by (e.g.) Tangram itself.
184			Classes:
185			C create -> yes, handled automatically
186			R retrieve -> no, not subject of this aspect since it is about deployment only
187			U update -> yes, automatically for Class meta-attributes, yes/no for Class variables (look at the rules down here)
188			D delete -> yes, just by user-interaction
189			Class variables:
190			C create -> yes, handled automatically
191			R retrieve -> no, not subject of this aspect since it is about deployment only
192			U update -> yes, just by user-interaction; maybe automatically if it can be determined that data wouldn't be lost
193			D delete -> yes, just by user-interaction
194	joko	1.2
195			It's all about not to be able to loose data simply while this is in pre-alpha stage.
196			And loosing data by being able to modify and redeploy schemas easily is definitely quite easy.
197
198			As we can see, creations of Classes and new Class variables is handled
199			automatically and this is believed to be the most common case under normal circumstances.
200	joko	1.1
201	joko	1.4	FEATURES
202	joko	1.1
203	joko	1.2	- Get this stuff together with UML (Unified Modeling Language) and/or standards from ODMG.
204			- Make it possible to load/save schemas in XMI (XML Metadata Interchange),
205			which seems to be most commonly used today, perhaps handle objects with OIFML.
206			Integrate/bundle this with a web-/html-based UML modeling tool or
207			some other interesting stuff like the "Co-operative UML Editor" from Uni Darmstadt. (web-/java-based)
208			- Enable Round Trip Engineering. Keep code and diagrams in sync. Don't annoy/bother the programmers.
209	joko	1.4	- Add support for some more handlers/locators to be able to
210			access the following standards/protocols/interfaces/programs/apis transparently:
211			+ DBD::CSV (via Data::Storage::Handler::DBI)
212			(-) Text::CSV, XML::CSV, XML::Excel
213			- MAPI
214			- LDAP
215			- DAV (look at PerlDAV: http://www.webdav.org/perldav/)
216			- Mbox (use formail for seperating/splitting entries/nodes)
217			- Cyrus (cyrdeliver - what about cyrretrieve (export)???)
218			- use File::DiffTree, use File::Compare
219			- Hibernate
220			- "Win32::UserAccountDb"
221			- "*nix::UserAccountDb"
222			- .wab - files (Windows Address Book)
223			- .pst - files (Outlook Post Storage?)
224			- XML (e.g. via XML::Simple?)
225			- Move to t3, look at InCASE
226	joko	1.5	- some kind of security layer for methods/objects
227			- acls (stored via tangram/ldap?) for functions, methods and objects (entity- & data!?)
228			- where are the hooks needed then?
229			- is Data::Storage & Co. okay, or do we have to touch the innards of DBI and/or Tangram?
230			- an attempt to start could be:
231			- 'sub getACLByObjectId($id, $context)'
232			- 'sub getACLByMethodname($id, $context)'
233			- 'sub getACLByName($id, $context)'
234			( would require a kinda registry to look up these very names pointing to arbitrary locations (code, data, ...) )
235	joko	1.1
236	joko	1.4	LINKS / REFERENCES
237	joko	1.1
238	joko	1.2	Specs:
239	joko	1.1	UML 1.3 Spec: http://cgi.omg.org/cgi-bin/doc?ad/99-06-08.pdf
240			XMI 1.1 Spec: http://cgi.omg.org/cgi-bin/doc?ad/99-10-02.pdf
241			XMI 2.0 Spec: http://cgi.omg.org/docs/ad/01-06-12.pdf
242			ODMG: http://odmg.org/
243			OIFML: http://odmg.org/library/readingroom/oifml.pdf
244
245	joko	1.2	CASE Tools:
246			Rational Rose (commercial): http://www.rational.com/products/rose/
247			Together (commercial): http://www.oi.com/products/controlcenter/index.jsp
248			InCASE - Tangram-based Universal Object Editor
249			Sybase PowerDesigner: http://www.sybase.com/powerdesigner
250
251			UML Editors:
252			Fujaba (free, university): http://www.fujaba.de/
253			ArgoUML (free): http://argouml.tigris.org/
254			Poseidon (commercial): http://www.gentleware.com/products/poseidonDE.php3
255			Co-operative UML Editor (research): http://www.darmstadt.gmd.de/concert/activities/internal/umledit.html
256			Metamill (commercial): http://www.metamill.com/
257			Violet (university, research, education): http://www.horstmann.com/violet/
258			PyUt (free): http://pyut.sourceforge.net/
259			(Dia (free): http://www.lysator.liu.se/~alla/dia/)
260			UMLet (free, university): http://www.swt.tuwien.ac.at/umlet/index.html
261			Voodoo (free): http://voodoo.sourceforge.net/
262	joko	1.3	Umbrello UML Modeller: http://uml.sourceforge.net/
263	joko	1.2
264			UML Tools:
265			http://www.objectsbydesign.com/tools/umltools_byPrice.html
266
267			Further readings:
268	joko	1.1	http://www.google.com/search?q=web+based+uml+editor&hl=en&lr=&ie=UTF-8&oe=UTF-8&start=10&sa=N
269			http://www.fernuni-hagen.de/DVT/Aktuelles/01FHHeidelberg.pdf
270			http://www.enhyper.com/src/documentation/
271			http://cis.cs.tu-berlin.de/Dokumente/Diplomarbeiten/2001/skinner.pdf
272			http://citeseer.nj.nec.com/vilain00diagrammatic.html
273			http://archive.devx.com/uml/articles/Smith01/Smith01-3.asp
274
MailToCvsAdmin">MailToCvsAdmin	ViewVC Help
Powered by ViewVC 1.1.26