From f42aee2f498d1a39daf3d0e634a7fae3626d86bc Mon Sep 17 00:00:00 2001 From: Boris Kolpackov Date: Wed, 9 Oct 2013 05:17:00 +0200 Subject: Document schema evolution support --- doc/manual.xhtml | 2924 +++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 2680 insertions(+), 244 deletions(-) (limited to 'doc') diff --git a/doc/manual.xhtml b/doc/manual.xhtml index c84f1bc..36863aa 100644 --- a/doc/manual.xhtml +++ b/doc/manual.xhtml @@ -192,6 +192,31 @@ for consistency. text-align: left; } + /* scenarios table */ + .scenarios { + margin: 2em 0 2em 0; + + border-collapse : collapse; + border : 1px solid; + border-color : #000000; + + font-size : 11px; + line-height : 14px; + } + + .scenarios th, .scenarios td { + border: 1px solid; + padding : 0.9em 0.9em 0.7em 0.9em; + } + + .scenarios th { + background : #cde8f6; + } + + .scenarios td { + text-align: left; + } + /* specifiers table */ .specifiers { margin: 2em 0 2em 0; @@ -303,7 +328,7 @@ for consistency. 2Hello World Example - + @@ -311,8 +336,9 @@ for consistency. - - + + +
2.1Declaring a Persistent Class
2.1Declaring Persistent Classes
2.2Generating Database Support Code
2.3Compiling and Running
2.4Making Objects Persistent
2.6Updating Persistent Objects
2.7Defining and Using Views
2.8Deleting Persistent Objects
2.9Accessing Multiple Databases
2.10Summary
2.9Changing Persistent Classes
2.10Accessing Multiple Databases
2.11Summary
@@ -466,6 +492,31 @@ for consistency. + 13Database Schema Evolution + + + + + + + + + +
13.1Object Model Version and Changelog
13.2Schema Migration
13.3Data Migration + + + +
13.3.1Immediate Data Migration
13.3.2Gradual Data Migration
+
13.4Soft Object Model Changes + + + +
13.4.1Reuse Inheritance Changes
13.4.2Polymorphism Inheritance Changes
+
+ + + + 14ODB Pragma Language @@ -484,6 +535,7 @@ for consistency. +
14.1.11definition
14.1.12transient
14.1.13sectionable
14.1.14deleted
@@ -551,18 +603,20 @@ for consistency. 14.4.19table 14.4.20load/update 14.4.21section - 14.4.22index_type - 14.4.23key_type - 14.4.24value_type - 14.4.25value_null/value_not_null - 14.4.26id_options - 14.4.27index_options - 14.4.28key_options - 14.4.29value_options - 14.4.30id_column - 14.4.31index_column - 14.4.32key_column - 14.4.33value_column + 14.4.22added + 14.4.23deleted + 14.4.24index_type + 14.4.25key_type + 14.4.26value_type + 14.4.27value_null/value_not_null + 14.4.28id_options + 14.4.29index_options + 14.4.30key_options + 14.4.31value_options + 14.4.32id_column + 14.4.33index_column + 14.4.34key_column + 14.4.35value_column @@ -576,21 +630,28 @@ for consistency. + + 14.6Object Model Pragmas + + +
14.6.1version
+ + - 14.6Index Definition Pragmas + 14.7Index Definition Pragmas - 14.7Database Type Mapping Pragmas + 14.8Database Type Mapping Pragmas - 14.8C++ Compiler Warnings + 14.9C++ Compiler Warnings - - - - - - + + + + + +
14.8.1GNU C++
14.8.2Visual C++
14.8.3Sun C++
14.8.4IBM XL C++
14.8.5HP aC++
14.8.6Clang
14.9.1GNU C++
14.9.2Visual C++
14.9.3Sun C++
14.9.4IBM XL C++
14.9.5HP aC++
14.9.6Clang
@@ -674,6 +735,7 @@ for consistency. 18.5.4Constraint Violations 18.5.5Sharing of Queries 18.5.6Forced Rollback + 18.5.7Database Schema Evolution @@ -738,6 +800,7 @@ for consistency. 20.5.6Timezones 20.5.7LONG Types 20.5.8LOB Types and By-Value Accessors/Modifiers + 20.5.9Database Schema Evolution @@ -1218,7 +1281,7 @@ for consistency. hello example which can be found in the odb-examples package of the ODB distribution.

-

2.1 Declaring a Persistent Class

+

2.1 Declaring Persistent Classes

In our "Hello World" example we will depart slightly from the norm and say hello to people instead of the world. People @@ -2130,7 +2193,138 @@ max age: 33 } -

2.9 Working with Multiple Databases

+

2.9 Changing Persistent Classes

+ +

When the definition of a transient C++ class is changed, for + example by adding or deleting a data member, we don't have to + worry about any existing instances of this class not matching + the new definition. After all, to make the class changes + effective we have to restart the application and none of the + transient instances will survive this.

+ +

Things are not as simple for persistent classes. Because they + are stored in the database and therefore survive application + restarts, we have a new problem: what happens to the state of + existing objects (which correspond to the old definition) once + we change our persistent class?

+ +

The problem of working with old object, called database + schema evolution, is a complex issue and ODB provides + comprehensive support for handling it. While this support + is covered in detail in Chapter 13, + "Database Schema Evolution", let us consider a simple + example that should give us a sense of the functionality + provided by ODB in this area.

+ +

Suppose that after using our person persistent + class for some time and creating a number of databases + containing its instances, we realized that for some people + we also need to store their middle name. If we go ahead and + just add the new data member, everything will work fine + with new databases. Existing databases, however, have a + table that does not correspond to the new class definition. + Specifically, the generated database support code now + expects there to be a column to store the middle name. + But such a column was never created in the old databases.

+ +

ODB can automatically generate SQL statements that will + migrate old databases to match the new class definitions. + But first, we need to enable schema evolution support by + defining a version for our object model:

+ +
+// person.hxx
+//
+
+#pragma db model version(1, 1)
+
+class person
+{
+  ...
+
+  std::string first_;
+  std::string last_;
+  unsigned short age_;
+};
+  
+ +

The first number in the version pragma is the + base model version. This is the lowest version we will be + able to migrate from. The second number is the current model + version. Since we haven't made any changes yet to our + persistent class, both of these values are 1.

+ +

Next we need to re-compile our person.hxx header + file with the ODB compiler, just as we did before:

+ +
+odb -d mysql --generate-query --generate-schema person.hxx
+  
+ +

If we now look at the list of files produced by the ODB compiler, + we will notice a new file: person.xml. This + is a changelog file where the ODB compiler keeps track of the + database changes corresponding to our class changes. Note that + this file is automatically maintained by the ODB compiler and + all that we have to do is to keep it around between + re-compilations.

+ +

Now we are ready to add the middle name to our person + class. We also give it a default value (empty string) which + is what will be assigned to existing objects in old databases. + Notice that we have also incremented the current version:

+ +
+// person.hxx
+//
+
+#pragma db model version(1, 2)
+
+class person
+{
+  ...
+
+  std::string first_;
+
+  #pragma db default("")
+  std::string middle_;
+
+  std::string last_;
+  unsigned short age_;
+};
+  
+ +

If we now recompile the person.hxx header again, we will + see two extra generated files: person-002-pre.sql + and person-002-post.sql. These two files contain + schema migration statements from version 1 to + version 2. Similar to schema creation, schema + migration statements can also be embedded into the generated + C++ code.

+ +

person-002-pre.sql and person-002-post.sql + are the pre and post schema migration files. To migrate + one of our old databases, we first execute the pre migration + file:

+ +
+mysql --user=odb_test --database=odb_test < person-002-pre.sql
+  
+ +

Between the pre and post schema migrations we can run data + migration code, if required. At this stage, we can both + access the old and store the new data. In our case we don't + need any data migration code since we assigned the default + value to the middle name for all the existing objects.

+ +

To finish the migration process we execute the post migration + statements:

+ +
+mysql --user=odb_test --database=odb_test < person-002-post.sql
+  
+ +

2.10 Working with Multiple Databases

Accessing multiple databases (that is, data stores) is simply a matter of creating multiple odb::<db>::database @@ -2141,12 +2335,13 @@ odb::mysql::database db1 ("john", "secret", "test_db1"); odb::mysql::database db2 ("john", "secret", "test_db2"); -

A more interesting question is how we access multiple database - systems (that is, database implementations) from the same application. - For example, our application may need to store some objects in a - remote MySQL database and others in a local SQLite file. Or, our - application may need to be able to store its objects in a database - system that is selected by the user at runtime.

+

Some database systems also allow attaching multiple databases to + the same instance. A more interesting question is how we access + multiple database systems (that is, database implementations) from + the same application. For example, our application may need to store + some objects in a remote MySQL database and others in a local SQLite + file. Or, our application may need to be able to store its objects + in a database system that is selected by the user at runtime.

ODB provides comprehensive multi-database support that ranges from tight integration with specific database systems to being able to @@ -2278,7 +2473,7 @@ psql --user=odb_test --dbname=odb_test -f person-pgsql.sql ./driver pgsql --user odb_test --database odb_test -

2.10 Summary

+

2.11 Summary

This chapter presented a very simple application which, nevertheless, exercised all of the core database functions: persist(), @@ -2911,6 +3106,12 @@ namespace odb drop_schema() functions should be called within a transaction.

+

ODB also provides support for database schema evolution. Similar + to schema creation, schema migration statements can be generated + either as standalone SQL files or embedded into the generated C++ + code. For more information on schema evolution support, refer to + Chapter 13, "Database Schema Evolution".

+

Finally, we can also use a custom database schema with ODB. This approach can work similarly to the standalone SQL file described above except that the database schema is hand-written or produced by another program. Or we @@ -4215,6 +4416,15 @@ namespace odb what () const throw (); }; + struct unknown_schema_version: exception + { + schema_version + version () const; + + virtual const char* + what () const throw (); + }; + // Section exceptions. // struct section_not_loaded: exception @@ -4320,7 +4530,11 @@ namespace odb

The unknown_schema exception is thrown by the odb::schema_catalog class if a schema with the specified name is not found. Refer to Section 3.4, "Database" - for details.

+ for details. The unknown_schema_version exception is + thrown by the schema_catalog functions that deal with + database schema evolution if the passed version is unknow. Refer + to Chapter 13, "Database Schema Evolution" for + details.

The section_not_loaded exception is thrown if we attempt to update an object section that hasn't been loaded. @@ -4509,13 +4723,13 @@ namespace odb is_null() - value is NULL + value is NULL query::age.is_null () is_not_null() - value is not NULL + value is NOT NULL query::age.is_not_null () @@ -5513,7 +5727,7 @@ private: column, they can occupy multiple columns. For an ordered container table the ODB compiler also defines two indexes: one for the object id column(s) and the other for the index - column. Refer to Section 14.6, "Index Definition + column. Refer to Section 14.7, "Index Definition Pragmas" for more information on how to customize these indexes.

@@ -5608,7 +5822,7 @@ private: id or element value are composite, then, instead of a single column, they can occupy multiple columns. ODB compiler also defines an index on a set container table for the object id - column(s). Refer to Section 14.6, "Index Definition + column(s). Refer to Section 14.7, "Index Definition Pragmas" for more information on how to customize this index.

@@ -5675,7 +5889,7 @@ private: element value are composite, then instead of a single column they can occupy multiple columns. ODB compiler also defines an index on a map container table for the object id - column(s). Refer to Section 14.6, "Index Definition + column(s). Refer to Section 14.7, "Index Definition Pragmas" for more information on how to customize this index.

@@ -6135,8 +6349,8 @@ class employee use the not_null pragma (Section 14.4.6, "null/not_null") for single object pointers and the value_not_null pragma - (Section - 14.4.25, "value_null/value_not_null") + (Section + 14.4.27, "value_null/value_not_null") for containers of object pointers. For example:

@@ -7681,9 +7895,9 @@ CREATE TABLE person (
   

The same principle applies when a composite value type is used as an element of a container, except that instead of db column, either the db value_column - (Section 14.4.33, "value_column") or + (Section 14.4.35, "value_column") or db key_column - (Section 14.4.32, "key_column") + (Section 14.4.34, "key_column") pragmas are used to specify the column prefix.

When a composite value type contains a container, an extra table @@ -11142,177 +11356,2227 @@ for (bool done (false); !done; )


-

14 ODB Pragma Language

- -

As we have already seen in previous chapters, ODB uses a pragma-based - language to capture database-specific information about C++ types. - This chapter describes the ODB pragma language in more detail. It - can be read together with other chapters in the manual to get a - sense of what kind of configurations and mapping fine-tuning are - possible. You can also use this chapter as a reference at a later - stage.

+

13 Database Schema Evolution

+ +

When we add new persistent classes or change the existing ones, for + example, by adding or deleting data members, the database schema + necessary to store the new object model changes as well. At the + same time, we may have existing databases that contain existing data. + If new versions of your application don't need to handle + old databases, then the schema creating functionality is all that + you need. However, most applications will need to work with data + stored by older versions of the same application.

+ +

We will call database schema evolution the overall task + of updating the database to match the changes in the object model. + Schema evolution usually consists of two sub-tasks: schema + migration and data migration. Schema migration + modifies the database schema to correspond to the current + object model. In a relational database, this, for example, could + require adding or dropping tables and columns. The data migration + task involves converting the data stored in the existing database + from the old format to the new one.

+ +

If performed manually, database schema evolution is a tedious and + error-prone task. As a result, ODB provides comprehensive support + for automated or, more precisely, semi-automated schema + evolution. Specifically, ODB does fully-automatic schema + migration and provides facilities to help you with data + migration.

+ +

The topic of schema evolution is a complex and sensitive + issue since normally there would be valuable, production data at + stake. As a result, the approach taken by ODB is to provide simple + and bullet-proof elementary building blocks (or migration steps) + that we can understand and trust. Using these elementary blocks we + can then implement more complex migration scenarios. In particular, + ODB does not try to handle data migration automatically since in most + cases this requires understanding of application-specific semantics. + In other words, there is no magic.

+ +

There are two general approaches to working with older data: the + application can either convert it to correspond to the new format + or it can be made capable of working with multiple versions of this + format. There is also a hybrid approach where the application + may convert the data to the new format gradually as part of its + normal functionality. ODB is capable of handling all these + scenarios. That is, there is support for working with older + models without performing any migration (schema or data). + Alternatively, we can migrate the schema after + which we have the choice of either also immediately migrating the + data (immediate data migration) or doing it gradually + (gradual data migration).

+ +

Schema evolution is already a complex task and we should not + unnecessarily use a more complex approach where a simpler one + would be sufficient. From the above, the simplest approach is + the immediate schema migration that does not require any data + migration. An example of such a change would be adding a new + data member with the default value (Section + 14.3.4, "default"). This case ODB can handle + completely automatically.

+ +

If we do require data migration, then the next simplest approach + is the immediate schema and data migration. Here we have to write + custom migration code. However, it is separate from the rest of + the core application logic and is executed at a well defined point + (database migration). In other words, the core application logic + need not be aware of older model versions. The potential drawback + of this approach is performance. It may take a lot of resources + and/or time to convert all the data upfront.

+ +

If the immediate migration is not possible, then the next option + is the immediate schema migration followed by the gradual data + migration. With this approach, both old and new data must co-exist + in the new database. We also have to change the application + logic to both account for different sources of the same data (for + example, when either old or new version of the object is loaded) + as well as migrate the data when appropriate (for example, when + the old version of the object is updated). At some point, usually + when the majority of the data has been converted, gradual migrations + is terminate with an immediate migration.

+ +

The most complex approach is working with multiple versions of + the database without performing any migrations, schema or data. + ODB does provide support for implementing this approach + (Section 13.4, "Soft Object Model Changes"), + however we will not cover it any further in this chapter. + Generally, this will require embedding knowledge about each + version into the core application logic which makes it hard + to maintain for any non-trivial object model.

+ +

Note also that when it comes to data migration, we can use + the immediate variant for some changes and gradual for others. + We will discuss various migration scenarios in greater detail + in section Section 13.3, "Data Migration".

+ +

13.1 Object Model Version and Changelog

+ +

To enable schema evolution support in ODB we need to specify + the object model version, or, more precisely, two versions. + The first is the base model version. It is the lowest model + version from which we will be able to migrate. The second + version is the current model version. In ODB we can migrate + from multiple previous versions by successively migrating + from one to the next until we reach the current version. + We use the db model version pragma + to specify both the base and current versions.

+ +

When we enable schema evolution for the first time, our + base and current versions will be the same, for example:

+ +
+#pragma db model version(1, 1)
+  
+ +

Once we release our application, its users may create databases + with the schema corresponding to this version of the object + model. This means that if we make any modifications to our + object model that also change the schema, then we will need + to be able to migrate the old databases to this new schema. + As a result, before making any new changes after a release, + we increment the current version, for example:

+ +
+#pragma db model version(1, 2)
+  
+ +

To put this another way, we can stay on the same version + during development and keep adding new changes to it. But + once we release it, any new changes to the object model will + have to be done in a new version.

+ +

It is easy to forget to increment the version before + making new changes to the object model. To help solve this + problem, the db model version pragma + accepts a third optional argument that specify whether the + current version is open or closed for changes. For example:

+ +
+#pragma db model version(1, 2, open)   // Can add new changes to
+                                       // version 2.
+  
+ +
+#pragma db model version(1, 2, closed) // Can no longer add new
+                                       // changes to version 2.
+  
+ +

If the current version is closed, ODB will refuse to accept + any new schema changes. In this situation you would + normally increment the current version and mark it as open + or you could re-open the existing version if, for example, + you need to fix something. Note, however, that re-opening + versions that have been released will most likely result + in migration malfunctions. By default the version is open.

+ +

Normally, an application will have a range of older database + versions from which it is able to migrate. When we change + this range by removing support for older versions, we also + need to adjust the base model version. This will make sure + that ODB does not keep unnecessary information around.

+ +

A model version (both base and current) is a 64-bit unsigned + integer (unsigned long long). 0 + is reserved to signify special situations, such as the lack of + schema in the database. Other than that, we can use any values + as versions as long as they are monotonically increasing. In + particular, we don't have to start with version 1 + and can increase the versions by any increment.

+ +

One versioning approach is to use an independent + object model version by starting from version 1 + and also incrementing by 1. The alternative + is to make the model version correspond to the application + version. For example, if our application is using the + X.Y.Z version format, then we could encode it + as a hexadecimal number and use that as our model version, + for example:

-

An ODB pragma has the following syntax:

+
+#pragma db model version(0x020000, 0x020306) // 2.0.0-2.3.6
+  
-

#pragma db qualifier [specifier specifier ...]

+

Most real-world object models will be spread over multiple + header files and it will be burdensome to repeat the + db model version pragma in each of + them. The recommended way to handle this situation is to + place the version pragma into a separate header + file and include it into the object model files. If your + project already has a header file that defines the + application version, then it is natural to place this + pragma there. For example:

-

The qualifier tells the ODB compiler what kind of C++ construct - this pragma describes. Valid qualifiers are object, - view, value, member, - namespace, index, and map. - A pragma with the object qualifier describes a persistent - object type. It tells the ODB compiler that the C++ class it describes - is a persistent class. Similarly, pragmas with the view - qualifier describe view types, the value qualifier - describes value types and the member qualifier is used - to describe data members of persistent object, view, and value types. - The namespace qualifier is used to describe common - properties of objects, views, and value types that belong to - a C++ namespace. The index qualifier defines a - database index. And, finally, the map qualifier - describes a mapping between additional database types and types - for which ODB provides built-in support.

+
+// version.hxx
+//
+// Define the application version.
+//
 
-  

The specifier informs the ODB compiler about a particular - database-related property of the C++ declaration. For example, the - id member specifier tells the ODB compiler that this - member contains this object's identifier. Below is the declaration - of the person class that shows how we can use ODB - pragmas:

+#define MYAPP_VERSION 0x020306 // 2.3.6 -
-#pragma db object
-class person
-{
-  ...
-private:
-  #pragma db member id
-  unsigned long id_;
-  ...
-};
+#ifdef ODB_COMPILER
+#pragma db model version(1, 7)
+#endif
   
-

In the above example we don't explicitly specify which C++ class or - data member the pragma belongs to. Rather, the pragma applies to - a C++ declaration that immediately follows the pragma. Such pragmas - are called positioned pragmas. In positioned pragmas that - apply to data members, the member qualifier can be - omitted for brevity, for example:

+

Note that we can also use macros in the version + pragma which allows us to specify all the versions in a single + place. For example:

-  #pragma db id
-  unsigned long id_;
+#define MYAPP_VERSION      0x020306 // 2.3.6
+#define MYAPP_BASE_VERSION 0x020000 // 2.0.0
+
+#ifdef ODB_COMPILER
+#pragma db model version(MYAPP_BASE_VERSION, MYAPP_VERSION)
+#endif
   
-

Note also that if the C++ declaration immediately following a - position pragma is incompatible with the pragma qualifier, an - error will be issued. For example:

+

It is also possible to have multiple object models within the + same application that have different versions. Such models + must be independent, that is, no headers from one model shall + include a header from another. You will also need to assign + different schema names to each model with the + --schema-name ODB compiler option.

+ +

Once we specify the object model version, the ODB compiler + starts tracking database schema changes in a changelog file. + Changelog has an XML-based, line-oriented format. It uses + XML in order to provide human readability while also + facilitating, if desired, processing and analysis with + custom tools. The line orientation makes it easy to review + with tools like diff.

+ +

The changelog is maintained by the ODB compiler. Specifically, + you do not need to make any manual changes to this file. You + will, however, need to keep it around from one invocation of + the ODB compiler to the next. In other words, the changelog + file is both the input to and the output of the ODB compiler. This, + for example, means that if your project's source code is stored + in a version control repository, then you will most likely want + to store the changelog there as well. If you delete the changelog, + then any ability to do schema migration will be lost.

+ +

The only operation that you may want to perform with the + changelog is to review the database schema changes that resulted + from the C++ object model changes. For this you can use a tool + like diff or, better yet, the change review facilities + offered by your revision control system. For this purpose the + contents of a changelog will be self-explanatory.

+ +

As an example, consider the following initial object model:

-  #pragma db object  // Error: expected class instead of data member.
-  unsigned long id_;
-  
+// person.hxx +// -

While keeping the C++ declarations and database declarations close - together eases maintenance and increases readability, we can also - place them in different parts of the same header file or even - factor them to a separate file. To achieve this we use the so called - named pragmas. Unlike positioned pragmas, named pragmas - explicitly specify the C++ declaration to which they apply by - adding the declaration name after the pragma qualifier. For example:

+#include <string> -
+#pragma db model version(1, 1)
+
+#pragma db object
 class person
 {
   ...
-private:
+
+  #pragma db id auto
   unsigned long id_;
-  ...
-};
 
-#pragma db object(person)
-#pragma db member(person::id_) id
+  std::string first_;
+  std::string last_;
+};
   
-

Note that in the named pragmas for data members the member - qualifier is no longer optional. The C++ declaration name in the - named pragmas is resolved using the standard C++ name resolution - rules, for example:

+

We then compile this header file with the ODB compiler (using the + PostgreSQL database as an example):

-
-namespace db
-{
-  class person
-  {
-    ...
-  private:
-    unsigned long id_;
-    ...
-  };
-}
+  
+odb --database pgsql --generate-schema person.hxx
+  
-namespace db -{ - #pragma db object(person) // Resolves db::person. -} +

If we now look at the list of generated files, then in addition to + the now familiar person-odb.?xx and person.sql, + we will also see person.xml — the changelog file. + Just for illustration, below are the contents of this changelog.

-#pragma db member(db::person::id_) id +
+<changelog database="pgsql">
+  <model version="1">
+    <table name="person" kind="object">
+      <column name="id" type="BIGINT" null="false"/>
+      <column name="first" type="TEXT" null="false"/>
+      <column name="last" type="TEXT" null="false"/>
+      <primary-key auto="true">
+        <column name="id"/>
+      </primary-key>
+    </table>
+  </model>
+</changelog>
   
-

As another example, the following code fragment shows how to use the - named value type pragma to map a C++ type to a native database type:

+

Let's say we now would like to add another data member to the + person class — the middle name. We increment + the version and make the change:

-#pragma db value(bool) type("INT")
+#pragma db model version(1, 2)
 
 #pragma db object
 class person
 {
   ...
-private:
-  bool married_; // Mapped to INT NOT NULL database type.
-  ...
+
+  #pragma db id auto
+  unsigned long id_;
+
+  std::string first_;
+  std::string middle_;
+  std::string last_;
 };
   
-

If we would like to factor the ODB pragmas into a separate file, - we can include this file into the original header file (the one - that defines the persistent types) using the #include - directive, for example:

+

We use exactly the same command line to re-compile our file:

-
-// person.hxx
+  
+odb --database pgsql --generate-schema person.hxx
+  
+ +

This time the ODB compiler will read in the old changelog, update + it, and write out the new version. Again, for illustration only, + below are the updated changelog contents:

+ +
+<changelog database="pgsql">
+  <changeset version="2">
+    <alter-table name="person">
+      <add-column name="middle" type="TEXT" null="false"/>
+    </alter-table>
+  </changeset>
+
+  <model version="1">
+    <table name="person" kind="object">
+      <column name="id" type="BIGINT" null="false"/>
+      <column name="first" type="TEXT" null="false"/>
+      <column name="last" type="TEXT" null="false"/>
+      <primary-key auto="true">
+        <column name="id"/>
+      </primary-key>
+    </table>
+  </model>
+</changelog>
+  
+ +

Just to reiterate, while the changelog may look like it could + be written by hand, it is maintained completely automatically + by the ODB compiler and the only reason you may want to look + at its contents is to review the database schema changes. For + example, if we compare the above to changelogs with + diff, we will get the following summary of the + database schema changes:

+ +
+--- person.xml.orig
++++ person.xml
+@@ -1,4 +1,10 @@
+<changelog database="pgsql">
++  <changeset version="2">
++    <alter-table name="person">
++      <add-column name="middle" type="TEXT" null="false"/>
++    </alter-table>
++  </changeset>
++
+  <model version="1">
+    <table name="person" kind="object">
+      <column name="id" type="BIGINT" null="false"/>
+  
+ +

The changelog is only written when we generate the database schema, + that is, the --generate-schema option is specified. + Invocations of the ODB compiler that only produce the database + support code (C++) do not read or update the changelog. To put it + another way, the changelog tracks changes in the resulting database + schema, not the C++ object model.

+ +

ODB ignores column order when comparing database schemas. This means + that we can re-order data members in a class without causing any + schema changes. Member renames, however, will result in schema + changes since the column name changes as well (unless we specified + the column name explicitly). From ODB's perspective such a rename + looks like the deletion of one data member and the addition of + another. If we don't want this to be treated as a schema change, + then we will need to keep the old column name by explicitly + specifying it with the db column pragma. For + example, here is how we can rename middle_ to + middle_name_ without causing any schema changes:

+ +
+#pragma db model version(1, 2)
 
+#pragma db object
 class person
 {
   ...
+
+  #pragma db column("middle") // Keep the original column name.
+  std::string middle_name_;
+
+  ...
 };
+  
-#ifdef ODB_COMPILER -# include "person-pragmas.hxx" -#endif +

If your object model consists of a large number of header files and + you generate the database schema for each of them individually, then + a changelog will be created for each of your header files. This may + be what you want, however, the large number of changelogs can quickly + become unwieldy. In fact, if you are generating the database schema + as standalone SQL files, then you may have already experienced a + similar problem caused by a large number of .sql files, + one for each header.

+ +

The solution to both of these problems is to generate a combined + database schema file and a single changelog. For example, assume + we have three header files in our object mode: + person.hxx, employee.hxx, and + employer.hxx. To generate the database support code + we compile them as usual but without specifying the + --generate-schema option. In this case no changelog + is created or updated:

+ +
+odb --database pgsql person.hxx
+odb --database pgsql employee.hxx
+odb --database pgsql employer.hxx
   
-

Alternatively, instead of using the #include directive, - we can use the --odb-epilogue option to make the pragmas - known to the ODB compiler when compiling the original header file, - for example:

+

To generate the database schema, we perform a separate invocation + of the ODB compiler. This time, however, we instruct it to only + generate the schema (--generate-schema-only) and + produce it combined (--at-once) for all the files + in our object model:

---odb-epilogue  '#include "person-pragmas.hxx"'
+odb --database pgsql --generate-schema-only --at-once \
+--input-name company person.hxx employee.hxx employer.hxx
   
-

The following sections cover the specifiers applicable to all the - qualifiers mentioned above.

+

The result of the above command is a single company.sql + file (the name is derived from the --input-name value) + that contains the database schema for our entire object model. There + is also a single corresponding changelog file — + company.xml.

-

The C++ header file that defines our persistent classes and - normally contains one or more ODB pragmas is compiled by both - the ODB compiler to generate the database support code and - the C++ compiler to build the application. Some C++ compilers - issue warnings about pragmas that they do not recognize. There - are several ways to deal with this problem which are covered - at the end of this chapter in Section 14.8, - "C++ Compiler Warnings".

+

The same can be achieved for the embedded schema by instructing + the ODB compiler to generate the database creation code into a + separate C++ file (--schema-format separate):

+ +
+odb --database pgsql --generate-schema-only --schema-format separate \
+--at-once --input-name company person.hxx employee.hxx employer.hxx
+  
+ +

The result of this command is a single company-schema.cxx + file and, again, company.xml.

+ +

Note also that by default the changelog file is not placed into + the directory specified with the --output-dir option. + This is due to the changelog being both an input and an output file + at the same time. As a result, by default, the ODB compiler will + place it in the directory of the input header file.

+ +

There is, however, a number of command line options (including + --changelog-dir) that allow us to fine-tune the name and + location of the changelog file. For example, you can instruct the ODB + compiler to read the changelog from one file while write it to + another. This, for example, can be useful if you want to review + the changes before discarding the old file. For more information + on these options, refer to the + ODB + Compiler Command Line Manual and search for "changelog".

+ +

When we were discussing version increments above, we used the + terms development and release. Specifically, + we talked about keeping the same object model versions during + development periods and incrementing them after releases. + What is a development period and a release in this context? + These definitions can vary from project to project. + Generally, during a development period we work on one or + more changes to the object model that result in the changes + to the database schema. A release is a point where we + make our changes available to someone else who may have an + older database to migrate from. In a tradition sense, a release + is a point where you make a new version of your application available + to its users. However, for schema evolution purposes, a release + could also mean simply making your schema-altering changes + available to other developers on your team. Let us consider + two common scenarios to illustrate how all this fits together.

+ +

One way to setup a project would be to re-use the application + development period and application release for schema evolution. + That is, during a new application version development we keep + a single object model version and when we release the application, + we increment the model version. In this case it makes sense to + also reuse the application version as a model version for + consistency. Here is a step-by-step guide for this setup:

+ +
    +
  1. During development, keep the current object model version open.
  2. + +
  3. Before the release (for example, when entering a "feature freeze") + close the version.
  4. + +
  5. After the release, update the version and open it.
  6. + +
  7. For each new feature, review the changeset at the top of the + changelog, for example, with diff or your + version control facilities. If you are using a version + control, then this is best done just before committing + your changes to the repository.
  8. +
+ +

An alternative way to setup schema versioning in a project would + be to define the development period as working on a single + feature and the release as making this feature available to + other people (developers, testers, etc.) on your team, for + example, by committing the changes to a public version control + repository. In this case, the object model version will be + independent of the application version and can simply be + a sequence that starts with 1 and is + incremented by 1. Here is a step-by-step guide + for this setup:

+ +
    +
  1. Keep the current model version closed. Once a change is made + that affects the database schema, the ODB compiler will refuse + to update the changelog.
  2. + +
  3. If the change is legitimate, open a new version, that is, + increment the current version and make it open.
  4. + +
  5. Once the feature is implemented and tested, review the final + set of database changes (with diff or your + version control facilities), close the version, and commit + the changes to the version control repository (if using).
  6. +
+ +

If you are using a version control repository that supports + pre-commit checks, then you may want to consider adding such + a check to make sure the committed version is always closed.

+ +

If we are just starting schema evolution in our project, which + approach should we choose? The two approaches will work better + in different situations since they have a different set of + advantages and disadvantages. The first approach, which we + can call version per application release, is best suited + for simpler projects with smaller releases since otherwise + a single migration will bundle a large number of unrelated + actions corresponding to different features. This can + become difficult to review and, if things go wrong, debug.

+ +

The second approach, which we can call version per feature, + is much more modular and provides a number additional benefits. + We can perform migrations for each feature as a discreet step + which makes it easier to debug. We can also place each such + migration step into a separate transaction further improving + reliability. It also scales much better in larger teams + where multiple developers can work concurrently on features + that affect the database schema. For example, if you find + yourself in a situation where another developer on your + team used the same version as you and managed to commit his + changes before you (that is, you have a merge conflict), + then you can simply change the version to the next available, + regenerate the changelog, and continue with your commit.

+ +

Overall, unless you have strong reasons to prefer the version + per application release approach, choose version per + feature even though it may seem more complex at the + beginning. Also, if you do select the first approach, consider + provisioning for switching to the second method by reserving + a sub-version number. For example, for an application version + in the form 2.3.4 your can make the object model + version to be in the form 0x0203040000, reserving + the last two bytes for a sub-version. Later on you can use it to + switch to the version per feature approach.

+ +

13.2 Schema Migration

+ +

Once we enable schema evolution by specifying the object model + version, in addition to the schema creation statements, the + ODB compiler starts generating schema migration statements + for each version all the way from the base to the current. + As with schema creation, schema migration can be generated + either as a set of SQL files or embedded into the generated + C++ code (--schema-format option).

+ +

For each migration step, that is from one version to the next, + ODB generates two sets of statements: pre-migration and + post-migration. The pre-migration statements "relax" + the database schema so that both old and new data can co-exist. + At this stage new columns and tables are added while old + constraints are dropped. The post-migration statements + "tighten" the database schema back so that only + data conforming to the new format can remain. At this stage + old columns and tables are dropped and new constraints are + added. Now you can probably guess where the data + migration fits into this — between the pre and post + schema migrations where we can both access the old data + and create the new one.

+ +

If the schema is being generated as standalone SQL files, + then we end up with a pair of files for each step: the pre-migration + file and the post-migration file. For the person + example we started in the previous section we will have the + person-002-pre.sql and person-002-post.sql + files. Here 002 is the version to which + we are migrating while the pre and post + suffixes specify the migration stage. So if we wanted to migrate + a person database from version 1 + to 2, then we would first execute + person-002-pre.sql, then migrate the data, if any + (discussed in more detail in the next section), and finally + execute person-002-post.sql. If our database is + several versions behind, for example the database has version + 1 while the current version is 5, + then we simply perform this set of steps for each version + until we reach the current version.

+ +

If we look at the contents of the person-002-pre.sql + file, we will see the following (or equivalent, depending on the + database used) statement:

+ +
+ALTER TABLE "person"
+  ADD COLUMN "middle" TEXT NULL;
+  
+ +

As we would expect, this statement adds a new column corresponding + to the new data member. An observant reader would notice, + however, that the column is added as NULL + even though we never requested this semantics in our object model. + Why is the column added as NULL? If during migration + the person table already contains rows (that is, existing + objects), then an attempt to add a non-NULL column that + doesn't have a default value will fail. As a result, ODB will initially + add a new column that doesn't have a default value as NULL + but then clean this up at the post-migration stage. This way your data + migration code is given a chance to assign some meaningful values for + the new data member for all the existing objects. Here are the contents + of the person-002-post.sql file:

+ +
+ALTER TABLE "person"
+  ALTER COLUMN "middle" SET NOT NULL;
+  
+ +

Currently ODB directly supports the following elementary database + schema changes:

+ +
    +
  • add table
  • +
  • drop table
  • +
  • add column
  • +
  • drop column
  • +
  • alter column, set NULL/NOT NULL
  • +
  • add foreign key
  • +
  • drop foreign key
  • +
  • add index
  • +
  • drop index
  • +
+ +

More complex changes can normally be implemented in terms of + these building blocks. For example, to change a type of a + data member (which leads to a change of a column type), we + can add a new data member with the desired type (add column), + migrate the data, and then delete the old data member (drop + column). ODB will issue diagnostics for cases that are + currently not supported directly. Note also that some database + systems (notably SQLite) have a number of limitations in their + support for schema changes. For more information on these + database-specific limitations, refer to the "Limitations" sections + in Part II, "Database Systems".

+ +

How do we know what is the current database version is? That is, the + version from which we need to migrate? We need to know this, + for example, in order to determine the set of migrations we have to + perform. By default, when schema evolution is enabled, ODB maintains + this information in a special table called schema_version + that has the following (or equivalent, depending on the database + used) definition:

+ +
+CREATE TABLE "schema_version" (
+  "name" TEXT NOT NULL PRIMARY KEY,
+  "version" BIGINT NOT NULL,
+  "migration" BOOLEAN NOT NULL);
+  
+ +

The name column is the schema name as specified with + the --schema-name option. It is empty for the default + schema. The version column contains the current database + version. And, finally, the migration flag indicates + whether we are in the process of migrating the database, that is, + between the pre and post-migration stages.

+ +

The schema creation statements (person.sql in our case) + create this table and populate it with the initial model version. For + example, if we executed person.sql corresponding to + version 1 of our object model, then name + would have been empty (which signifies the default schema since we + didn't specify --schema-name), version will + be 1 and migration will be + FALSE.

+ +

The pre-migration statements update the version and set the migration + flag to TRUE. Continuing with our example, after executing + person-002-pre.sql, version will + become 2 and migration will be set to + TRUE. The post-migration statements simply clear the + migration flag. In our case, after running + person-002-post.sql, version will + remain 2 while migration will be reset + to FALSE.

+ +

Note also that above we mentioned that the schema creation statements + (person.sql) create the schema_version table. + This means that if we enable schema evolution support in the middle + of a project, then we could already have existing databases that + don't include this table. As a result, ODB will not be able to handle + migrations for such databases unless we manually add the + schema_version table and populate it with correct + version information. For this reason, it is highly recommended that + you consider whether to use schema evolution and enable it if so + from the beginning of your project.

+ +

The odb::database class provides an API for accessing + and modifying the current database version:

+ +
+namespace odb
+{
+  typedef unsigned long long schema_version;
+
+  struct LIBODB_EXPORT schema_version_migration
+  {
+    schema_version_migration (schema_version = 0,
+                              bool migration = false);
+
+    schema_version version;
+    bool migration;
+
+    // This class also provides the ==, !=, <, >, <=, and >= operators.
+    // Version ordering is as follows: {1,f} < {2,t} < {2,f} < {3,t}.
+  };
+
+  class database
+  {
+  public:
+    ...
+
+    schema_version
+    schema_version (const std::string& name = "") const;
+
+    bool
+    schema_migration (const std::string& name = "") const;
+
+    const schema_version_migration&
+    schema_version_migration (const std::string& name = "") const;
+
+    // Set schema version and migration state manually.
+    //
+    void
+    schema_version_migration (schema_version,
+                              bool migration,
+                              const std::string& name = "");
+
+    void
+    schema_version_migration (const schema_version_migration&,
+                              const std::string& name = "");
+
+    // Set default schema version table for all schemas.
+    //
+    void
+    schema_version_table (const std::string& table_name);
+
+    // Set schema version table for a specific schema.
+    //
+    void
+    schema_version_table (const std::string& table_name,
+                          const std::string& name);
+  };
+}
+  
+ +

The schema_version() and schema_migration() + accessors return the current database version and migration flag, + respectively. The optional name argument is the schema + name. If the database schema hasn't been created (that is, there is + no corresponding entry in the schema_version table or + this table does not exist), then schema_version() returns + 0. The schema_version_migration() accessor + returns both version and migration flag together in the + schema_version_migration struct.

+ +

You may already have a version table in your database or you (or your + database administrator) may prefer to keep track of versions your own + way. You can instruct ODB not to create the schema_version + table with the --suppress-schema-version option. However, + ODB still needs to know the current database version in order for certain + schema evolution mechanisms to function properly. As a result, in + this case, you will need to set the schema version on the database + instance manually using the schema_version_migration() modifier. + Note that the modifier API is not thread-safe. That is, you should + not modify the schema version while other threads may be accessing + or modifying the same information.

+ +

Note also that the accessors we discussed above will only query the + schema_version table once and, if the version could + be determined, cache the result. If, however, the version could + not be determined (that is, schema_version() returned + 0), then a subsequent call will re-query the table. While it is + probably a bad idea to modify the database schema while the + application is running (other than via the schema_catalog + API, as discussed below), if for some reason you need ODB to re-query + the version, then you can manually set it to 0 using the + schema_version_migration() modifier.

+ +

It is also possible to change the name of the table that stores + the schema version using the --schema-version-table + option. You will also need to specify this alternative name on + the database instance using the schema_version_table() + modifier. The first version specifies the default table that is + used for all the schema names. The second version specifies the + table for a specific schema. The table name should be + database-quoted, if necessary.

+ +

If we are generating our schema migrations as standalone SQL files, + then the migration workflow could look like this:

+ +
    +
  1. Database administrator determines the current database version. + If migration is required, then for each migration step (that + is, from one version to the next), he performs the following:
  2. + +
  3. Execute the pre-migration file.
  4. + +
  5. Execute our application (or a separate migration program) + to perform data migration (discussed later). Our application + can determine that is is executed in the "migration mode" + by calling schema_migration() and then which + migration code to run by calling schema_version().
  6. + +
  7. Execute the post-migration file.
  8. +
+ +

These steps become more integrated and automatic if we embed the + schema creation and migration code into the generated C++ code. + Now we can perform schema creation, schema migration, and data + migration as well as determine when each step is necessary + programmatically from within the application.

+ +

Schema evolution support adds the following extra functions to + the odb::schema_catalog class, which we first discussed + in Section 3.4, "Database".

+ +
+namespace odb
+{
+  class schema_catalog
+  {
+  public:
+    ...
+
+
+    // Schema migration.
+    //
+    static void
+    migrate_schema_pre (database&,
+                        schema_version,
+                        const std::string& name = "");
+
+    static void
+    migrate_schema_post (database&,
+                         schema_version,
+                         const std::string& name = "");
+
+    static void
+    migrate_schema (database&,
+                    schema_version,
+                    const std::string& name = "");
+
+    // Data migration.
+    //
+    // Discussed in the next section.
+
+
+    // Combined schema and data migration.
+    //
+    static void
+    migrate (database&,
+             schema_version = 0,
+             const std::string& name = "");
+
+    // Schema version information.
+    //
+    static schema_version
+    base_version (const database&,
+                  const std::string& name = "");
+
+    static schema_version
+    base_version (database_id,
+                  const std::string& name = "");
+
+    static schema_version
+    current_version (const database&,
+                     const std::string& name = "");
+
+    static schema_version
+    current_version (database_id,
+                     const std::string& name = "");
+
+    static schema_version
+    next_version (const database&,
+                  schema_version = 0,
+                  const std::string& name = "");
+
+    static schema_version
+    next_version (database_id,
+                  schema_version,
+                  const std::string& name = "");
+  };
+}
+  
+ +

The migrate_schema_pre() and + migrate_schema_post() static functions perform + a single stage (that is, pre or post) of a single migration + step (that is, from one version to the next). The version + argument specifies the version we are migrating to. For + instance, in our person example, if we know that + the database version is 1 and the next version + is 2, then we can execute code like this:

+ +
+transaction t (db.begin ());
+
+schema_catalog::migrate_schema_pre (db, 2);
+
+// Data migration goes here.
+
+schema_catalog::migrate_schema_post (db, 2);
+
+t.commit ();
+  
+ +

If you don't have any data migration code to run, then you can + perform both stages with a single call using the + migrate_schema() static function.

+ +

The migrate() static function perform both schema + and data migration (we discuss data migration in the next section). + It can also perform several migration steps at once. If we don't + specify its target version, then it will migrate (if necessary) + all the way to the current model version. As an extra convenience, + migrate() will also create the database schema if + none exists. As a result, if we don't have any data migration + code or we have registered it with schema_catalog (as + discussed later), then the database schema creation and migration, + whichever is necessary, if at all, can be performed with a single + function call:

+ +
+transaction t (db.begin ());
+schema_catalog::migrate (db);
+t.commit ();
+  
+ +

Note also that schema_catalog is integrated with the + odb::database schema version API. In particular, + schema_catalog functions will query and synchronize + the schema version on the database instance if and + when required.

+ +

The schema_catalog class also allows you to iterate + over known versions (remember, there could be "gaps" in version + numbers) with the base_version(), + current_version() and next_version() + static functions. The base_version() and + current_version() functions return the base and + current object model versions, respectively. That is, the + lowest version from which we can migrate and the version that + we ultimately want to migrate to. The next_version() + function returns the next known version. If the passed version is + greater or equal to the current version, then this function + will return the current version plus one (that is, one past + current). If we don't specify the version, then + next_version() will use the current database version + as the starting point. Note also that the schema version information + provided by these functions is only available if we embed the schema + migration code into the generated C++ code. For standalone SQL file + migrations this information is normally not needed since the migration + process is directed by an external entity, such as a database + administrator or a script.

+ +

Most schema_catalog functions presented above also + accept the optional schema name argument. If the passed schema + name is not found, then the odb::unknown_schema exception + is thrown. Similarly, functions that accept the schema version + argument will throw the odb::unknown_schema_version exception + if the passed version is invalid. Refer to Section + 3.14, "ODB Exceptions" for more information on these exceptions.

+ +

To illustrate how all these parts fit together, consider the + following more realistic database schema management example. + Here we want to handle the schema creation in a special way + and perform each migration step in its own transaction.

+ +
+schema_version v (db.schema_version ());
+schema_version bv (schema_catalog::base_version (db));
+schema_version cv (schema_catalog::current_version (db));
+
+if (v == 0)
+{
+  // No schema in the database. Create the schema and
+  // initialize the database.
+  //
+  transaction t (db.begin ());
+  schema_catalog::create_schema (db);
+
+  // Populate the database with initial data, if any.
+
+  t.commit ();
+}
+else if (v < cv)
+{
+  // Old schema (and data) in the database, migrate them.
+  //
+
+  if (v < bv)
+  {
+    // Error: migration from this version is no longer supported.
+  }
+
+  for (v = schema_catalog::next_version (db, v);
+       v <= cv;
+       v = schema_catalog::next_version (db, v))
+  {
+    transaction t (db.begin ());
+    schema_catalog::migrate_schema_pre (db, v);
+
+    // Data migration goes here.
+
+    schema_catalog::migrate_schema_post (db, v);
+    t.commit ();
+  }
+}
+else if (v > cv)
+{
+  // Error: old application trying to access new database.
+}
+  
+ +

13.3 Data Migration

+ +

In quite a few cases specifying the default value for new data + members will be all that's required to handle the existing objects. + For example, the natural default value for the new middle name + that we have added is an empty string. And we can handle + this case with the db default pragma and without + any extra C++ code:

+ +
+#pragma db model version(1, 2)
+
+#pragma db object
+class person
+{
+  ...
+
+
+  #pragma db default("")
+  std::string middle_;
+};
+  
+ +

However, there will be situations where we would need to perform + more elaborate data migrations, that is, convert old data to the + new format. As an example, suppose we want to add gender to our + person class. And, instead of leaving it unassigned + for all the existing objects, we will try to guess it from the + first name. Not particularly accurate but could be sufficient + for our hypothetical application:

+ +
+#pragma db model version(1, 3)
+
+enum gender {male, female};
+
+#pragma db object
+class person
+{
+  ...
+
+  gender gender_;
+};
+  
+ +

As we have discussed earlier, there are two ways to perform data + migration: immediate and gradual. To recap, with immediate + migration we migrate all the existing objects at once, normally + after the schema pre-migration statements but before the + post-migration statements. With gradual migration, we make sure + the new object model can accommodate both old and new data and + gradually migrate existing objects as the application runs and + the opportunities to do so arise, for example, an object is + updated.

+ +

There is also another option for data migration that is not + discussed further in this section. Instead of using our C++ + object model we could execute ad-hoc SQL statements that + perform the necessary conversions and migrations directly + on the database server. While in certain cases this can be + a better option from the performance point of view, this + approach is often limited in terms of the migration logic + that we can handle.

+ +

13.3.1 Immediate Data Migration

+ +

Let's first see how we can implement an immediate migration for the + new gender_ data member we have added above. If we + are using standalone SQL files for migration, then we could add + code along these lines somewhere early in main(), + before the main application logic:

+ +
+int
+main ()
+{
+  ...
+
+  odb::database& db = ...
+
+  // Migrate data if necessary.
+  //
+  if (db.schema_migration ())
+  {
+    switch (db.schema_version ())
+    {
+    case 3:
+      {
+        // Assign gender to all the existing objects.
+        //
+        transaction t (db.begin ());
+
+        for (person& p: db.query<person> ())
+        {
+          p.gender (guess_gender (p.first ()));
+          db.update (p);
+        }
+
+        t.commit ();
+        break;
+      }
+    }
+  }
+
+  ...
+}
+  
+ +

If you have a large number of objects to migrate, it may also be + a good idea, from the performance point of view, to break one big + transaction that we have now into multiple smaller transactions + (Section 3.5, "Transactions"). For example:

+ +
+case 3:
+  {
+    transaction t (db.begin ());
+
+    size_t n (0);
+    for (person& p: db.query<person> ())
+    {
+      p.gender (guess_gender (p.first ()));
+      db.update (p);
+
+      // Commit the current transaction and start a new one after
+      // every 100 updates.
+      //
+      if (n++ % 100 == 0)
+      {
+        t.commit ();
+        t.reset (db.begin ());
+      }
+    }
+
+    t.commit ();
+    break;
+  }
+  
+ +

While it looks straightforward enough, as we add more migration + snippets, this approach can quickly become unmaintainable. Instead + of having all the migrations in a single function and determining + when to run each piece ourselves, we can package each migration into + a separate function, register it with the schema_catalog + class, and let ODB figure out when to run which migration functions. + To support this functionality, schema_catalog provides + the following data migration API:

+ +
+namespace odb
+{
+  class schema_catalog
+  {
+  public:
+    ...
+
+    // Data migration.
+    //
+    static std::size_t
+    migrate_data (database&,
+                  schema_version = 0,
+                  const std::string& name = "");
+
+    // C++98/03 version:
+    //
+    typedef void (*data_migration_function_type) (database&);
+
+    // C++11 version:
+    //
+    typedef std::function<void (database&)> data_migration_function_type;
+
+    // Common (for all the databases) data migration.
+    //
+    template <schema_version v, schema_version base>
+    static void
+    data_migration_function (data_migration_function_type,
+                             const std::string& name = "");
+
+    // Database-specific data migration.
+    //
+    template <schema_version v, schema_version base>
+    static void
+    data_migration_function (database&,
+                             data_migration_function_type,
+                             const std::string& name = "");
+
+    template <schema_version v, schema_version base>
+    static void
+    data_migration_function (database_id,
+                             data_migration_function_type,
+                             const std::string& name = "");
+  };
+
+  // Static data migration function registration.
+  //
+  template <schema_version v, schema_version base>
+  struct data_migration_entry
+  {
+    data_migration_entry (data_migration_function_type,
+                          const std::string& name = "");
+
+    data_migration_entry (database_id,
+                          data_migration_function_type,
+                          const std::string& name = "");
+  };
+}
+  
+ +

The migrate_data() static function performs data + migration for the specified version. If no version is specified, + then it will use the current database version and also check + whether the database is in migration, that is, + database::schema_migration() returns true. + As a result, all we need to do in our main() is call + this function. It will check if migration is required and if so, + call all the migration functions registered for this version. For + example:

+ +
+int
+main ()
+{
+  ...
+
+  database& db = ...
+
+  // Check if we need to migrate any data and do so
+  // if that's the case.
+  //
+  schema_catalog::migrate_data (db);
+
+  ...
+}
+  
+ +

The migrate_data() function returns the number of + migration functions called. You can use this value for debugging + or logging.

+ +

The only other step that we need to perform is register our data + migration functions with schema_catalog. At the + lower level we can call the data_migration_function() + static function for every migration function we have, for example, + at the beginning of main(). Data migration functions + are called in the order of registration.

+ +

A more convenient approach, however, is to use the + data_migration_entry helper class template to register the + migration functions during static initialization. This way we + can keep the migration function and its registration code next + to each other. Here is how we can reimplement our gender + migration code to use this mechanism:

+ +
+static void
+migrate_gender (odb::database& db)
+{
+  transaction t (db.begin ());
+
+  for (person& p: db.query<person> ())
+  {
+    p.gender (guess_gender (p.first ()));
+    db.update (p);
+  }
+
+  t.commit ();
+}
+
+static const odb::data_migration_entry<3, MYAPP_BASE_VERSION>
+migrate_gender_entry (&migrate_gender);
+  
+ +

The first template argument to the data_migration_entry + class template is the version we want this data migration function + to be called for. The second template argument is the base model + version. This second argument is necessary to detect the situation + where we no longer need this data migration function. Remember + that when we move the base model version forward, migrations from + any version below the new base are no longer possible. We, however, + may still have migration functions registered for those lower + versions. Since these functions will never be called, they are + effectively dead code and it would be useful to identify and + remove them. To assist with this, data_migration_entry + (and lower lever data_migration_function()) will + check at compile time (that is, static_assert) that + the registration version is greater than the base model version.

+ +

In the above example we use the MYAPP_BASE_VERSION + macro that is presumably defined in a central place, for example, + version.hxx. This is the recommended approach since + we can update the base version in a single place and have the + C++ compiler automatically identify all the data migration + functions that can be removed.

+ +

In C++11 we can also create a template alias so that we don't + have to repeat the base model macro in every registration, for + example:

+ +
+template <schema_version v>
+using migration_entry = odb::data_migration_entry<v, MYAPP_BASE_VERSION>;
+
+static const migration_entry<3>
+migrate_gender_entry (&migrate_gender);
+  
+ +

For cases where you need to by-pass the base version check, for + example, to implement your own registration helper, ODB also + provides "unsafe" versions of the data_migration_function() + functions that take the version as a function argument rather than + as a template parameter.

+ +

In C++11 we can also use lambdas as migration functions, which makes + the migration code more concise:

+ +
+static const migration_entry<3>
+migrate_gender_entry (
+  [] (odb::database& db)
+  {
+    transaction t (db.begin ());
+
+    for (person& p: db.query<person> ())
+    {
+      p.gender (guess_gender (p.first ()));
+      db.update (p);
+    }
+
+    t.commit ();
+  });
+  
+ +

If we are using embedded schema migrations, then both schema and + data migration is integrated and can be performed with a single + call to the schema_catalog::migrate() function that + we discussed earlier. For example:

+ +
+int
+main ()
+{
+  ...
+
+  database& db = ...
+
+  // Check if we need to migrate the database and do so
+  // if that's the case.
+  //
+  {
+    transaction t (db.begin ());
+    schema_catalog::migrate (db);
+    t.end ();
+  }
+
+  ...
+}
+  
+ +

Note, however, that in this case we call migrate() + within a transaction (for the schema migration part) which means + that our migration functions will also be called within this + transaction. As a result, we will need to adjust our migration + functions not to start their own transaction:

+ +
+static void
+migrate_gender (odb::database& db)
+{
+  // Assume we are already in a transaction.
+  //
+  for (person& p: db.query<person> ())
+  {
+    p.gender (guess_gender (p.first ()));
+    db.update (p);
+  }
+}
+  
+ +

If, however, we want more granular transactions, then we can + use the lower-level schema_catalog functions to + gain more control, as we have seen at the end of previous + section. Here is the relevant part of that example with + an added data migration call:

+ +
+  // Old schema (and data) in the database, migrate them.
+  //
+  for (v = schema_catalog::next_version (db, v);
+       v <= cv;
+       v = schema_catalog::next_version (db, v))
+  {
+    transaction t (db.begin ());
+    schema_catalog::migrate_schema_pre (db, v);
+    schema_catalog::migrate_data (db, v);
+    schema_catalog::migrate_schema_post (db, v);
+    t.commit ();
+  }
+  
+ +

13.3.2 Gradual Data Migration

+ +

If the number of existing objects that require migration is large, + then an all-at-once, immediate migration, while simple, may not + be practical from the performance point of view. In this case, + we can perform a gradual migration as the application does + its normal functions.

+ +

With gradual migrations, the object model must be capable of + representing data that conforms to both old and new formats at + the same time since, in general, the database will contain a + mixture of old and new objects. For example, in case of our + gender data member, we need a special value that + represents the "no gender assigned yet" case (an old object). + We also need to assign this special value to all the existing + objects during the schema pre-migration stage. One way to do + this would be add a special value to our gender + enum and then make it the default value with the + db default pragma. A cleaner and easier approach, + however, is to use NULL as a special value. We + can add support for the NULL value semantics + to any existing type by wrapping it with + odb::nullable, boost::optional + or similar (Section 7.3, "Pointers and NULL + Value Semantics"). We also don't need to specify the default value + explicitly since NULL is used automatically. Here + is how we can use this approach in our gender + example:

+ +
+#include <odb/nullable.hxx>
+
+#pragma db object
+class person
+{
+  ...
+
+  odb::nullable<gender> gender_;
+};
+  
+ +

A variety of strategies can be employed to implement gradual + migrations. For example, we can migrate the data when the object + is updated as part of the normal application logic. While there + is no migration cost associated with this approach (the object + is updated anyway), depending on how often objects are typically + updated, this strategy can take a long time to complete. An + alternative strategy would be to perform an update whenever + an old object is loaded. Yet another strategy is to have a + separate thread that slowly migrates all the old objects as + the application runs.

+ +

As an example, let us implement the first approach for our + gender migration. While we could have added + the necessary code throughout the application, from the + maintenance point of view, it is best to try and localize + the gradual migration logic to the persistent classes that + it affects. And for this database operation callbacks + (Section 14.1.7, "callback") + are a very useful mechanism. In our case, all we have to do is handle + the post_load event where we guess the gender + if it is NULL:

+ +
+#include <odb/core.hxx>     // odb::database
+#include <odb/callback.hxx> // odb::callback_event
+#include <odb/nullable.hxx>
+
+#pragma db object callback(migrate)
+class person
+{
+  ...
+
+  void
+  migrate (odb::callback_event e, odb::database&)
+  {
+    if (e == odb::callback_event::post_load)
+    {
+      // Guess gender if not assigned.
+      //
+      if (gender_.null ())
+        gender_ = guess_gender (first_);
+    }
+  }
+
+  odb::nullable<gender> gender_;
+};
+  
+ +

In particular, we don't have to touch any of the accessors + or modifiers or the application logic — all of them + can assume that the value can never be NULL. + And when the object is next updated, the new gender + value will be stored automatically.

+ +

All gradual migrations normally end up with a terminating + immediate migration some number of versions down the line, + when the bulk of the objects has presumably been converted. + This way we don't have to keep the gradual migration code + around forever. Here is how we could implement a terminating + migration for our example:

+ +
+// person.hxx
+//
+#pragma db model version(1, 4)
+
+#pragma db object
+class person
+{
+  ...
+
+  gender gender_;
+};
+
+// person.cxx
+//
+static void
+migrate_gender (odb::database& db)
+{
+  typedef odb::query<person> query;
+
+  for (person& p: db.query<person> (query::gender.is_null ()))
+  {
+    p.gender (guess_gender (p.first ()));
+    db.update (p);
+  }
+}
+
+static const odb::data_migration_entry<4, MYAPP_BASE_VERSION>
+migrate_gender_entry (&migrate_gender);
+  
+ +

A couple of points to note about this code. Firstly, we + removed all the gradual migration logic (the callback) + from the class and replaced it with the immediate migration + function. We also removed the odb::nullable + wrapper (and therefore disallowed the NULL values) + since after this migration all the objects will have been + converted. Finally, in the migration function, we only query + the database for objects that need migration, that is, have + NULL gender.

+ +

13.4 Soft Object Model Changes

+ +

Let us consider another common kind of an object model change: + we delete an old member, add a new one, and need to copy + the data from the old to the new, perhaps applying some + conversion. For example, we may realize that in our application + it is a better idea to store a person's name as a single string + rather than split it into three fields. So what we would like to do + is add a new data member, let's call it name_, convert + all the existing split names, and then delete the first_, + middle_, and last_ data members.

+ +

While this sounds straightforward, there is a problem. If we + delete (that is, physically remove from the source code) the + old data members, then we won't be able to access the old + data. The data will still be available in the database between + the schema pre and post-migrations, it is just we will no longer + be able to access it through our object model. And if we keep + the old data members around, then the old data will remain + stored in the database even after the schema post-migration.

+ +

There is also a more subtle problem that has to do with existing + migrations for previous version. Remember, in version 3 + of our person example we've added the gender_ + data member. We also have a data migration function which guesses + the gender based on the first name. Deleting the first_ + data member from our class will obviously break this code. But + even adding the new name_ data member will cause + problems because when we try to update the object in order to + store the new gender, ODB will try to update name_ + as well. But there is no corresponding column in the database + yet. When we run this migration function, we are still several + versions away from the point where the name column + will be added.

+ +

This is a very subtle but also very important implication to + understand. Unlike the main application logic, which only needs + to deal with the current model version, data migration code works + on databases that can be multiple versions behind the current + version.

+ +

How can we resolve this problem? It appears what we need is the + ability to add or delete data members starting from a specific + version. In ODB this mechanism is called soft member additions + and deletions. A soft-added member is only treated as persistent + starting from the addition version. A soft-deleted member is + persistent until the deletion version (but including the migration + stage). In its essence, soft model changes allow us to maintain + multiple versions of our object model all with a single set of + persistent classes. Let us now see how this functionality can + help implement our changes:

+ +
+#pragma db model version(1, 4)
+
+#pragma db object
+class person
+{
+  ...
+
+  #pragma db id auto
+  unsigned long id_;
+
+  #pragma db deleted(4)
+  std::string first_;
+
+  #pragma db deleted(4)
+  std::string middle_;
+
+  #pragma db deleted(4)
+  std::string last_;
+
+  #pragma db added(4)
+  std::string name_;
+
+  gender gender_;
+};
+  
+ +

The migration function for this change could then look like + this:

+ +
+static void
+migrate_name (odb::database& db)
+{
+  for (person& p: db.query<person> ())
+  {
+    p.name (p.first () + " " +
+            p.middle () + (p.middle ().empty () ? "" : " ") +
+            p.last ());
+    db.update (p);
+  }
+}
+
+static const odb::data_migration_entry<4, MYAPP_BASE_VERSION>
+migrate_name_entry (&migrate_name);
+  
+ +

Note also that no changes are required to the gender migration + function.

+ +

As you may have noticed, in the code above we assumed that the + person class still provides public accessors for + the now deleted data members. This might not be ideal since now + they should not be used by the application logic. The only code + that may still need to access them is the migration functions. The + recommended way to resolve this is to remove the accessors/modifiers + corresponding to the deleted data member, make migration functions + static function of the class being migrated, and then access + the deleted data members directly. For example:

+ +
+#pragma db model version(1, 4)
+
+#pragma db object
+class person
+{
+  ...
+
+private:
+  friend class odb::access;
+
+  #pragma db id auto
+  unsigned long id_;
+
+  #pragma db deleted(4)
+  std::string first_;
+
+  #pragma db deleted(4)
+  std::string middle_;
+
+  #pragma db deleted(4)
+  std::string last_;
+
+  #pragma db added(4)
+  std::string name_;
+
+  gender gender_;
+
+private:
+  static void
+  migrate_gender (odb::database&);
+
+  static void
+  migrate_name (odb::database&);
+};
+
+void person::
+migrate_gender (odb::database& db)
+{
+  for (person& p: db.query<person> ())
+  {
+    p.gender_ = guess_gender (p.first_);
+    db.update (p);
+  }
+}
+
+static const odb::data_migration_entry<3, MYAPP_BASE_VERSION>
+migrate_name_entry (&migrate_gender);
+
+void person::
+migrate_name (odb::database& db)
+{
+  for (person& p: db.query<person> ())
+  {
+    p.name_ = p.first_ + " " +
+              p.middle_ + (p.middle_.empty () ? "" : " ") +
+              p.last_;
+    db.update (p);
+  }
+}
+
+static const odb::data_migration_entry<4, MYAPP_BASE_VERSION>
+migrate_name_entry (&migrate_name);
+  
+ +

Another potential issue with the soft-deletion is the requirement + to keep the delete data members in the class. While they will not + be initialized in the normal operation of the application (that + is, not a migration), this can still be a problem if we need to + minimize the memory footprint of our classes. For example, we may + cache a large number of objects in memory and having three + std::string data members can be a significant + overhead.

+ +

The recommended way to resolve this issue is to place all the + deleted data members into a dynamically allocated composite + value type. For example:

+ +
+#pragma db model version(1, 4)
+
+#pragma db object
+class person
+{
+  ...
+
+  #pragma db id auto
+  unsigned long id_;
+
+  #pragma db added(4)
+  std::string name_;
+
+  gender gender_;
+
+  #pragma db value
+  struct deleted_data
+  {
+    #pragma db deleted(4)
+    std::string first_;
+
+    #pragma db deleted(4)
+    std::string middle_;
+
+    #pragma db deleted(4)
+    std::string last_;
+  };
+
+  std::unique_ptr<deleted_data> dd_;
+
+  ...
+};
+  
+ +

ODB will then automatically allocate the deleted value type if + any of the deleted data members are being loaded. During the normal + operation, however, the pointer will stay NULL and + therefore reducing the common case overhead to a single pointer + per class.

+ +

Soft-added and deleted data members can be used in objects, + composite values, views, and container value types. We can + also soft-add and delete data members of simple, composite, + pointer to object, and container types. Only special data + members, such as the object id and the optimistic concurrency + version, cannot be soft-added or deleted.

+ +

It is also possible to soft-delete a persistent class. We + can still work with the existing objects of such a class, + however, no table is created in new databases for soft-deleted + classes. To put it another way, a soft-delete class is like an + abstract class (no table) but which can still be loaded, updated, + etc. Soft-added persistent classes do not make much sense and + are therefore not supported.

+ +

As an example of a soft-deleted class, suppose we want to + replace our person class with the new + employee object and migrate the data. Here is + how we could do this:

+ +
+#pragma db model version(1, 5)
+
+#pragma db object deleted(5)
+class person
+{
+  ...
+};
+
+#pragma db object
+class employee
+{
+  ...
+
+  #pragma db id auto
+  unsigned long id_;
+
+  std::string name_;
+  gender gender_;
+
+  static void
+  migrate_person (odb::database&);
+};
+
+void employee::
+migrate_person (odb::database& db)
+{
+  for (person& p: db.query<person> ())
+  {
+    employee e (p.name (), p.gender ());
+    db.persist (e);
+  }
+}
+
+static const odb::data_migration_entry<5, MYAPP_BASE_VERSION>
+migrate_person_entry (&migrate_person);
+  
+ +

As we have seen above, hard member additions and deletions can + (and most likely will) break existing data migration code. Why, + then, not treat all the changes, or at least additions, as soft? + ODB requires you to explicitly request this semantics because + support for soft-added and deleted data members incurs runtime + overhead. And there can be plenty of cases where there is no + existing data migration and therefore hard additions and deletions + are sufficient.

+ +

In some cases a hard addition or deletion will result in a + compile-time error. For example, one of the data migration + functions may reference the data member we just deleted. In + many cases, however, such errors can only be detected at + runtime, and, worse yet, only when the migration function + is executed. For example, we may hard-add a new data member + that an existing migration function will try to indirectly + store in the database as part of an object update. As a result, + it is highly recommended that you always test your application + with the database that starts at the base version so that every + data migration function is called and therefore made sure to + still work correctly.

+ +

To help with this problem you can also instruct ODB to warn + you about any hard additions or deletions with the + --warn-hard-add, --warn-hard-delete, + and --warn-hard command line options. ODB will + only warn you about hard changes in the current version and + only for as long as it is open, which makes this mechanism + fairly usable.

+ +

You may also be wondering why we have to specify the addition + and deletion versions explicitly. It may seem like the ODB compiler + should be able to figure this out automatically. While it is + theoretically possible, to achieve this, ODB would have to also + maintain a separate changelog of the C++ object model in + addition to the database schema changelog it already maintains. + While being a lot more complex, such an additional changelog + would also complicate the workflow significantly. In this light, + maintaining this change information as part of the original + source files appears to be a cleaner and simpler approach.

+ +

As we discussed before, when we move the base model version + forward we essentially drop support for migrations from + versions before the new base. As a result, it is no longer + necessary to maintain the soft semantics of additions and + deletions up to and including the new base version. ODB + will issue diagnostics for all such members and classes. + For soft deletions we can simply remove the data member or + class entirely. For soft additions we only need to remove the + db added pragma.

+ +

13.4.1 Reuse Inheritance Changes

+ +

Besides adding and deleting data member, another way to alter + the object's table is using reuse-style inheritance. If we add + a new reuse base, then, from the database schema point of view, + this is equivalent to adding all its columns to the derived + object's table. Similarly, deleting reuse inheritance result in + all the base's columns being deleted from the derived's table.

+ +

In the future ODB may provide direct support for soft addition + and deletion of inheritance. Currently, however, this semantics + can be emulated with soft-added and deleted data members. The + following table describes the most common scenarios depending + on where columns are added or deleted, that is, base table, + derived table, or both.

+ + + + + + + + + + + + + + + + + + + + + + + + + + +
DELETEHARDSOFT
In both (delete inheritance and base)Delete inheritance and base. Move object id to derived.Soft-delete base. Mark all data members (except id) in + base as soft-deleted.
In base only (delete base)Option 1: mark base as abstract.

+ Option 2: move all the base member to derived, delete base.
Soft-delete base.
In derived only (delete inheritance)Delete inheritance, add object id to derived.Option 1: copy base to a new soft-deleted base, inherit + from it instead. Mark all the data members (expect id) in + this new base as soft-deleted. Note: we add the new base + as soft-deleted to get notified when we can remove it.

+ Option 2: Copy all the data members from base to derived + and mark them as soft-deleted in derived.
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
ADDHARDSOFT
In both (add new base and inheritance)Add new base and inheritance. Potentially move object id + member from derived to base.Add new base and mark all its data members as soft-added. + Add inheritance. Move object id from derived to base.
In base only (refactor existing data to new base)Add new base and move data members from derived to base. + Note: in most cases the new base will be made abstract + which make this scenario non-schema changing.The same as HARD.
In derived only (add inheritance to existing base)Add inheritance, delete object id in derived.Copy existing base to a new abstract base and inherit + from it. Mark all the database members in the new base + as soft-added (except object id). When notified by the + ODB compiler that the soft addition of the data members + is not longer necessary, delete the copy and inherit from + the original base.
+ +

13.4.2 Polymorphism Inheritance Changes

+ +

Unlike reuse inheritance, adding or deleting a polymorphic base + does not result in the base's data members being added or deleted + from the derived object's table because each class in a polymorphic + hierarchy is stored in a separate table. There are, however, other + complications due to the presence of special columns (discriminator + in the root table and object id links in derived tables) which makes + altering the hierarchy structure difficult to handle automatically. + Adding or deleting (including soft-deleting) of leaf classes (or + leaf sub-hierarchies) in a polymorphic hierarchy is fully supported. + Any more complex changes, such as adding or deleting the root or + an intermediate base or getting an existing class into or out of + a polymorphic hierarchy can be handled by creating a new leaf class + (or leaf sub-hierarchy), soft-deleting the old class, and migrating + the data.

+ + + + +
+

14 ODB Pragma Language

+ +

As we have already seen in previous chapters, ODB uses a pragma-based + language to capture database-specific information about C++ types. + This chapter describes the ODB pragma language in more detail. It + can be read together with other chapters in the manual to get a + sense of what kind of configurations and mapping fine-tuning are + possible. You can also use this chapter as a reference at a later + stage.

+ +

An ODB pragma has the following syntax:

+ +

#pragma db qualifier [specifier specifier ...]

+ +

The qualifier tells the ODB compiler what kind of C++ construct + this pragma describes. Valid qualifiers are object, + view, value, member, + namespace, model, index, and + map. + A pragma with the object qualifier describes a persistent + object type. It tells the ODB compiler that the C++ class it describes + is a persistent class. Similarly, pragmas with the view + qualifier describe view types, the value qualifier + describes value types and the member qualifier is used + to describe data members of persistent object, view, and value types. + The namespace qualifier is used to describe common + properties of objects, views, and value types that belong to + a C++ namespace while the model qualifier describes + the whole C++ object model. The index qualifier defines + a database index. And, finally, the map qualifier + describes a mapping between additional database types and types + for which ODB provides built-in support.

+ +

The specifier informs the ODB compiler about a particular + database-related property of the C++ declaration. For example, the + id member specifier tells the ODB compiler that this + member contains this object's identifier. Below is the declaration + of the person class that shows how we can use ODB + pragmas:

+ +
+#pragma db object
+class person
+{
+  ...
+private:
+  #pragma db member id
+  unsigned long id_;
+  ...
+};
+  
+ +

In the above example we don't explicitly specify which C++ class or + data member the pragma belongs to. Rather, the pragma applies to + a C++ declaration that immediately follows the pragma. Such pragmas + are called positioned pragmas. In positioned pragmas that + apply to data members, the member qualifier can be + omitted for brevity, for example:

+ +
+  #pragma db id
+  unsigned long id_;
+  
+ +

Note also that if the C++ declaration immediately following a + position pragma is incompatible with the pragma qualifier, an + error will be issued. For example:

+ +
+  #pragma db object  // Error: expected class instead of data member.
+  unsigned long id_;
+  
+ +

While keeping the C++ declarations and database declarations close + together eases maintenance and increases readability, we can also + place them in different parts of the same header file or even + factor them to a separate file. To achieve this we use the so called + named pragmas. Unlike positioned pragmas, named pragmas + explicitly specify the C++ declaration to which they apply by + adding the declaration name after the pragma qualifier. For example:

+ +
+class person
+{
+  ...
+private:
+  unsigned long id_;
+  ...
+};
+
+#pragma db object(person)
+#pragma db member(person::id_) id
+  
+ +

Note that in the named pragmas for data members the member + qualifier is no longer optional. The C++ declaration name in the + named pragmas is resolved using the standard C++ name resolution + rules, for example:

+ +
+namespace db
+{
+  class person
+  {
+    ...
+  private:
+    unsigned long id_;
+    ...
+  };
+}
+
+namespace db
+{
+  #pragma db object(person)  // Resolves db::person.
+}
+
+#pragma db member(db::person::id_) id
+  
+ +

As another example, the following code fragment shows how to use the + named value type pragma to map a C++ type to a native database type:

+ +
+#pragma db value(bool) type("INT")
+
+#pragma db object
+class person
+{
+  ...
+private:
+  bool married_; // Mapped to INT NOT NULL database type.
+  ...
+};
+  
+ +

If we would like to factor the ODB pragmas into a separate file, + we can include this file into the original header file (the one + that defines the persistent types) using the #include + directive, for example:

+ +
+// person.hxx
+
+class person
+{
+  ...
+};
+
+#ifdef ODB_COMPILER
+#  include "person-pragmas.hxx"
+#endif
+  
+ +

Alternatively, instead of using the #include directive, + we can use the --odb-epilogue option to make the pragmas + known to the ODB compiler when compiling the original header file, + for example:

+ +
+--odb-epilogue  '#include "person-pragmas.hxx"'
+  
+ +

The following sections cover the specifiers applicable to all the + qualifiers mentioned above.

+ +

The C++ header file that defines our persistent classes and + normally contains one or more ODB pragmas is compiled by both + the ODB compiler to generate the database support code and + the C++ compiler to build the application. Some C++ compilers + issue warnings about pragmas that they do not recognize. There + are several ways to deal with this problem which are covered + at the end of this chapter in Section 14.9, + "C++ Compiler Warnings".

14.1 Object Type Pragmas

@@ -11406,6 +13670,12 @@ class person 14.1.13 + + deleted + persistent class is soft-deleted + 14.1.14 + +

14.1.1 table

@@ -12033,6 +14303,14 @@ class employer Section 9.2, "Sections and Optimistic Concurrency".

+

14.1.14 deleted

+ +

The deleted specifier marks the persistent class as + soft-deleted. The single required argument to this specifier is + the deletion version. For more information on this functionality, + refer to Section 13.4, "Soft Object Model + Changes".

+

14.2 View Type Pragmas

A pragma with the view qualifier declares a C++ class @@ -12341,7 +14619,7 @@ class person refer to Part II, "Database Systems". The null and not_null (Section 14.3.3, "null/not_null") specifiers - can be used to control the NULL semantics of a type.

+ can be used to control the NULL semantics of a type.

In the above example we changed the mapping for the bool type which is now mapped to the INT database type. In @@ -12682,7 +14960,8 @@ typedef std::vector<std::string> names;

The value_null and value_not_null (Section 14.3.13, "value_null/value_not_null") specifiers - can be used to control the NULL semantics of a value column.

+ can be used to control the NULL semantics of a value + column.

14.3.13 value_null/value_not_null

@@ -12725,7 +15004,7 @@ typedef std::vector<std::string> nicknames;

The semantics of the id_options specifier for a container type are similar to those of the id_options specifier for - a container data member (Section 14.4.26, + a container data member (Section 14.4.28, "id_options").

@@ -12742,7 +15021,7 @@ typedef std::vector<std::string> nicknames;

The semantics of the index_options specifier for a container type are similar to those of the index_options specifier for - a container data member (Section 14.4.27, + a container data member (Section 14.4.29, "index_options").

@@ -12759,7 +15038,7 @@ typedef std::map<std::string, std::string> properties;

The semantics of the key_options specifier for a container type are similar to those of the key_options specifier for - a container data member (Section 14.4.28, + a container data member (Section 14.4.30, "key_options").

@@ -12776,7 +15055,7 @@ typedef std::set<std::string> nicknames;

The semantics of the value_options specifier for a container type are similar to those of the value_options specifier for - a container data member (Section 14.4.29, + a container data member (Section 14.4.31, "value_options").

@@ -12981,75 +15260,87 @@ typedef std::map<unsigned short, float> age_weight_map; + added + member is soft-added + 14.4.22 + + + + deleted + member is soft-deleted + 14.4.23 + + + index_type database type for a container's index type - 14.4.22 + 14.4.24 key_type database type for a container's key type - 14.4.23 + 14.4.25 value_type database type for a container's value type - 14.4.24 + 14.4.26 value_null/value_not_null container's value can/cannot be NULL - 14.4.25 + 14.4.27 id_options database options for a container's id column - 14.4.26 + 14.4.28 index_options database options for a container's index column - 14.4.27 + 14.4.29 key_options database options for a container's key column - 14.4.28 + 14.4.30 value_options database options for a container's value column - 14.4.29 + 14.4.31 id_column column name for a container's object id - 14.4.30 + 14.4.32 index_column column name for a container's index - 14.4.31 + 14.4.33 key_column column name for a container's key - 14.4.32 + 14.4.34 value_column column name for a container's value - 14.4.33 + 14.4.35 @@ -13138,8 +15429,8 @@ class person

The null and not_null (Section 14.4.6, "null/not_null") specifiers - can be used to control the NULL semantics of a data member. It is - also possible to specify the database type on the per-type instead + can be used to control the NULL semantics of a data member. + It is also possible to specify the database type on the per-type instead of the per-member basis using the value type specifier (Section 14.3.1, "type").

@@ -14101,7 +16392,7 @@ class person

For more information on defining database indexes, refer to - Section 14.6, "Index Definition Pragmas".

+ Section 14.7, "Index Definition Pragmas".

14.4.17 unique

@@ -14120,7 +16411,7 @@ class person

For more information on defining database indexes, refer to - Section 14.6, "Index Definition Pragmas".

+ Section 14.7, "Index Definition Pragmas".

14.4.18 unordered

@@ -14212,7 +16503,23 @@ class person members of a persistent class. For more information on object sections, refer to Chapter 9, "Sections".

-

14.4.22 index_type

+

14.4.22 added

+ +

The added specifier marks the data member as + soft-added. The single required argument to this specifier is + the addition version. For more information on this functionality, + refer to Section 13.4, "Soft Object Model + Changes".

+ +

14.4.23 deleted

+ +

The deleted specifier marks the data member as + soft-deleted. The single required argument to this specifier is + the deletion version. For more information on this functionality, + refer to Section 13.4, "Soft Object Model + Changes".

+ +

14.4.24 index_type

The index_type specifier specifies the native database type that should be used for an ordered container's @@ -14232,7 +16539,7 @@ class person };

-

14.4.23 key_type

+

14.4.25 key_type

The key_type specifier specifies the native database type that should be used for a map container's @@ -14252,7 +16559,7 @@ class person };

-

14.4.24 value_type

+

14.4.26 value_type

The value_type specifier specifies the native database type that should be used for a container's @@ -14273,11 +16580,12 @@ class person

The value_null and value_not_null - (Section 14.4.25, + (Section 14.4.27, "value_null/value_not_null") specifiers - can be used to control the NULL semantics of a value column.

+ can be used to control the NULL semantics of a value + column.

-

14.4.25 value_null/value_not_null

+

14.4.27 value_null/value_not_null

The value_null and value_not_null specifiers specify that a container's element value for the data member can or @@ -14310,7 +16618,7 @@ class account Multiset Containers") the element value is automatically treated as not allowing a NULL value.

-

14.4.26 id_options

+

14.4.28 id_options

The id_options specifier specifies additional column definition options that should be used for a container's @@ -14334,7 +16642,7 @@ class person of the options specifier (Section 14.4.8, "options").

-

14.4.27 index_options

+

14.4.29 index_options

The index_options specifier specifies additional column definition options that should be used for a container's @@ -14355,7 +16663,7 @@ class person of the options specifier (Section 14.4.8, "options").

-

14.4.28 key_options

+

14.4.30 key_options

The key_options specifier specifies additional column definition options that should be used for a container's @@ -14376,7 +16684,7 @@ class person of the options specifier (Section 14.4.8, "options").

-

14.4.29 value_options

+

14.4.31 value_options

The value_options specifier specifies additional column definition options that should be used for a container's @@ -14397,7 +16705,7 @@ class person of the options specifier (Section 14.4.8, "options").

-

14.4.30 id_column

+

14.4.32 id_column

The id_column specifier specifies the column name that should be used to store the object id in a @@ -14421,7 +16729,7 @@ class person

If the column name is not specified, then object_id is used by default.

-

14.4.31 index_column

+

14.4.33 index_column

The index_column specifier specifies the column name that should be used to store the element index in an @@ -14445,7 +16753,7 @@ class person

If the column name is not specified, then index is used by default.

-

14.4.32 key_column

+

14.4.34 key_column

The key_column specifier specifies the column name that should be used to store the key in a map @@ -14469,7 +16777,7 @@ class person

If the column name is not specified, then key is used by default.

-

14.4.33 value_column

+

14.4.35 value_column

The value_column specifier specifies the column name that should be used to store the element value in a @@ -14709,7 +17017,48 @@ namespace hr "session"). For more information on sessions, refer to Chapter 11, "Session".

-

14.6 Index Definition Pragmas

+

14.6 Object Model Pragmas

+ +

A pragma with the model qualifier describes the + whole C++ object model. For example:

+ +
+#pragma db model ...
+  
+ +

The model qualifier can be followed, in any order, + by one or more specifiers summarized in the table below:

+ + + + + + + + + + + + + + + +
SpecifierSummarySection
versionobject model version14.6.1
+ +

14.6.1 version

+ +

The version specifier specifies the object model + version when schema evolution support is used. The first two + required arguments to this specifier are the base and current + model versions, respectively. The third optional argument + specifies whether the current version is open for changes. + Valid values for this argument are open (the + default) and closed. For more information on + this functionality, refer to Chapter 13, + "Database Schema Evolution".

+ + +

14.7 Index Definition Pragmas

While it is possible to manually add indexes to the generated database schema, it is more convenient to do this as part of @@ -14909,7 +17258,7 @@ class object }; -

14.7 Database Type Mapping Pragmas

+

14.8 Database Type Mapping Pragmas

A pragma with the map qualifier describes a mapping between two database types. For each database system @@ -15110,7 +17459,7 @@ class object for each database, shows how to provide custom mapping for some of the extended types.

-

14.8 C++ Compiler Warnings

+

14.9 C++ Compiler Warnings

When a C++ header file defining persistent classes and containing ODB pragmas is used to build the application, the C++ compiler may @@ -15163,7 +17512,7 @@ class person

The disadvantage of this approach is that it can quickly become overly verbose when positioned pragmas are used.

-

14.8.1 GNU C++

+

14.9.1 GNU C++

GNU g++ does not issue warnings about unknown pragmas unless requested with the -Wall command line option. @@ -15175,7 +17524,7 @@ class person g++ -Wall -Wno-unknown-pragmas ... -

14.8.2 Visual C++

+

14.9.2 Visual C++

Microsoft Visual C++ issues an unknown pragma warning (C4068) at warning level 1 or higher. This means that unless we have disabled @@ -15209,7 +17558,7 @@ class person #pragma warning (pop) -

14.8.3 Sun C++

+

14.9.3 Sun C++

The Sun C++ compiler does not issue warnings about unknown pragmas unless the +w or +w2 option is specified. @@ -15221,7 +17570,7 @@ class person CC +w -erroff=unknownpragma ... -

14.8.4 IBM XL C++

+

14.9.4 IBM XL C++

IBM XL C++ issues an unknown pragma warning (1540-1401) by default. To disable this warning we can add the -qsuppress=1540-1401 @@ -15231,7 +17580,7 @@ CC +w -erroff=unknownpragma ... xlC -qsuppress=1540-1401 ... -

14.8.5 HP aC++

+

14.9.5 HP aC++

HP aC++ (aCC) issues an unknown pragma warning (2161) by default. To disable this warning we can add the +W2161 @@ -15241,7 +17590,7 @@ xlC -qsuppress=1540-1401 ... aCC +W2161 ... -

14.8.6 Clang

+

14.9.6 Clang

Clang does not issue warnings about unknown pragmas unless requested with the -Wall command line option. @@ -16350,7 +18699,7 @@ class object

It is also possible to add support for additional MySQL types, such as geospatial types. For more information, refer to - Section 14.7, "Database Type Mapping + Section 14.8, "Database Type Mapping Pragmas".

17.1.1 String Type Mapping

@@ -16911,7 +19260,7 @@ namespace odb

17.6 MySQL Index Definitions

-

When the index pragma (Section 14.6, +

When the index pragma (Section 14.7, "Index Definition Pragmas") is used to define a MySQL index, the type clause specifies the index type (for example, UNIQUE, FULLTEXT, SPATIAL), @@ -17096,7 +19445,7 @@ class object

It is also possible to add support for additional SQLite types, such as NUMERIC. For more information, refer to - Section 14.7, "Database Type Mapping + Section 14.8, "Database Type Mapping Pragmas".

18.1.1 String Type Mapping

@@ -17824,9 +20173,81 @@ CREATE TABLE Employee ( Recovery). As a result, the recommended way to handle this exception is to re-execute the affected transaction.

+

18.5.7 Database Schema Evolution

+ +

From the list of schema migration changes supported by ODB + (Section 13.2, "Schema Migration"), the + following are not supported by SQLite:

+ + + +

The biggest problem is the lack of support for dropping columns. + This means that it would be impossible to delete a data member + in a persistent class. To work around this limitation ODB + implements logical delete for columns that allow + NULL values. In this case, instead of dropping + the column (in the post-migration stage), the schema migration + statements will automatically reset this column in all the + existing rows to NULL. Any new rows that are + inserted later will also automatically have this column set + to NULL (unless the column specifies a default + value).

+ +

Since it is also impossible to change the column's + NULL/NOT NULL attribute after it + has been added, to make schema evolution support usable in + SQLite, all the columns should be added as NULL + even if semantically they should not allow NULL + values. We should also normally refrain from assigning + default value to columns (Section 14.4.7, + default), unless the space overhead of + a default value is not a concern. Explicitly making all + the data members NULL would be burdensome + and ODB provides the --sqlite-override-null + command line option that forces all the columns, even those + that were explicitly marked NOT NULL, to be + NULL in SQLite.

+ +

SQLite only supports adding foreign keys as part of the + column addition. As a result, we can only add a new + data member of an object pointer type if it points + to an object with a simple (single-column) object id.

+ +

SQLite also doesn't support dropping of foreign keys. + Leaving a foreign key around works well with logical + delete unless we also want to delete the pointed-to + object. In this case we will have to leave an + empty table corresponding to the pointed-to object + around. An alternative would be to make a copy of the + pointing object without the object pointer, migrate the + data, and then delete both the old pointing and the + pointed-to objects. Since this will result in dropping + of the pointing table, the foreign key will be dropped + as well. Yet another, more radical, solution to this + problem is to disable foreign keys checking altogether + (see the foreign_keys SQLite pragma).

+ +

To summarize, to make schema evolution support usable + in SQLite we should pass the --sqlite-override-null + option when compiling our persistent classes and also refrain + from assigning default values to data members. Note also that + this has to be done from the start so that every column is added + as NULL and therefore can be logically deleted later. + In particular, you cannot add the --sqlite-override-null + option when you realize you need to delete a data member. At this + point it is too late since the column has already been added + as NOT NULL in existing databases. We should also + avoid composite object ids if we are planning to use object + relationships.

+

18.6 SQLite Index Definitions

-

When the index pragma (Section 14.6, +

When the index pragma (Section 14.7, "Index Definition Pragmas") is used to define an SQLite index, the type clause specifies the index type (for example, UNIQUE) while the method and @@ -18002,7 +20423,7 @@ class object such as NUMERIC, geometry types, XML, JSON, enumeration types, composite types, arrays, geospatial types, and the key-value store (HSTORE). - For more information, refer to Section 14.7, + For more information, refer to Section 14.8, "Database Type Mapping Pragmas".

19.1.1 String Type Mapping

@@ -18562,7 +20983,7 @@ SHOW integer_datetimes

ODB does not currently natively support the PostgreSQL date-time types with timezone information. However, these types can be accessed by mapping them to one of the natively supported types, as discussed - in Section 14.7, "Database Type Mapping Pragmas".

+ in Section 14.8, "Database Type Mapping Pragmas".

19.5.6 NUMERIC Type Support

@@ -18572,13 +20993,13 @@ SHOW integer_datetimes store NUMERIC values refer to the PostgreSQL documentation. An alternative approach to accessing NUMERIC values is to map this type to one of the natively supported - ones, as discussed in Section 14.7, "Database + ones, as discussed in Section 14.8, "Database Type Mapping Pragmas".

19.6 PostgreSQL Index Definitions

-

When the index pragma (Section 14.6, +

When the index pragma (Section 14.7, "Index Definition Pragmas") is used to define a PostgreSQL index, the type clause specifies the index type (for example, UNIQUE), the method clause specifies the @@ -18760,7 +21181,7 @@ class object

It is also possible to add support for additional Oracle types, such as XML, geospatial types, user-defined types, and collections (arrays, table types). For more information, refer to - Section 14.7, "Database Type Mapping + Section 14.8, "Database Type Mapping Pragmas".

20.1.1 String Type Mapping

@@ -19455,8 +21876,8 @@ CREATE TABLE Employee (

An alternative approach to accessing large FLOAT and NUMBER values is to map these type to one of the - natively supported ones, as discussed in Section - 14.7, "Database Type Mapping Pragmas".

+ natively supported ones, as discussed in Section + 14.8, "Database Type Mapping Pragmas".

Note that a NUMBER type that is used to represent a floating point number (declared by specifying NUMBER @@ -19468,14 +21889,14 @@ CREATE TABLE Employee (

ODB does not currently support the Oracle date-time types with timezone information. However, these types can be accessed by mapping them to one of the natively supported types, as discussed in - Section 14.7, "Database Type Mapping Pragmas".

+ Section 14.8, "Database Type Mapping Pragmas".

20.5.7 LONG Types

ODB does not support the deprecated Oracle LONG and LONG RAW data types. However, these types can be accessed by mapping them to one of the natively supported types, as discussed - in Section 14.7, "Database Type Mapping Pragmas".

+ in Section 14.8, "Database Type Mapping Pragmas".

20.5.8 LOB Types and By-Value Accessors/Modifiers

@@ -19488,9 +21909,24 @@ CREATE TABLE Employee ( data members. As a result, by-reference accessors and modifiers should be used for these data types.

+

20.5.9 Database Schema Evolution

+ +

In Oracle, the type of the name column in the + schema_version table is VARCHAR2(512). + Because this column is a primary key and VARCHAR2 + represents empty strings as NULL values, it is + impossible to store an empty string in this column, which + is what is used to represent the default schema name. As a + result, in Oracle, the empty schema name is stored as a + string containing a single space character. ODB performs + all the necessary translations automatically and normally + you do not need to worry about this implementation detail + unless you are querying or modifying the schema_version + table directly.

+

20.6 Oracle Index Definitions

-

When the index pragma (Section 14.6, +

When the index pragma (Section 14.7, "Index Definition Pragmas") is used to define an Oracle index, the type clause specifies the index type (for example, UNIQUE, BITMAP), the method @@ -19698,7 +22134,7 @@ class object

It is also possible to add support for additional SQL Server types, such as geospatial types, XML, and user-defined types. - For more information, refer to Section 14.7, "Database + For more information, refer to Section 14.8, "Database Type Mapping Pragmas".

21.1.1 String Type Mapping

@@ -20624,7 +23060,7 @@ namespace odb

21.6 SQL Server Index Definitions

-

When the index pragma (Section 14.6, +

When the index pragma (Section 14.7, "Index Definition Pragmas") is used to define an SQL Server index, the type clause specifies the index type (for example, UNIQUE, CLUSTERED), the method @@ -21634,8 +24070,8 @@ class object

Instances of the QString and QByteArray - types are stored as a NULL value if their isNull() - member function returns true.

+ types are stored as a NULL value if their + isNull() member function returns true.

Note also that the QString type is mapped differently depending on whether a member of this type @@ -21697,8 +24133,8 @@ class Person

Instances of the QString and QByteArray types - are stored as a NULL value if their isNull() member - function returns true.

+ are stored as a NULL value if their isNull() + member function returns true.

24.1.3 PostgreSQL Database Type Mapping

@@ -21733,8 +24169,8 @@ class Person

Instances of the QString and QByteArray types - are stored as a NULL value if their isNull() member - function returns true.

+ are stored as a NULL value if their isNull() + member function returns true.

The basic sub-profile also provides support for mapping QString to the CHAR @@ -21788,8 +24224,8 @@ class Person

Instances of the QString and QByteArray types - are stored as a NULL value if their isNull() member - function returns true.

+ are stored as a NULL value if their isNull() + member function returns true.

The basic sub-profile also provides support for mapping QString to the CHAR, @@ -21847,8 +24283,8 @@ class Person

Instances of the QString and QByteArray types - are stored as a NULL value if their isNull() member - function returns true.

+ are stored as a NULL value if their isNull() + member function returns true.

Note also that the QString type is mapped differently depending on whether a member of this type @@ -22249,8 +24685,8 @@ namespace odb

Instances of the QDate, QTime, and - QDateTime types are stored as a NULL value if their - isNull() member function returns true.

+ QDateTime types are stored as a NULL value + if their isNull() member function returns true.

The date-time sub-profile implementation also provides support for mapping QDateTime to the TIMESTAMP @@ -22340,8 +24776,8 @@ class Person

Instances of the QDate, QTime, and - QDateTime types are stored as a NULL value if their - isNull() member function returns true.

+ QDateTime types are stored as a NULL value + if their isNull() member function returns true.

The date-time sub-profile implementation also provides support for mapping QDate and QDateTime to the @@ -22402,8 +24838,8 @@ class Person

Instances of the QDate, QTime, and - QDateTime types are stored as a NULL value if their - isNull() member function returns true.

+ QDateTime types are stored as a NULL value + if their isNull() member function returns true.

24.4.4 Oracle Database Type Mapping

@@ -22438,8 +24874,8 @@ class Person

Instances of the QDate, QTime, and - QDateTime types are stored as a NULL value if their - isNull() member function returns true.

+ QDateTime types are stored as a NULL value + if their isNull() member function returns true.

The date-time sub-profile implementation also provides support for mapping QDateTime to the @@ -22492,8 +24928,8 @@ class person

Instances of the QDate, QTime, and - QDateTime types are stored as a NULL value if their - isNull() member function returns true.

+ QDateTime types are stored as a NULL value + if their isNull() member function returns true.

Note that the DATE, TIME, and DATETIME2 types are only available in SQL Server 2008 and -- cgit v1.1