[Proposal] Use liquibase to manage the database schema

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Proposal] Use liquibase to manage the database schema

Denis Gervalle-2
While trying to reduce the likelihood of duplicate ids for documents, and
extending my patch to provide a proper solution for objects, I fall on a
really unexpected issue: the type of the object identifiers are Integer
where those of documents are Long. This is completely abnormal since we
have several objects per documents, and therefore we need more distinct ids
for objects than documents.

I have therefore upgraded the ids of objects to use Long, and provide an
implementation that use the lowest 64bits of an MD5 key for object in the
same way I do for documents.  This implementation introduce two new
serializer: UidStringEntityReferenceSerializer and
LocalUidStringEntityReferenceSerializer.
I have also bridged the statistics that derived from objects. The new
implementation works perfectly on a new database but...

... I am unable to provide a proper migration procedure, since hibernate
cannot manage changing the types of existing columns. It does not complains
during schema update, it simply do nothing about them. And later the data
migration breaks since my hashes cannot fit in the database properly. After
thorough googling, I understand that hibernate schema updates were not made
for production use and are really limited opposed to the general idea we
had of it.

Since I am currently stucked, I propose to move ahead and use a new tool to
properly manage our database schema upgrade:

http://www.liquibase.org/

Liquibase is Apache licensed and provide a database agnostic version system
for migrating databases. It does not works like a diff tool, but more like
a patch tool, where you provide several XML description of your wanted
changes and it manage to apply or rollback there changes in an ordered
manner to upgrade the database to the latest schema.
There is several way to apply these changes and I would like to see if it
could be integrated in our current migration procedure, or at least as an
independent listener.

WDYT ?

--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Use liquibase to manage the database schema

Roman Muntyanu
Hi Denis,

  Just of curiosity, how did you choose liquibase from the rest? E.g.
    http://code.google.com/p/dbmigrate/
    http://www.dbmaintain.org/overview.html
    http://code.google.com/p/flyway/
    and etc ...
 What were pros and cons?

Regards,
  Roman

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Denis Gervalle
Sent: Thursday, January 26, 2012 21:29 PM
To: XWiki Developers
Subject: [xwiki-devs] [Proposal] Use liquibase to manage the database schema

While trying to reduce the likelihood of duplicate ids for documents, and extending my patch to provide a proper solution for objects, I fall on a really unexpected issue: the type of the object identifiers are Integer where those of documents are Long. This is completely abnormal since we have several objects per documents, and therefore we need more distinct ids for objects than documents.

I have therefore upgraded the ids of objects to use Long, and provide an implementation that use the lowest 64bits of an MD5 key for object in the same way I do for documents.  This implementation introduce two new
serializer: UidStringEntityReferenceSerializer and LocalUidStringEntityReferenceSerializer.
I have also bridged the statistics that derived from objects. The new implementation works perfectly on a new database but...

... I am unable to provide a proper migration procedure, since hibernate cannot manage changing the types of existing columns. It does not complains during schema update, it simply do nothing about them. And later the data migration breaks since my hashes cannot fit in the database properly. After thorough googling, I understand that hibernate schema updates were not made for production use and are really limited opposed to the general idea we had of it.

Since I am currently stucked, I propose to move ahead and use a new tool to properly manage our database schema upgrade:

http://www.liquibase.org/

Liquibase is Apache licensed and provide a database agnostic version system for migrating databases. It does not works like a diff tool, but more like a patch tool, where you provide several XML description of your wanted changes and it manage to apply or rollback there changes in an ordered manner to upgrade the database to the latest schema.
There is several way to apply these changes and I would like to see if it could be integrated in our current migration procedure, or at least as an independent listener.

WDYT ?

--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Use liquibase to manage the database schema

Denis Gervalle-2
Hi Roman,

On Thu, Jan 26, 2012 at 21:53, Roman Muntyanu <[hidden email]>wrote:

> Hi Denis,
>
>  Just of curiosity, how did you choose liquibase from the rest? E.g.
>    http://code.google.com/p/dbmigrate/
>    http://www.dbmaintain.org/overview.html
>    http://code.google.com/p/flyway/
>    and etc ...
>  What were pros and cons?
>

Thanks for asking.
All of the one you mentioned does not provide the most important
IMO: database agnosticism.
You may also add Solidbase and AutoPatch to your list.

Moreover, Liquibase provide an interesting extension for hibernate, and if
we drop the hibernate schema update, we will need it to produce the script
to support our older databases. The Liquibase API seems also more fine
grained and clearer IMO. I do not want a complete black box and I already
have the migration mechanism in place for data.

The best will be the tool that integrate with it and provide the same
database isolation that hibernate provide for data, else I could simply
written the proprietary scripts in my existing system. If you know some
that really compete with liquibase, let me know.



> Regards,
>  Roman
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of
> Denis Gervalle
> Sent: Thursday, January 26, 2012 21:29 PM
> To: XWiki Developers
> Subject: [xwiki-devs] [Proposal] Use liquibase to manage the database
> schema
>
> While trying to reduce the likelihood of duplicate ids for documents, and
> extending my patch to provide a proper solution for objects, I fall on a
> really unexpected issue: the type of the object identifiers are Integer
> where those of documents are Long. This is completely abnormal since we
> have several objects per documents, and therefore we need more distinct ids
> for objects than documents.
>
> I have therefore upgraded the ids of objects to use Long, and provide an
> implementation that use the lowest 64bits of an MD5 key for object in the
> same way I do for documents.  This implementation introduce two new
> serializer: UidStringEntityReferenceSerializer and
> LocalUidStringEntityReferenceSerializer.
> I have also bridged the statistics that derived from objects. The new
> implementation works perfectly on a new database but...
>
> ... I am unable to provide a proper migration procedure, since hibernate
> cannot manage changing the types of existing columns. It does not complains
> during schema update, it simply do nothing about them. And later the data
> migration breaks since my hashes cannot fit in the database properly. After
> thorough googling, I understand that hibernate schema updates were not made
> for production use and are really limited opposed to the general idea we
> had of it.
>
> Since I am currently stucked, I propose to move ahead and use a new tool
> to properly manage our database schema upgrade:
>
> http://www.liquibase.org/
>
> Liquibase is Apache licensed and provide a database agnostic version
> system for migrating databases. It does not works like a diff tool, but
> more like a patch tool, where you provide several XML description of your
> wanted changes and it manage to apply or rollback there changes in an
> ordered manner to upgrade the database to the latest schema.
> There is several way to apply these changes and I would like to see if it
> could be integrated in our current migration procedure, or at least as an
> independent listener.
>
> WDYT ?
>
> --
> Denis Gervalle
> SOFTEC sa - CEO
> eGuilde sarl - CTO
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>



--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Use liquibase to manage the database schema

Roman Muntyanu
Denis,
Thanks for reply :)
Unfortunately I've never had a chance to adopt DB upgrade software on the project. That's why I was asking - to have a qualified reference for the future cases :)

Not to simply take, I'll share one simple and obvious but efficient criterion for choosing a third-party tool (which I've learned by my own bitter experience): Number of active committers >=3 (well, at least >1) and live activity stream with regular releases (at least 1/6 months).

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Denis Gervalle
Sent: Friday, January 27, 2012 00:34 AM
To: XWiki Developers
Subject: Re: [xwiki-devs] [Proposal] Use liquibase to manage the database schema

Hi Roman,

On Thu, Jan 26, 2012 at 21:53, Roman Muntyanu <[hidden email]>wrote:

> Hi Denis,
>
>  Just of curiosity, how did you choose liquibase from the rest? E.g.
>    http://code.google.com/p/dbmigrate/
>    http://www.dbmaintain.org/overview.html
>    http://code.google.com/p/flyway/
>    and etc ...
>  What were pros and cons?
>

Thanks for asking.
All of the one you mentioned does not provide the most important
IMO: database agnosticism.
You may also add Solidbase and AutoPatch to your list.

Moreover, Liquibase provide an interesting extension for hibernate, and if we drop the hibernate schema update, we will need it to produce the script to support our older databases. The Liquibase API seems also more fine grained and clearer IMO. I do not want a complete black box and I already have the migration mechanism in place for data.

The best will be the tool that integrate with it and provide the same database isolation that hibernate provide for data, else I could simply written the proprietary scripts in my existing system. If you know some that really compete with liquibase, let me know.



> Regards,
>  Roman
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf
> Of Denis Gervalle
> Sent: Thursday, January 26, 2012 21:29 PM
> To: XWiki Developers
> Subject: [xwiki-devs] [Proposal] Use liquibase to manage the database
> schema
>
> While trying to reduce the likelihood of duplicate ids for documents,
> and extending my patch to provide a proper solution for objects, I
> fall on a really unexpected issue: the type of the object identifiers
> are Integer where those of documents are Long. This is completely
> abnormal since we have several objects per documents, and therefore we
> need more distinct ids for objects than documents.
>
> I have therefore upgraded the ids of objects to use Long, and provide
> an implementation that use the lowest 64bits of an MD5 key for object
> in the same way I do for documents.  This implementation introduce two
> new
> serializer: UidStringEntityReferenceSerializer and
> LocalUidStringEntityReferenceSerializer.
> I have also bridged the statistics that derived from objects. The new
> implementation works perfectly on a new database but...
>
> ... I am unable to provide a proper migration procedure, since
> hibernate cannot manage changing the types of existing columns. It
> does not complains during schema update, it simply do nothing about
> them. And later the data migration breaks since my hashes cannot fit
> in the database properly. After thorough googling, I understand that
> hibernate schema updates were not made for production use and are
> really limited opposed to the general idea we had of it.
>
> Since I am currently stucked, I propose to move ahead and use a new
> tool to properly manage our database schema upgrade:
>
> http://www.liquibase.org/
>
> Liquibase is Apache licensed and provide a database agnostic version
> system for migrating databases. It does not works like a diff tool,
> but more like a patch tool, where you provide several XML description
> of your wanted changes and it manage to apply or rollback there
> changes in an ordered manner to upgrade the database to the latest schema.
> There is several way to apply these changes and I would like to see if
> it could be integrated in our current migration procedure, or at least
> as an independent listener.
>
> WDYT ?
>
> --
> Denis Gervalle
> SOFTEC sa - CEO
> eGuilde sarl - CTO
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>



--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Use liquibase to manage the database schema

Sergiu Dumitriu-2
In reply to this post by Denis Gervalle-2
On 01/26/2012 02:29 PM, Denis Gervalle wrote:

> While trying to reduce the likelihood of duplicate ids for documents, and
> extending my patch to provide a proper solution for objects, I fall on a
> really unexpected issue: the type of the object identifiers are Integer
> where those of documents are Long. This is completely abnormal since we
> have several objects per documents, and therefore we need more distinct ids
> for objects than documents.
>
> I have therefore upgraded the ids of objects to use Long, and provide an
> implementation that use the lowest 64bits of an MD5 key for object in the
> same way I do for documents.  This implementation introduce two new
> serializer: UidStringEntityReferenceSerializer and
> LocalUidStringEntityReferenceSerializer.
> I have also bridged the statistics that derived from objects. The new
> implementation works perfectly on a new database but...
>
> ... I am unable to provide a proper migration procedure, since hibernate
> cannot manage changing the types of existing columns. It does not complains
> during schema update, it simply do nothing about them. And later the data
> migration breaks since my hashes cannot fit in the database properly. After
> thorough googling, I understand that hibernate schema updates were not made
> for production use and are really limited opposed to the general idea we
> had of it.
>
> Since I am currently stucked, I propose to move ahead and use a new tool to
> properly manage our database schema upgrade:
>
> http://www.liquibase.org/
>
> Liquibase is Apache licensed and provide a database agnostic version system
> for migrating databases. It does not works like a diff tool, but more like
> a patch tool, where you provide several XML description of your wanted
> changes and it manage to apply or rollback there changes in an ordered
> manner to upgrade the database to the latest schema.
> There is several way to apply these changes and I would like to see if it
> could be integrated in our current migration procedure, or at least as an
> independent listener.
>
> WDYT ?
>

+1, sounds good.

--
Sergiu Dumitriu
http://purl.org/net/sergiu/
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs