[Proposal] Strategy for migrating to the new Model

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Proposal] Strategy for migrating to the new Model

vmassol
Administrator
Hi devs,

Just wanted to share my vision of how we should tackle migrating to the new Model. I see the following steps:

* Step 1: Define new model interfaces (status: in progress)
* Step 2: Implement a "bridged" version which uses the oldcore (status: in progress).
* Step 3: Start moving code to use the new API as the new API and its implementation progress. Note: we should start using the produce of step1 and step2 ASAP to tune the details (status: not started)
* Step 4: At the same time, start a new implementation based on a RDBMS (probably hibernate-based, to be decided) (status: not started). I'd also like that we start other implementations not based on a RDBMS just to prove that it works with other storages. Ideally I'd like some NoSQL impl (Caleb maybe?) and I'd also like to try a Git-based implementation (using jgit)
* Step 5: Deprecate all our search apis located in XWikiHibernateStore and make everyone use the new QueryManager module. This needs some tuning on the QueryManager for missing stuff but that's doable (I need to send some proposal on missing stuff). (status: in progress). The idea here is to decouple search from storage. Note that we'll need to write some translator from HQL to XWQL or the new search query language.
* Step 6: As we progress in step 2, 3, 4, introduce a configuration parameter to decide which implementation to use ("bridged", etc) so that users can start playing with new implementations (status: not started)
* Step 7: Rewrite a new Importer/Exporter that exports everything (all the data in the current DB) + all configuration files/data. To see what we are currently not exporting, see http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Backup#HUsingtheXWikiExportfeature This new exporter should probably be based on the XWiki Streams module being developed here: https://github.com/xwiki-contrib/wiki-stream (status: in progress but not active)
* Step 8: When users want to migrate from one implementation ("bridged" for ex) to a new implementation they export their wiki, set the new implementation in the configuration file and reimport. (status: not started)

For me Step3 can almost begin (I probably need one or 2 more weeks to be ready to have some use cases implemented and I'll send a vote to merge my work in feature-newmodel branch in master - Would be good if you guys start looking at it and give comments to be ready for this).

Then we need volunteer for Step 4 for:
* new RDBMS implementation. Who?
* noSQL impl. Cassandra? other? Who?
* git implementation. Vincent

WDYT about the plan?

In term of time required it's probably going to take us about a year to have a first working version for all the steps by working at a leisurely pace.

Thanks
-Vincent

_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Strategy for migrating to the new Model

Thomas Mortagne
Administrator
On Sat, Sep 15, 2012 at 8:52 AM, Vincent Massol <[hidden email]> wrote:

> Hi devs,
>
> Just wanted to share my vision of how we should tackle migrating to the new Model. I see the following steps:
>
> * Step 1: Define new model interfaces (status: in progress)
> * Step 2: Implement a "bridged" version which uses the oldcore (status: in progress).
> * Step 3: Start moving code to use the new API as the new API and its implementation progress. Note: we should start using the produce of step1 and step2 ASAP to tune the details (status: not started)
> * Step 4: At the same time, start a new implementation based on a RDBMS (probably hibernate-based, to be decided) (status: not started). I'd also like that we start other implementations not based on a RDBMS just to prove that it works with other storages. Ideally I'd like some NoSQL impl (Caleb maybe?) and I'd also like to try a Git-based implementation (using jgit)
> * Step 5: Deprecate all our search apis located in XWikiHibernateStore and make everyone use the new QueryManager module. This needs some tuning on the QueryManager for missing stuff but that's doable (I need to send some proposal on missing stuff). (status: in progress). The idea here is to decouple search from storage. Note that we'll need to write some translator from HQL to XWQL or the new search query language.
> * Step 6: As we progress in step 2, 3, 4, introduce a configuration parameter to decide which implementation to use ("bridged", etc) so that users can start playing with new implementations (status: not started)
> * Step 7: Rewrite a new Importer/Exporter that exports everything (all the data in the current DB) + all configuration files/data. To see what we are currently not exporting, see http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Backup#HUsingtheXWikiExportfeature This new exporter should probably be based on the XWiki Streams module being developed here: https://github.com/xwiki-contrib/wiki-stream (status: in progress but not active)
> * Step 8: When users want to migrate from one implementation ("bridged" for ex) to a new implementation they export their wiki, set the new implementation in the configuration file and reimport. (status: not started)

Sounds good but just to be clear it's not steps that has to be fully
completed one after another, right ?

>
> For me Step3 can almost begin (I probably need one or 2 more weeks to be ready to have some use cases implemented and I'll send a vote to merge my work in feature-newmodel branch in master - Would be good if you guys start looking at it and give comments to be ready for this).

Yes we need to start step3 ASAP (I insist on the fact that it does not
require steps 1 and 2 to be fully completed, we should keep the API
experimental as long as we don't are fully happy with it), that's the
only way to do a good API for us IMO.

>
> Then we need volunteer for Step 4 for:
> * new RDBMS implementation. Who?
> * noSQL impl. Cassandra? other? Who?
> * git implementation. Vincent
>
> WDYT about the plan?
>
> In term of time required it's probably going to take us about a year to have a first working version for all the steps by working at a leisurely pace.

I would say probably more than that unless we really push on it.

>
> Thanks
> -Vincent
>
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs



--
Thomas Mortagne
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Strategy for migrating to the new Model

sasinda
Hi,
*FYI*
> * noSQL impl. Cassandra? other? Who?
You may like to look at the Hibernate OGM(Object to Grid Mapping) project.
http://docs.jboss.org/hibernate/ogm/3.0/reference/en-US/html_single/#ogm-howtocontribute-contribute

Unfortunately the "dialect" for Cassandra is yet a work in progress.
Full support is available for Infinispan data grid.

Thanks
Sasinda.


On Mon, Sep 17, 2012 at 12:19 PM, Thomas Mortagne <[hidden email]
> wrote:

> On Sat, Sep 15, 2012 at 8:52 AM, Vincent Massol <[hidden email]>
> wrote:
> > Hi devs,
> >
> > Just wanted to share my vision of how we should tackle migrating to the
> new Model. I see the following steps:
> >
> > * Step 1: Define new model interfaces (status: in progress)
> > * Step 2: Implement a "bridged" version which uses the oldcore (status:
> in progress).
> > * Step 3: Start moving code to use the new API as the new API and its
> implementation progress. Note: we should start using the produce of step1
> and step2 ASAP to tune the details (status: not started)
> > * Step 4: At the same time, start a new implementation based on a RDBMS
> (probably hibernate-based, to be decided) (status: not started). I'd also
> like that we start other implementations not based on a RDBMS just to prove
> that it works with other storages. Ideally I'd like some NoSQL impl (Caleb
> maybe?) and I'd also like to try a Git-based implementation (using jgit)
> > * Step 5: Deprecate all our search apis located in XWikiHibernateStore
> and make everyone use the new QueryManager module. This needs some tuning
> on the QueryManager for missing stuff but that's doable (I need to send
> some proposal on missing stuff). (status: in progress). The idea here is to
> decouple search from storage. Note that we'll need to write some translator
> from HQL to XWQL or the new search query language.
> > * Step 6: As we progress in step 2, 3, 4, introduce a configuration
> parameter to decide which implementation to use ("bridged", etc) so that
> users can start playing with new implementations (status: not started)
> > * Step 7: Rewrite a new Importer/Exporter that exports everything (all
> the data in the current DB) + all configuration files/data. To see what we
> are currently not exporting, see
> http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Backup#HUsingtheXWikiExportfeatureThis new exporter should probably be based on the XWiki Streams module
> being developed here: https://github.com/xwiki-contrib/wiki-stream(status: in progress but not active)
> > * Step 8: When users want to migrate from one implementation ("bridged"
> for ex) to a new implementation they export their wiki, set the new
> implementation in the configuration file and reimport. (status: not started)
>
> Sounds good but just to be clear it's not steps that has to be fully
> completed one after another, right ?
>
> >
> > For me Step3 can almost begin (I probably need one or 2 more weeks to be
> ready to have some use cases implemented and I'll send a vote to merge my
> work in feature-newmodel branch in master - Would be good if you guys start
> looking at it and give comments to be ready for this).
>
> Yes we need to start step3 ASAP (I insist on the fact that it does not
> require steps 1 and 2 to be fully completed, we should keep the API
> experimental as long as we don't are fully happy with it), that's the
> only way to do a good API for us IMO.
>
> >
> > Then we need volunteer for Step 4 for:
> > * new RDBMS implementation. Who?
> > * noSQL impl. Cassandra? other? Who?
> > * git implementation. Vincent
> >
> > WDYT about the plan?
> >
> > In term of time required it's probably going to take us about a year to
> have a first working version for all the steps by working at a leisurely
> pace.
>
> I would say probably more than that unless we really push on it.
>
> >
> > Thanks
> > -Vincent
> >
> > _______________________________________________
> > devs mailing list
> > [hidden email]
> > http://lists.xwiki.org/mailman/listinfo/devs
>
>
>
> --
> Thomas Mortagne
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Strategy for migrating to the new Model

Caleb James DeLisle
In reply to this post by vmassol


On 09/15/2012 08:52 AM, Vincent Massol wrote:
> Hi devs,
>
> Just wanted to share my vision of how we should tackle migrating to the new Model. I see the following steps:
>
> * Step 1: Define new model interfaces (status: in progress)

We will probably need to revisit this step a number of times to get a really compelling interface.

> * Step 2: Implement a "bridged" version which uses the oldcore (status: in progress).
> * Step 3: Start moving code to use the new API as the new API and its implementation progress. Note: we should start using the produce of step1 and step2 ASAP to tune the details (status: not started)
> * Step 4: At the same time, start a new implementation based on a RDBMS (probably hibernate-based, to be decided) (status: not started). I'd also like that we start other implementations not based on a RDBMS just to prove that it works with other storages. Ideally I'd like some NoSQL impl (Caleb maybe?) and I'd also like to try a Git-based implementation (using jgit)
> * Step 5: Deprecate all our search apis located in XWikiHibernateStore and make everyone use the new QueryManager module. This needs some tuning on the QueryManager for missing stuff but that's doable (I need to send some proposal on missing stuff). (status: in progress). The idea here is to decouple search from storage. Note that we'll need to write some translator from HQL to XWQL or the new search query language.
> * Step 6: As we progress in step 2, 3, 4, introduce a configuration parameter to decide which implementation to use ("bridged", etc) so that users can start playing with new implementations (status: not started)
> * Step 7: Rewrite a new Importer/Exporter that exports everything (all the data in the current DB) + all configuration files/data. To see what we are currently not exporting, see http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Backup#HUsingtheXWikiExportfeature This new exporter should probably be based on the XWiki Streams module being developed here: https://github.com/xwiki-contrib/wiki-stream (status: in progress but not active)
> * Step 8: When users want to migrate from one implementation ("bridged" for ex) to a new implementation they export their wiki, set the new implementation in the configuration file and reimport. (status: not started)
>
> For me Step3 can almost begin (I probably need one or 2 more weeks to be ready to have some use cases implemented and I'll send a vote to merge my work in feature-newmodel branch in master - Would be good if you guys start looking at it and give comments to be ready for this).
>
> Then we need volunteer for Step 4 for:
> * new RDBMS implementation. Who?

IMO we should do some kind of a cost/benefit analysis on different ORMs and different schema options.
I can share some of my experiences with DataNucleus/Cassandra which might help us at least with designing a fast and scalable schema.
As a side note, if DataNucleus/RDBMS looks like a good option, we can reuse a large amount of code from DN/Cassandra.
Of course implementing 2 out of 3 implementations with DataNucleus carries risk of lock-in.

> * noSQL impl. Cassandra? other? Who?

Porting the existing DataNucleus/Cassandra implementation should be easy, it stores generic user defined classes so it can represent basically anything.

> * git implementation. Vincent
>
> WDYT about the plan?

+1

Thanks,
Caleb


>
> In term of time required it's probably going to take us about a year to have a first working version for all the steps by working at a leisurely pace.
>
> Thanks
> -Vincent
>
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>


_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Strategy for migrating to the new Model

Jérôme Velociter
On 09/19/12 13:09, Caleb James DeLisle wrote:

>
> On 09/15/2012 08:52 AM, Vincent Massol wrote:
>> Hi devs,
>>
>> Just wanted to share my vision of how we should tackle migrating to the new Model. I see the following steps:
>>
>> * Step 1: Define new model interfaces (status: in progress)
> We will probably need to revisit this step a number of times to get a really compelling interface.
>
>> * Step 2: Implement a "bridged" version which uses the oldcore (status: in progress).
>> * Step 3: Start moving code to use the new API as the new API and its implementation progress. Note: we should start using the produce of step1 and step2 ASAP to tune the details (status: not started)
>> * Step 4: At the same time, start a new implementation based on a RDBMS (probably hibernate-based, to be decided) (status: not started). I'd also like that we start other implementations not based on a RDBMS just to prove that it works with other storages. Ideally I'd like some NoSQL impl (Caleb maybe?) and I'd also like to try a Git-based implementation (using jgit)
>> * Step 5: Deprecate all our search apis located in XWikiHibernateStore and make everyone use the new QueryManager module. This needs some tuning on the QueryManager for missing stuff but that's doable (I need to send some proposal on missing stuff). (status: in progress). The idea here is to decouple search from storage. Note that we'll need to write some translator from HQL to XWQL or the new search query language.
>> * Step 6: As we progress in step 2, 3, 4, introduce a configuration parameter to decide which implementation to use ("bridged", etc) so that users can start playing with new implementations (status: not started)
>> * Step 7: Rewrite a new Importer/Exporter that exports everything (all the data in the current DB) + all configuration files/data. To see what we are currently not exporting, see http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Backup#HUsingtheXWikiExportfeature This new exporter should probably be based on the XWiki Streams module being developed here: https://github.com/xwiki-contrib/wiki-stream (status: in progress but not active)
>> * Step 8: When users want to migrate from one implementation ("bridged" for ex) to a new implementation they export their wiki, set the new implementation in the configuration file and reimport. (status: not started)
>>
>> For me Step3 can almost begin (I probably need one or 2 more weeks to be ready to have some use cases implemented and I'll send a vote to merge my work in feature-newmodel branch in master - Would be good if you guys start looking at it and give comments to be ready for this).
>>
>> Then we need volunteer for Step 4 for:
>> * new RDBMS implementation. Who?
> IMO we should do some kind of a cost/benefit analysis on different ORMs and different schema options.

+1

> I can share some of my experiences with DataNucleus/Cassandra which might help us at least with designing a fast and scalable schema.
> As a side note, if DataNucleus/RDBMS looks like a good option, we can reuse a large amount of code from DN/Cassandra.
> Of course implementing 2 out of 3 implementations with DataNucleus carries risk of lock-in.

Yes and that would be the same if we go Hibernate/Hibernate OGM.
One thing to weight is which standard we use underneath. JPA seems to
have more active Open Source implementations, so that would mean a
stronger intermediary mapping layer that can potentially be shared for
multiple vendors (hibernate, DN, etc.). By opposition, there seems to be
only DataNucleus as a JDO 3 implementation, but it has the advantage to
be more thought out to be backed by any kind of datastore, and DN brings
a lot such implementations (though it does not support Cassandra, if we
decide to go for Cassandra, it means we support the DN/Cassandra plugin
ourselves).


>
>> * noSQL impl. Cassandra? other? Who?
> Porting the existing DataNucleus/Cassandra implementation should be easy, it stores generic user defined classes so it can represent basically anything.
>
>> * git implementation. Vincent
>>
>> WDYT about the plan?

+1

Jerome

> +1
>
> Thanks,
> Caleb
>
>
>> In term of time required it's probably going to take us about a year to have a first working version for all the steps by working at a leisurely pace.
>>
>> Thanks
>> -Vincent
>>
>> _______________________________________________
>> devs mailing list
>> [hidden email]
>> http://lists.xwiki.org/mailman/listinfo/devs
>>
>
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs

_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs