[Brainstorming] XWiki Architecture after 10+ years: what's next?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Brainstorming] XWiki Architecture after 10+ years: what's next?

vmassol
Administrator
Hi devs,

We’ve been developing XWiki over 10 years now. I’d like to start a discussion about major architecture changes we may want to do in the coming 5 years.

With this discussion I have 2 goals in mind:
* Prepare XWiki for the challenges ahead (prevent it from being obsolete, have an architecture that can adapt to changes, etc)
* Make XWiki an interesting project to develop on for its developers (interesting technologies, interesting intellectual challenges, etc)

Let me start with a few large architecture items I can think about:

* Polyglotism for the front end. This is the ability to code the XWiki UI in any language (PHP, Node.js, javascript, native applications, etc). Right now XWiki UIs are coded mostly using Velocity (and a bit of javascript). It would be nice to be more agnostic and to attract non java developers. The core (Server part) could still be in Java but it would be nice that UIs and extensions could be developed in any language. One architecture that could achieve this would be to have a Core Server exposing all operations as REST APIs. This would decouple the Server part from the Client part. It also means having all XWiki API modules offering REST APIs and making it extra easy to add new REST APIs.

* Greatly simplify development of XWiki Extensions. There are several directions that I can think of:
** Direction 1: Expose the resources making up an Extension on the file system and let users use their favorite tools to write the content (web IDEs, text editors, etc)
** Direction 2: Develop an IDE in the cloud. Again there are 2 options here:
*** Take ownership of the XWiki Web IDE application  and push it much further
*** Go in the direction of integrating with a well-known cloud IDE such as Eclipse Che/CodeEnvy (with pre-built workspaces, docker VMs and one click deploy to test your extension, direct deployment of extensions to e.x.o, etc).
** Other: In general, make it faster to develop extensions. The advantage of the Cloud IDE or Web IDE is that it removes the burden of setting up the tools on your local machine (maven, java, etc) so someone new to XWiki should be operational under 5 minutes to run the first tutorial.

* Promote our SOLR Query Language as the main query language for XWiki and deprecate XWQL/HQL. Goal: make querying independent of the stores (right now we have a single store implemented on a RDBMS but we can imagine in the future moving to another type of store, and even to multiple stores).
** Make it easy for extensions to contribute new SOLR indexes for their needs.
** Once we use only SOLR QL we can then more easily switch to a different database model. We would also need the next Generation XAR format to be able to fully export an XWiki instance (or some part of it) into a XAR and be able to reimport it (right now lots of stuff are not in the XAR such as data found in the permanent directory).

* Move towards a container-based approach to install/deploy XWiki (docker, etc). The idea being to start being microservices-friendly. Right now we already have 2 such services in XWiki: external SOLR + office imports with an office server. We need to be able to include those in distributions and add more. It should be easy to develop a new microservice and set it up in XWiki for redistribution.
** It may be interesting to investigate infrastructure frameworks such as kubernetes, apache karaf and the like. The goal being to build systems that are more responsive, resilient, auto-adaptive, elastic (see the reactive manifesto at http://www.reactivemanifesto.org/).
** We could imagine to be able to package our office import micro-services (ie including an office server) as an extension that can be installed and deployed in the XWiki system automatically.

Ok that’s already a good start.

Let me know if you feel excited by some of those and feel free to add more. Note that the idea here is to brainstorm about large architecture changes.

Thanks
-Vincent








_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Brainstorming] XWiki Architecture after 10+ years: what's next?

Caleb James DeLisle-3
Well this is a very interesting topic of discusssion and me and the XWiki SAS Research team
would like to do whatever we can to help evolve XWiki forward (but don't let us get in the way!)


On 04/11/16 14:34, Vincent Massol wrote:
> Hi devs,
>
> We’ve been developing XWiki over 10 years now. I’d like to start a discussion about major architecture changes we may want to do in the coming 5 years.
>
> With this discussion I have 2 goals in mind:
> * Prepare XWiki for the challenges ahead (prevent it from being obsolete, have an architecture that can adapt to changes, etc)
> * Make XWiki an interesting project to develop on for its developers (interesting technologies, interesting intellectual challenges, etc)


The futurist in me sees an ongoing struggle between hot new technologies and good working systems
as a reflection of the struggle between engineers who love to play and learn vs. managers who
have to balance budget. Of course we want something of a middle road where we pluck out the good
*ideas* from modern technologies without being swept up in every fad.


>
> Let me start with a few large architecture items I can think about:
>
> * Polyglotism for the front end. This is the ability to code the XWiki UI in any language (PHP, Node.js, javascript, native applications, etc). Right now XWiki UIs are coded mostly using Velocity (and a bit of javascript). It would be nice to be more agnostic and to attract non java developers. The core (Server part) could still be in Java but it would be nice that UIs and extensions could be developed in any language. One architecture that could achieve this would be to have a Core Server exposing all operations as REST APIs. This would decouple the Server part from the Client part. It also means having all XWiki API modules offering REST APIs and making it extra easy to add new REST APIs.


I agree with the thinking because I feel Velocity has lived out it's useful life, it drops all of
its variables to the global scope and doesn't have functions which makes debugging terrible and
it is positively alien to almost everyone who comes to XWiki.

That said, I want to caution about the lure of anything-anywhere-polyglotism. Each language has
it's own ecosystem. Virt.x project made a system where they advertized you could program in Java,
Javascript, Groovy, Ruby or Ceylon. As far as I know, this project has not gotten any significant
traction and I think this is because their Javascript is still alien to the Nodejs community,
their Ruby is still alien to the Rails community, etc.

My opinion is we must choose between being a little piece of a bigger ecosystem (node, php, etc.)
or we must build our own ecosystem (what we have been doing so far). That said, I still think we
should begin phasing out Velocity.

My recommendation is we pick a language which works reasonably well for our needs yet has
significant familiarity and then begin to make this language the Lingua franca of XWiki and begin
porting our Velocity code and documentation over to it.


>
> * Greatly simplify development of XWiki Extensions. There are several directions that I can think of:
> ** Direction 1: Expose the resources making up an Extension on the file system and let users use their favorite tools to write the content (web IDEs, text editors, etc)
> ** Direction 2: Develop an IDE in the cloud. Again there are 2 options here:
> *** Take ownership of the XWiki Web IDE application  and push it much further
> *** Go in the direction of integrating with a well-known cloud IDE such as Eclipse Che/CodeEnvy (with pre-built workspaces, docker VMs and one click deploy to test your extension, direct deployment of extensions to e.x.o, etc).
> ** Other: In general, make it faster to develop extensions. The advantage of the Cloud IDE or Web IDE is that it removes the burden of setting up the tools on your local machine (maven, java, etc) so someone new to XWiki should be operational under 5 minutes to run the first tutorial.


 From observation of the patterns which engineers choose to take when not pushed, I find that they
are generally more comfortable with direction 1. Obviously we in the Research team would be thrilled
to see WebIDE take on a new life as an XWiki product but I find that engineers will go to war for
their text editor and prefer a process which is more inline with what they know from nodejs, php or
ruby on rails.

The issues which I have observed blocking people who come to the platform from outside are:
* APIs are complex and undiscoverable, it's ok to put them on the website (php, nodejs do this) but they need to be first on google when you search XWiki API.
* No clear connection between development and github.
* Release process is crazy: requires maven flags, settings.xml, PGP keys, java versions. Compared to `npm publish` or bower where you need only git tag and you're done, this is stone age.
* Fragility because of layers of legacy code: for people used to throwing things together with node, ruby or php, the level of fault handling which is required to make a stable XWiki app is like working on the OS kernel.

1-3 are resolvable but #4 is something that is usually only fixed by doing a ground-up rewrite of
the application, something which will become easier and easier as time goes on due to the ever
maturing tools available in different ecosystems. This is an oppertunity and a risk, the oppertunity
is that we can take our body of knowledge and use it to build something new which is cheaper to
maintain and more exciting for others to use, the risk is that others will be increasingly able to
do the same.

A solution to everything is that we don't think of XWiki so much as a programming platform and try
to just make a very good KM tool, then we only fight #4 internally.


>
> * Promote our SOLR Query Language as the main query language for XWiki and deprecate XWQL/HQL. Goal: make querying independent of the stores (right now we have a single store implemented on a RDBMS but we can imagine in the future moving to another type of store, and even to multiple stores).
> ** Make it easy for extensions to contribute new SOLR indexes for their needs.
> ** Once we use only SOLR QL we can then more easily switch to a different database model. We would also need the next Generation XAR format to be able to fully export an XWiki instance (or some part of it) into a XAR and be able to reimport it (right now lots of stuff are not in the XAR such as data found in the permanent directory).


Personally I would run the other direction, limit the database type in order to limit the scope
of problems which we might have to solve so we can concentrate on things which make customer value.
One way to do this is use an internal database (absolute control) paired with a "replication" store
(absolute flexibility).


>
> * Move towards a container-based approach to install/deploy XWiki (docker, etc). The idea being to start being microservices-friendly. Right now we already have 2 such services in XWiki: external SOLR + office imports with an office server. We need to be able to include those in distributions and add more. It should be easy to develop a new microservice and set it up in XWiki for redistribution.


Docker images make problems go away for a tester so I agree with the idea, beware of microservices
for microservices sake though and also word on the street is nobody uses docker in production so
after the testing phase is over, people still need a solution to install for real.


> ** It may be interesting to investigate infrastructure frameworks such as kubernetes, apache karaf and the like. The goal being to build systems that are more responsive, resilient, auto-adaptive, elastic (see the reactive manifesto at http://www.reactivemanifesto.org/).

Buzzwords... please identify the pain-point first.

I suppose this is a good place to say that right now, everybody in the tech space runs the risk of
jumping from one dead technology to another because when an organization identifies that a
particular technology in their stack is dead, they are often lead to adopt the one which is making
the most buzz in the hopes of being a long time away from the (assumed inevitable) death of that
technology.

However, we must not lose sight of the Darwinian nature of technology lifecycles.

Moving from a dead technology such as PrototypeJS to a "hot" technology such as AngularJS runs a
great risk of picking a loser and then when we find that it's parent company abandons it in favor
of Angular2 and the community abandons both in favor of ReactJS, we're hosed. If instead we move
to a well established yet still reasonably modern technology such as JQuery, we have very little
risk of this being uprooted.


Globally I would recommend identifying what it is you want to make, then ensure there is a way to
make business around it (otherwise the project will go the way of GNU/everything) and then finally
start looking for the technologies/ecosystems which reach the best happy-medium between getting you
where you want to be, making innovation so they will still be with you in the future and being
stable and unlikely to go away.


Thanks,
Caleb


> ** We could imagine to be able to package our office import micro-services (ie including an office server) as an extension that can be installed and deployed in the XWiki system automatically.
>
> Ok that’s already a good start.
>
> Let me know if you feel excited by some of those and feel free to add more. Note that the idea here is to brainstorm about large architecture changes.
>
> Thanks
> -Vincent
>
>
>
>
>
>
>
>
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Brainstorming] XWiki Architecture after 10+ years: what's next?

Ecaterina Moraru (Valica)
In reply to this post by vmassol
There is an interesting point Caled did, the one about identifying what we
want to make first. I am not sure we have anywhere on xwiki.org a page that
describes what are the qualities that XWiki has/wants. Is Polyglotism one
of them? Is backwards compatibility an attribute of XWiki? Being a platform
is something that we will always be? Defining our core values as a
community would indeed make the direction selection easier and also would
help new developers see if this is an ecosystem they want to be a part of.

Most of the things you mentioned Vincent are a bit technical, so I'm not
sure I understood them fully, but I cannot say I disagree with anything.

The most interesting domain I would be interested in is the "Polyglotism
for the front end". Also I don't think we will find a solution that would
allow us to write once and then have the code easily updateble. Once every
couple of years we will need to rewrite parts of our code, in order to keep
them up to date. I love our ability to easily integrate any new
framework/language inside XWiki and make it work somehow. This will always
penalize us in terms of consistency of look and behavior or in terms of
performance, but being able to be that extensible and flexible is great (at
least from a developer's perspective).

There is another thread started by Guillaume D about "Bootstrap 3.x.x will
no longer be developed" http://markmail.org/thread/ba5oy3mqsrcp7zpp . Being
a product for such a long period of time, this is the hardest thing to do:
trying to keep up and adapt to new technologies. And adding new stuff is
easy. The hard part is knowing when to let something go. I'm interested in
replacing Velocity with something else as part of our standards and
recommendations, but we will still need to keep the Velocity support for a
long time. And this is a recurring question that we had for PrototypeJS,
that we have now for Bootstrap 3, for Velocity, etc.

I am not sure if we should have boundaries on the number of versions we
support for a certain language or framework, or the time we bundle
something in XWiki. Not sure if we can have like a rule or it's always a
case by case. It would be great to have a tool that analyses how much each
language / framework is used for a particular instance (also inside the
pages) or even for platform. Something like in GitHub, when they say your
product is 92% Java, 4% JavaScript, etc. Then it would be so easy to
justify the need to keep a particular language or framework || or to
understand how hard is to replace something like Velocity.


On Fri, Nov 4, 2016 at 3:34 PM, Vincent Massol <[hidden email]> wrote:

> Hi devs,
>
> We’ve been developing XWiki over 10 years now. I’d like to start a
> discussion about major architecture changes we may want to do in the coming
> 5 years.
>
> With this discussion I have 2 goals in mind:
> * Prepare XWiki for the challenges ahead (prevent it from being obsolete,
> have an architecture that can adapt to changes, etc)
> * Make XWiki an interesting project to develop on for its developers
> (interesting technologies, interesting intellectual challenges, etc)
>
> Let me start with a few large architecture items I can think about:
>
> * Polyglotism for the front end. This is the ability to code the XWiki UI
> in any language (PHP, Node.js, javascript, native applications, etc). Right
> now XWiki UIs are coded mostly using Velocity (and a bit of javascript). It
> would be nice to be more agnostic and to attract non java developers. The
> core (Server part) could still be in Java but it would be nice that UIs and
> extensions could be developed in any language. One architecture that could
> achieve this would be to have a Core Server exposing all operations as REST
> APIs. This would decouple the Server part from the Client part. It also
> means having all XWiki API modules offering REST APIs and making it extra
> easy to add new REST APIs.
>
> * Greatly simplify development of XWiki Extensions. There are several
> directions that I can think of:
> ** Direction 1: Expose the resources making up an Extension on the file
> system and let users use their favorite tools to write the content (web
> IDEs, text editors, etc)
> ** Direction 2: Develop an IDE in the cloud. Again there are 2 options
> here:
> *** Take ownership of the XWiki Web IDE application  and push it much
> further
> *** Go in the direction of integrating with a well-known cloud IDE such as
> Eclipse Che/CodeEnvy (with pre-built workspaces, docker VMs and one click
> deploy to test your extension, direct deployment of extensions to e.x.o,
> etc).
> ** Other: In general, make it faster to develop extensions. The advantage
> of the Cloud IDE or Web IDE is that it removes the burden of setting up the
> tools on your local machine (maven, java, etc) so someone new to XWiki
> should be operational under 5 minutes to run the first tutorial.
>

Since Extensions are the core of our platform, I agree that simplifying the
development of them would help our users create them. This and a powerful
store that allows findability of this contributed extensions. Regarding
Direction 1 and 2, I don't think they are exclusive. I always liked that
for Skins for example you have the 2 ways to create them: in pages and on
the file system. Each developer has it's preferences or access restrictions
when both ways are needed.


>
> * Promote our SOLR Query Language as the main query language for XWiki and
> deprecate XWQL/HQL. Goal: make querying independent of the stores (right
> now we have a single store implemented on a RDBMS but we can imagine in the
> future moving to another type of store, and even to multiple stores).
> ** Make it easy for extensions to contribute new SOLR indexes for their
> needs.
> ** Once we use only SOLR QL we can then more easily switch to a different
> database model. We would also need the next Generation XAR format to be
> able to fully export an XWiki instance (or some part of it) into a XAR and
> be able to reimport it (right now lots of stuff are not in the XAR such as
> data found in the permanent directory).
>
> * Move towards a container-based approach to install/deploy XWiki (docker,
> etc). The idea being to start being microservices-friendly. Right now we
> already have 2 such services in XWiki: external SOLR + office imports with
> an office server. We need to be able to include those in distributions and
> add more. It should be easy to develop a new microservice and set it up in
> XWiki for redistribution.
> ** It may be interesting to investigate infrastructure frameworks such as
> kubernetes, apache karaf and the like. The goal being to build systems that
> are more responsive, resilient, auto-adaptive, elastic (see the reactive
> manifesto at http://www.reactivemanifesto.org/).
> ** We could imagine to be able to package our office import micro-services
> (ie including an office server) as an extension that can be installed and
> deployed in the XWiki system automatically.
>
> Ok that’s already a good start.
>
> Let me know if you feel excited by some of those and feel free to add
> more. Note that the idea here is to brainstorm about large architecture
> changes.
>

At XWiki we always were ambitious in terms of the things we want to
achieve, even though we are not that many, we always dreamt big. This was
always a blessing and a curse (dramatic music :) ).

Thanks Vincent for this thread,
Caty


>
> Thanks
> -Vincent
>
>
>
>
>
>
>
>
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs