[Brainstomring] Size of XE distribution - What are the next steps?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[Brainstomring] Size of XE distribution - What are the next steps?

vmassol
Administrator
Hi devs,

FYI here’s how the size of our XE distribution has evolved over the years:
https://www.evernote.com/l/AHe8M7Mpc8dLYbLXUUR4BNshyArvRhxM2e4

The sad news is that we’re working to reduce the size of the core for some time already but globally over the years the XE size is still growing.

We started going in the right direction in XE 6.3 and 6.4:

* XE 6.2 = 222MB
* XE 6.3 = 212MB
* XE 6.4 = 210MB

But then starting with XE 7.0 we started to increase the XE size again:

* XE 7.0 = 211MB
* XE 7.1 = 225MB
* XE 7.2 = 226MB
* XE 7.3 = 230MB
* XE 7.4 = 230MB
* XE 8.0 = 229MB
* XE 8.1 = 230MB
* XE 8.2 = 233MB

FTR at some point we had:
* XE 2.0 = 79MB
* XE 3.0 = 111MB
* XE 4.0 = 142MB
* XE 5.0 = 181MB
* XE 6.0 = 211MB

Any idea of what we could do to back to reducing the size of XE?

Some idea:
* Remove the GWT Editor (close to 10MB)
* Introduce a check in the build to fail it when the distribution goes over, say, 0.5% more of a threshold (for 230MB, 1% = 1.15MB) so that when we increase the size we’ll have to look at what is making that size increase and if we think it’s ok then we increase the threshold. The idea would be to make us think about it regularly. WDYT?
* <more idea here>

FTR I was reminded of this by a comment on a blog post: "En rapport avec XWiki, à l'époque où j'ai connu Greenpepper, il y avait un pack de démo avec XWiki. J'ai voulu refaire ce pack, mais j'ai l'impression que XWiki a pris un certain nombre de méga bytes en quelques années ^_^ . Est ce qu'il y aurait une version light disponible ?”

In short the guy said that XWiki seems to have increased size over the years and was asking whether there was a lightweight version somewhere…

WDYT? Do we agree that we should try to contain the size and even start decreasing it if we can?

Thanks
-Vincent


_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Brainstomring] Size of XE distribution - What are the next steps?

Thomas Mortagne
Administrator
Are those jetty/hsqldb or WAR sizes ?

Apart from the size another important issue is the number of external
dependencies we have which among other things makes dependency
handling a pain for extensions.

Here are some numbers from the 8.3M1 WAR:

The WEB-INF/lib takes 172M from which only 12M is pure XWiki jars.

In the WAR some of the biggest features are (size are the feature and
the dependencies used only by that feature):
* Full text search (close to 70M)
** Solr (~30M), including probably totally useless Jetty (yes in the
WAR) stuff (~2M)
** Tika (more than 33M probably not far from 40M, lots of small stuff
I was too lazy to count since Tika has an insane dependency tree full
of stuff used only by it)
* Jython and all its standard libraries (37M): used by code and python macros
* Old GWT Editor (~26M)
* Groovy+Ivy (7M + 1M): not really used in the XE flavor but required
to write a scheduler and many contrib extensions use it and don't
declare it as dependency

Things that would not be too hard to move to Contrib that would also
take several external jars in their luggage (which is also nice for
extensions, the less core extensions the less dependencies issues):
* Office server related stuff (would probably be a good idea anyway to
synchronize it more with LibreOffice than XWiki)
* Chart macro and renderer
* Rendering syntaxes not enabled by default (we already started this
with markdown and mediawiki which are the most used, all the others
are probably never used or very rarely)
* Captcha macro/renderer
* Formula macro/renderer

8.3M1 still includes CKEditor and Tour extensions jars which should
not be there anymore in 8.3 final.

On Sun, Sep 11, 2016 at 1:50 PM, Vincent Massol <[hidden email]> wrote:

> Hi devs,
>
> FYI here’s how the size of our XE distribution has evolved over the years:
> https://www.evernote.com/l/AHe8M7Mpc8dLYbLXUUR4BNshyArvRhxM2e4
>
> The sad news is that we’re working to reduce the size of the core for some time already but globally over the years the XE size is still growing.
>
> We started going in the right direction in XE 6.3 and 6.4:
>
> * XE 6.2 = 222MB
> * XE 6.3 = 212MB
> * XE 6.4 = 210MB
>
> But then starting with XE 7.0 we started to increase the XE size again:
>
> * XE 7.0 = 211MB
> * XE 7.1 = 225MB
> * XE 7.2 = 226MB
> * XE 7.3 = 230MB
> * XE 7.4 = 230MB
> * XE 8.0 = 229MB
> * XE 8.1 = 230MB
> * XE 8.2 = 233MB
>
> FTR at some point we had:
> * XE 2.0 = 79MB
> * XE 3.0 = 111MB
> * XE 4.0 = 142MB
> * XE 5.0 = 181MB
> * XE 6.0 = 211MB
>
> Any idea of what we could do to back to reducing the size of XE?
>
> Some idea:
> * Remove the GWT Editor (close to 10MB)

The total is actually closer to 26M. You have to include things that
are only really needed for the WYSIWYG. gwt-servlet is 9M and WYSIWYG
WAR (which does not include gwt-jar) is 17M.

> * Introduce a check in the build to fail it when the distribution goes over, say, 0.5% more of a threshold (for 230MB, 1% = 1.15MB) so that when we increase the size we’ll have to look at what is making that size increase and if we think it’s ok then we increase the threshold. The idea would be to make us think about it regularly. WDYT?
> * <more idea here>
>
> FTR I was reminded of this by a comment on a blog post: "En rapport avec XWiki, à l'époque où j'ai connu Greenpepper, il y avait un pack de démo avec XWiki. J'ai voulu refaire ce pack, mais j'ai l'impression que XWiki a pris un certain nombre de méga bytes en quelques années ^_^ . Est ce qu'il y aurait une version light disponible ?”
>
> In short the guy said that XWiki seems to have increased size over the years and was asking whether there was a lightweight version somewhere…
>
> WDYT? Do we agree that we should try to contain the size and even start decreasing it if we can?
>
> Thanks
> -Vincent
>
>
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs



--
Thomas Mortagne
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Brainstomring] Size of XE distribution - What are the next steps?

vmassol
Administrator
Hi Thomas,

> On 11 Sep 2016, at 17:16, Thomas Mortagne <[hidden email]> wrote:
>
> Are those jetty/hsqldb or WAR sizes ?

I’ve used http://snippets.xwiki.org/xwiki/bin/view/Extension/Size+of+XE+Distributions which measures the Jetty/HSQLDB packaging.

> Apart from the size another important issue is the number of external
> dependencies we have which among other things makes dependency
> handling a pain for extensions.
>
> Here are some numbers from the 8.3M1 WAR:
>
> The WEB-INF/lib takes 172M from which only 12M is pure XWiki jars.
>
> In the WAR some of the biggest features are (size are the feature and
> the dependencies used only by that feature):
> * Full text search (close to 70M)
> ** Solr (~30M), including probably totally useless Jetty (yes in the
> WAR) stuff (~2M)
> ** Tika (more than 33M probably not far from 40M, lots of small stuff
> I was too lazy to count since Tika has an insane dependency tree full
> of stuff used only by it)

70M for text search, wow, that’s huge indeed. Are there are ways to take only some parts of SOLR (We use solr-core (36MB) and solr-solrj (654K))? Maybe we could ask on their list.

For Tika I see only 2 jars:
* tika-core: 602K
* tika-parsers: 782K

So I see only a total of ~1.5MB. Have I missed some to get 33MB+ you counted?

> * Jython and all its standard libraries (37M): used by code and python macros

Could be good to not use the code macro in the core (I’ve actually always made sure to not use it in the extensions I wrote) and bring the code macro as flavor dependency only. I’ve just checked quickly and there are very few places where we use the code macro in platform. Should be easy to remove those. Then we could move the macro to contrib and bundle it in the XE flavor for now.

> * Old GWT Editor (~26M)
> * Groovy+Ivy (7M + 1M): not really used in the XE flavor but required
> to write a scheduler and many contrib extensions use it and don't
> declare it as dependency

Yep we need to get rid of Groovy in the core. One area is indeed the scheduler extension and we know we need to replace the xproperty using groovy by an xproperty using wiki syntax to allow all script languages to be used. I don’t know of other places requiring absolutely Groovy in the platform. Do you?

> Things that would not be too hard to move to Contrib that would also
> take several external jars in their luggage (which is also nice for
> extensions, the less core extensions the less dependencies issues):
> * Office server related stuff (would probably be a good idea anyway to
> synchronize it more with LibreOffice than XWiki)

Yep +1. Might need some additional extension points defined to integrate cleanly with platform (in create dialog + in WYSIWYG editor).

> * Chart macro and renderer

Yep, I don’t know any platform code using that. I’m going to send a proposal email on this and I could move it.

> * Rendering syntaxes not enabled by default (we already started this
> with markdown and mediawiki which are the most used, all the others
> are probably never used or very rarely)

Agreed, we have a few more to move out.

> * Captcha macro/renderer

Yep

> * Formula macro/renderer

Yep

> 8.3M1 still includes CKEditor and Tour extensions jars which should
> not be there anymore in 8.3 final.

You mean that we should start distributing the platform distribution instead of XE and make XE a flavor (probably under another name, I’ll send a proposal mail about that) that users would pick at runtime, right?

[snip]

Thanks
-Vincent

_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Brainstomring] Size of XE distribution - What are the next steps?

Caleb James DeLisle-3


On 11/09/16 19:21, Vincent Massol wrote:

> Hi Thomas,
>
>> On 11 Sep 2016, at 17:16, Thomas Mortagne <[hidden email]> wrote:
>>
>> Are those jetty/hsqldb or WAR sizes ?
>
> I’ve used http://snippets.xwiki.org/xwiki/bin/view/Extension/Size+of+XE+Distributions which measures the Jetty/HSQLDB packaging.
>
>> Apart from the size another important issue is the number of external
>> dependencies we have which among other things makes dependency
>> handling a pain for extensions.
>>
>> Here are some numbers from the 8.3M1 WAR:
>>
>> The WEB-INF/lib takes 172M from which only 12M is pure XWiki jars.
>>
>> In the WAR some of the biggest features are (size are the feature and
>> the dependencies used only by that feature):
>> * Full text search (close to 70M)
>> ** Solr (~30M), including probably totally useless Jetty (yes in the
>> WAR) stuff (~2M)
>> ** Tika (more than 33M probably not far from 40M, lots of small stuff
>> I was too lazy to count since Tika has an insane dependency tree full
>> of stuff used only by it)
>
> 70M for text search, wow, that’s huge indeed. Are there are ways to take only some parts of SOLR (We use solr-core (36MB) and solr-solrj (654K))? Maybe we could ask on their list.
>
> For Tika I see only 2 jars:
> * tika-core: 602K
> * tika-parsers: 782K
>
> So I see only a total of ~1.5MB. Have I missed some to get 33MB+ you counted?
>
>> * Jython and all its standard libraries (37M): used by code and python macros
>
> Could be good to not use the code macro in the core (I’ve actually always made sure to not use it in the extensions I wrote) and bring the code macro as flavor dependency only. I’ve just checked quickly and there are very few places where we use the code macro in platform. Should be easy to remove those. Then we could move the macro to contrib and bundle it in the XE flavor for now.

+1

37MB for what is usually just {{code}} macro is a bit annoying.

>
>> * Old GWT Editor (~26M)

+1

>> * Groovy+Ivy (7M + 1M): not really used in the XE flavor but required
>> to write a scheduler and many contrib extensions use it and don't
>> declare it as dependency

-0 this is kind of nice to have {{groovy}} always available for debugging and 7M
is a lot by some standards but it's pretty small as a percentage.


Also +1 the others below...

Thanks,
Caleb


>
> Yep we need to get rid of Groovy in the core. One area is indeed the scheduler extension and we know we need to replace the xproperty using groovy by an xproperty using wiki syntax to allow all script languages to be used. I don’t know of other places requiring absolutely Groovy in the platform. Do you?
>
>> Things that would not be too hard to move to Contrib that would also
>> take several external jars in their luggage (which is also nice for
>> extensions, the less core extensions the less dependencies issues):
>> * Office server related stuff (would probably be a good idea anyway to
>> synchronize it more with LibreOffice than XWiki)
>
> Yep +1. Might need some additional extension points defined to integrate cleanly with platform (in create dialog + in WYSIWYG editor).
>
>> * Chart macro and renderer
>
> Yep, I don’t know any platform code using that. I’m going to send a proposal email on this and I could move it.
>
>> * Rendering syntaxes not enabled by default (we already started this
>> with markdown and mediawiki which are the most used, all the others
>> are probably never used or very rarely)
>
> Agreed, we have a few more to move out.
>
>> * Captcha macro/renderer
>
> Yep
>
>> * Formula macro/renderer
>
> Yep
>
>> 8.3M1 still includes CKEditor and Tour extensions jars which should
>> not be there anymore in 8.3 final.
>
> You mean that we should start distributing the platform distribution instead of XE and make XE a flavor (probably under another name, I’ll send a proposal mail about that) that users would pick at runtime, right?
>
> [snip]
>
> Thanks
> -Vincent
>
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Brainstomring] Size of XE distribution - What are the next steps?

vmassol
Administrator

> On 11 Sep 2016, at 19:33, Caleb James DeLisle <[hidden email]> wrote:
>
>
>
> On 11/09/16 19:21, Vincent Massol wrote:
>> Hi Thomas,
>>
>>> On 11 Sep 2016, at 17:16, Thomas Mortagne <[hidden email]> wrote:
>>>
>>> Are those jetty/hsqldb or WAR sizes ?
>>
>> I’ve used http://snippets.xwiki.org/xwiki/bin/view/Extension/Size+of+XE+Distributions which measures the Jetty/HSQLDB packaging.
>>
>>> Apart from the size another important issue is the number of external
>>> dependencies we have which among other things makes dependency
>>> handling a pain for extensions.
>>>
>>> Here are some numbers from the 8.3M1 WAR:
>>>
>>> The WEB-INF/lib takes 172M from which only 12M is pure XWiki jars.
>>>
>>> In the WAR some of the biggest features are (size are the feature and
>>> the dependencies used only by that feature):
>>> * Full text search (close to 70M)
>>> ** Solr (~30M), including probably totally useless Jetty (yes in the
>>> WAR) stuff (~2M)
>>> ** Tika (more than 33M probably not far from 40M, lots of small stuff
>>> I was too lazy to count since Tika has an insane dependency tree full
>>> of stuff used only by it)
>>
>> 70M for text search, wow, that’s huge indeed. Are there are ways to take only some parts of SOLR (We use solr-core (36MB) and solr-solrj (654K))? Maybe we could ask on their list.
>>
>> For Tika I see only 2 jars:
>> * tika-core: 602K
>> * tika-parsers: 782K
>>
>> So I see only a total of ~1.5MB. Have I missed some to get 33MB+ you counted?
>>
>>> * Jython and all its standard libraries (37M): used by code and python macros
>>
>> Could be good to not use the code macro in the core (I’ve actually always made sure to not use it in the extensions I wrote) and bring the code macro as flavor dependency only. I’ve just checked quickly and there are very few places where we use the code macro in platform. Should be easy to remove those. Then we could move the macro to contrib and bundle it in the XE flavor for now.
>
> +1
>
> 37MB for what is usually just {{code}} macro is a bit annoying.
>
>>
>>> * Old GWT Editor (~26M)
>
> +1
>
>>> * Groovy+Ivy (7M + 1M): not really used in the XE flavor but required
>>> to write a scheduler and many contrib extensions use it and don't
>>> declare it as dependency
>
> -0 this is kind of nice to have {{groovy}} always available for debugging and 7M
> is a lot by some standards but it's pretty small as a percentage.

You’d still have it in the “XE” flavor. The idea would be to remove it from the base flavor only, not “XE”, i.e not consider it core (i.e. moved to contrib). Note that a nice side-effect is that it makes it much simpler to ensure that no core module uses it by mistake.

Thanks
-Vincent

> Also +1 the others below...
>
> Thanks,
> Caleb
>
>
>>
>> Yep we need to get rid of Groovy in the core. One area is indeed the scheduler extension and we know we need to replace the xproperty using groovy by an xproperty using wiki syntax to allow all script languages to be used. I don’t know of other places requiring absolutely Groovy in the platform. Do you?
>>
>>> Things that would not be too hard to move to Contrib that would also
>>> take several external jars in their luggage (which is also nice for
>>> extensions, the less core extensions the less dependencies issues):
>>> * Office server related stuff (would probably be a good idea anyway to
>>> synchronize it more with LibreOffice than XWiki)
>>
>> Yep +1. Might need some additional extension points defined to integrate cleanly with platform (in create dialog + in WYSIWYG editor).
>>
>>> * Chart macro and renderer
>>
>> Yep, I don’t know any platform code using that. I’m going to send a proposal email on this and I could move it.
>>
>>> * Rendering syntaxes not enabled by default (we already started this
>>> with markdown and mediawiki which are the most used, all the others
>>> are probably never used or very rarely)
>>
>> Agreed, we have a few more to move out.
>>
>>> * Captcha macro/renderer
>>
>> Yep
>>
>>> * Formula macro/renderer
>>
>> Yep
>>
>>> 8.3M1 still includes CKEditor and Tour extensions jars which should
>>> not be there anymore in 8.3 final.
>>
>> You mean that we should start distributing the platform distribution instead of XE and make XE a flavor (probably under another name, I’ll send a proposal mail about that) that users would pick at runtime, right?
>>
>> [snip]
>>
>> Thanks
>> -Vincent

_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

List of modules to move to cotrib (was Re: [Brainstomring] Size of XE distribution - What are the next steps?)

vmassol
Administrator
In reply to this post by vmassol
On the topic of moving stuff out of core we already have the following 2 threads:
* http://markmail.org/message/viesrhmdavyvdaec
* http://markmail.org/message/y3bkch37mt5iwvxu

To summarize, ATM we’ve discussed about:
* xwiki-platform-blog
* xwiki-platform-release
* xwiki-platform-selenium
* some syntaxes: twiki, apt, confluence*, creole, docbook, doxia, jspwiki, tex
* some macro transformations: linkchecker, wikiword
* some macros like: the ctsreport one

Thanks
-Vincent

> On 11 Sep 2016, at 19:21, Vincent Massol <[hidden email]> wrote:
>
> Hi Thomas,
>
>> On 11 Sep 2016, at 17:16, Thomas Mortagne <[hidden email]> wrote:
>>
>> Are those jetty/hsqldb or WAR sizes ?
>
> I’ve used http://snippets.xwiki.org/xwiki/bin/view/Extension/Size+of+XE+Distributions which measures the Jetty/HSQLDB packaging.
>
>> Apart from the size another important issue is the number of external
>> dependencies we have which among other things makes dependency
>> handling a pain for extensions.
>>
>> Here are some numbers from the 8.3M1 WAR:
>>
>> The WEB-INF/lib takes 172M from which only 12M is pure XWiki jars.
>>
>> In the WAR some of the biggest features are (size are the feature and
>> the dependencies used only by that feature):
>> * Full text search (close to 70M)
>> ** Solr (~30M), including probably totally useless Jetty (yes in the
>> WAR) stuff (~2M)
>> ** Tika (more than 33M probably not far from 40M, lots of small stuff
>> I was too lazy to count since Tika has an insane dependency tree full
>> of stuff used only by it)
>
> 70M for text search, wow, that’s huge indeed. Are there are ways to take only some parts of SOLR (We use solr-core (36MB) and solr-solrj (654K))? Maybe we could ask on their list.
>
> For Tika I see only 2 jars:
> * tika-core: 602K
> * tika-parsers: 782K
>
> So I see only a total of ~1.5MB. Have I missed some to get 33MB+ you counted?
>
>> * Jython and all its standard libraries (37M): used by code and python macros
>
> Could be good to not use the code macro in the core (I’ve actually always made sure to not use it in the extensions I wrote) and bring the code macro as flavor dependency only. I’ve just checked quickly and there are very few places where we use the code macro in platform. Should be easy to remove those. Then we could move the macro to contrib and bundle it in the XE flavor for now.
>
>> * Old GWT Editor (~26M)
>> * Groovy+Ivy (7M + 1M): not really used in the XE flavor but required
>> to write a scheduler and many contrib extensions use it and don't
>> declare it as dependency
>
> Yep we need to get rid of Groovy in the core. One area is indeed the scheduler extension and we know we need to replace the xproperty using groovy by an xproperty using wiki syntax to allow all script languages to be used. I don’t know of other places requiring absolutely Groovy in the platform. Do you?
>
>> Things that would not be too hard to move to Contrib that would also
>> take several external jars in their luggage (which is also nice for
>> extensions, the less core extensions the less dependencies issues):
>> * Office server related stuff (would probably be a good idea anyway to
>> synchronize it more with LibreOffice than XWiki)
>
> Yep +1. Might need some additional extension points defined to integrate cleanly with platform (in create dialog + in WYSIWYG editor).
>
>> * Chart macro and renderer
>
> Yep, I don’t know any platform code using that. I’m going to send a proposal email on this and I could move it.
>
>> * Rendering syntaxes not enabled by default (we already started this
>> with markdown and mediawiki which are the most used, all the others
>> are probably never used or very rarely)
>
> Agreed, we have a few more to move out.
>
>> * Captcha macro/renderer
>
> Yep
>
>> * Formula macro/renderer
>
> Yep
>
>> 8.3M1 still includes CKEditor and Tour extensions jars which should
>> not be there anymore in 8.3 final.
>
> You mean that we should start distributing the platform distribution instead of XE and make XE a flavor (probably under another name, I’ll send a proposal mail about that) that users would pick at runtime, right?
>
> [snip]
>
> Thanks
> -Vincent

_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: [Brainstomring] Size of XE distribution - What are the next steps?

Thomas Mortagne
Administrator
In reply to this post by vmassol
On Sun, Sep 11, 2016 at 7:21 PM, Vincent Massol <[hidden email]> wrote:
> Hi Thomas,
>
>> On 11 Sep 2016, at 17:16, Thomas Mortagne <[hidden email]> wrote:
>>
>> Are those jetty/hsqldb or WAR sizes ?
>
> I’ve used http://snippets.xwiki.org/xwiki/bin/view/Extension/Size+of+XE+Distributions which measures the Jetty/HSQLDB packaging.

WAR size is more interesting because that what real production
actually download and jetty/hsqldb include Jetty size and database
size which are not very relevant IMO.

>
>> Apart from the size another important issue is the number of external
>> dependencies we have which among other things makes dependency
>> handling a pain for extensions.
>>
>> Here are some numbers from the 8.3M1 WAR:
>>
>> The WEB-INF/lib takes 172M from which only 12M is pure XWiki jars.
>>
>> In the WAR some of the biggest features are (size are the feature and
>> the dependencies used only by that feature):
>> * Full text search (close to 70M)
>> ** Solr (~30M), including probably totally useless Jetty (yes in the
>> WAR) stuff (~2M)
>> ** Tika (more than 33M probably not far from 40M, lots of small stuff
>> I was too lazy to count since Tika has an insane dependency tree full
>> of stuff used only by it)
>
> 70M for text search, wow, that’s huge indeed. Are there are ways to take only some parts of SOLR (We use solr-core (36MB) and solr-solrj (654K))? Maybe we could ask on their list.
>
> For Tika I see only 2 jars:
> * tika-core: 602K
> * tika-parsers: 782K
>
> So I see only a total of ~1.5MB. Have I missed some to get 33MB+ you counted?

You missed "and the dependencies used only by that feature" and "Tika
has an insane dependency tree".

>
>> * Jython and all its standard libraries (37M): used by code and python macros
>
> Could be good to not use the code macro in the core (I’ve actually always made sure to not use it in the extensions I wrote) and bring the code macro as flavor dependency only. I’ve just checked quickly and there are very few places where we use the code macro in platform. Should be easy to remove those. Then we could move the macro to contrib and bundle it in the XE flavor for now.
>
>> * Old GWT Editor (~26M)
>> * Groovy+Ivy (7M + 1M): not really used in the XE flavor but required
>> to write a scheduler and many contrib extensions use it and don't
>> declare it as dependency
>
> Yep we need to get rid of Groovy in the core. One area is indeed the scheduler extension and we know we need to replace the xproperty using groovy by an xproperty using wiki syntax to allow all script languages to be used. I don’t know of other places requiring absolutely Groovy in the platform. Do you?

Only parseGroovyFromPage API AFAIK. We got rid of the other big place
that depend on Groovy when we moved xwiki/1.0 rendering framework out
of platform.

>
>> Things that would not be too hard to move to Contrib that would also
>> take several external jars in their luggage (which is also nice for
>> extensions, the less core extensions the less dependencies issues):
>> * Office server related stuff (would probably be a good idea anyway to
>> synchronize it more with LibreOffice than XWiki)
>
> Yep +1. Might need some additional extension points defined to integrate cleanly with platform (in create dialog + in WYSIWYG editor).
>
>> * Chart macro and renderer
>
> Yep, I don’t know any platform code using that. I’m going to send a proposal email on this and I could move it.
>
>> * Rendering syntaxes not enabled by default (we already started this
>> with markdown and mediawiki which are the most used, all the others
>> are probably never used or very rarely)
>
> Agreed, we have a few more to move out.
>
>> * Captcha macro/renderer
>
> Yep
>
>> * Formula macro/renderer
>
> Yep
>
>> 8.3M1 still includes CKEditor and Tour extensions jars which should
>> not be there anymore in 8.3 final.
>
> You mean that we should start distributing the platform distribution instead of XE and make XE a flavor (probably under another name, I’ll send a proposal mail about that) that users would pick at runtime, right?

I'm talking about the WAR here so what I'm saying is that we don't
want them in the WAR already and they are planned to be triggered by
the XE Flavor only. See http://markmail.org/message/mbut2lhatyeu6wzv.

>
> [snip]
>
> Thanks
> -Vincent
>
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs



--
Thomas Mortagne
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs
Reply | Threaded
Open this post in threaded view
|

Re: List of modules to move to cotrib (was Re: [Brainstomring] Size of XE distribution - What are the next steps?)

Ecaterina Moraru (Valica)
In reply to this post by vmassol
+1

Thanks,
Caty

On Sun, Sep 11, 2016 at 10:16 PM, Vincent Massol <[hidden email]> wrote:

> On the topic of moving stuff out of core we already have the following 2
> threads:
> * http://markmail.org/message/viesrhmdavyvdaec
> * http://markmail.org/message/y3bkch37mt5iwvxu
>
> To summarize, ATM we’ve discussed about:
> * xwiki-platform-blog
> * xwiki-platform-release
> * xwiki-platform-selenium
> * some syntaxes: twiki, apt, confluence*, creole, docbook, doxia, jspwiki,
> tex
> * some macro transformations: linkchecker, wikiword
> * some macros like: the ctsreport one
>
> Thanks
> -Vincent
>
> > On 11 Sep 2016, at 19:21, Vincent Massol <[hidden email]> wrote:
> >
> > Hi Thomas,
> >
> >> On 11 Sep 2016, at 17:16, Thomas Mortagne <[hidden email]>
> wrote:
> >>
> >> Are those jetty/hsqldb or WAR sizes ?
> >
> > I’ve used http://snippets.xwiki.org/xwiki/bin/view/Extension/Size+
> of+XE+Distributions which measures the Jetty/HSQLDB packaging.
> >
> >> Apart from the size another important issue is the number of external
> >> dependencies we have which among other things makes dependency
> >> handling a pain for extensions.
> >>
> >> Here are some numbers from the 8.3M1 WAR:
> >>
> >> The WEB-INF/lib takes 172M from which only 12M is pure XWiki jars.
> >>
> >> In the WAR some of the biggest features are (size are the feature and
> >> the dependencies used only by that feature):
> >> * Full text search (close to 70M)
> >> ** Solr (~30M), including probably totally useless Jetty (yes in the
> >> WAR) stuff (~2M)
> >> ** Tika (more than 33M probably not far from 40M, lots of small stuff
> >> I was too lazy to count since Tika has an insane dependency tree full
> >> of stuff used only by it)
> >
> > 70M for text search, wow, that’s huge indeed. Are there are ways to take
> only some parts of SOLR (We use solr-core (36MB) and solr-solrj (654K))?
> Maybe we could ask on their list.
> >
> > For Tika I see only 2 jars:
> > * tika-core: 602K
> > * tika-parsers: 782K
> >
> > So I see only a total of ~1.5MB. Have I missed some to get 33MB+ you
> counted?
> >
> >> * Jython and all its standard libraries (37M): used by code and python
> macros
> >
> > Could be good to not use the code macro in the core (I’ve actually
> always made sure to not use it in the extensions I wrote) and bring the
> code macro as flavor dependency only. I’ve just checked quickly and there
> are very few places where we use the code macro in platform. Should be easy
> to remove those. Then we could move the macro to contrib and bundle it in
> the XE flavor for now.
> >
> >> * Old GWT Editor (~26M)
> >> * Groovy+Ivy (7M + 1M): not really used in the XE flavor but required
> >> to write a scheduler and many contrib extensions use it and don't
> >> declare it as dependency
> >
> > Yep we need to get rid of Groovy in the core. One area is indeed the
> scheduler extension and we know we need to replace the xproperty using
> groovy by an xproperty using wiki syntax to allow all script languages to
> be used. I don’t know of other places requiring absolutely Groovy in the
> platform. Do you?
> >
> >> Things that would not be too hard to move to Contrib that would also
> >> take several external jars in their luggage (which is also nice for
> >> extensions, the less core extensions the less dependencies issues):
> >> * Office server related stuff (would probably be a good idea anyway to
> >> synchronize it more with LibreOffice than XWiki)
> >
> > Yep +1. Might need some additional extension points defined to integrate
> cleanly with platform (in create dialog + in WYSIWYG editor).
> >
> >> * Chart macro and renderer
> >
> > Yep, I don’t know any platform code using that. I’m going to send a
> proposal email on this and I could move it.
> >
> >> * Rendering syntaxes not enabled by default (we already started this
> >> with markdown and mediawiki which are the most used, all the others
> >> are probably never used or very rarely)
> >
> > Agreed, we have a few more to move out.
> >
> >> * Captcha macro/renderer
> >
> > Yep
> >
> >> * Formula macro/renderer
> >
> > Yep
> >
> >> 8.3M1 still includes CKEditor and Tour extensions jars which should
> >> not be there anymore in 8.3 final.
> >
> > You mean that we should start distributing the platform distribution
> instead of XE and make XE a flavor (probably under another name, I’ll send
> a proposal mail about that) that users would pick at runtime, right?
> >
> > [snip]
> >
> > Thanks
> > -Vincent
>
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs