[Proposal] Jobs for the multiple configuration testing

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[Proposal] Jobs for the multiple configuration testing

vmassol
Administrator
Hi devs,

I’m still not sure but FTM I was thinking of having 2 pipeline jobs:

1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.

2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.

Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.

WDYT?

Thanks
-Vincent

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Jobs for the multiple configuration testing

Thomas Mortagne
Administrator
On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol <[hidden email]> wrote:
>
> Hi devs,
>
> I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
>
> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.

That's interesting indeed. +1

>
> 2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.

Can't those configs be executed in parallel in different agents ?

>
> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
>
> WDYT?
>
> Thanks
> -Vincent
>


--
Thomas Mortagne
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Jobs for the multiple configuration testing

Simon Urli


On 07/11/2018 08:44, Thomas Mortagne wrote:

> On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol <[hidden email]> wrote:
>>
>> Hi devs,
>>
>> I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
>>
>> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
>
> That's interesting indeed. +1
>
>>
>> 2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
>
> Can't those configs be executed in parallel in different agents ?
>
>>
>> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
>>
>> WDYT?

+1 for both jobs
>>
>> Thanks
>> -Vincent
>>
>
>

--
Simon Urli
Software Engineer at XWiki SAS
[hidden email]
More about us at http://www.xwiki.com
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Jobs for the multiple configuration testing

vmassol
Administrator
In reply to this post by Thomas Mortagne
Hi Thomas,

> On 7 Nov 2018, at 08:44, Thomas Mortagne <[hidden email]> wrote:
>
> On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol <[hidden email]> wrote:
>>
>> Hi devs,
>>
>> I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
>>
>> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
>
> That's interesting indeed. +1
>
>>
>> 2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
>
> Can't those configs be executed in parallel in different agents ?

Yes this is fully parallelizable. However we have only one agent with docker ATM. Once we migrate more agents then we can have them in parallel.

Thanks
-Vincent

>
>>
>> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
>>
>> WDYT?
>>
>> Thanks
>> -Vincent
>>
>
>
> --
> Thomas Mortagne

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Jobs for the multiple configuration testing

Thomas Mortagne
Administrator
On Wed, Nov 7, 2018 at 9:20 AM Vincent Massol <[hidden email]> wrote:

>
> Hi Thomas,
>
> > On 7 Nov 2018, at 08:44, Thomas Mortagne <[hidden email]> wrote:
> >
> > On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol <[hidden email]> wrote:
> >>
> >> Hi devs,
> >>
> >> I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
> >>
> >> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
> >
> > That's interesting indeed. +1
> >
> >>
> >> 2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
> >
> > Can't those configs be executed in parallel in different agents ?
>
> Yes this is fully parallelizable. However we have only one agent with docker ATM. Once we migrate more agents then we can have them in parallel.

Indeed forgot about that. Anyway +1 for both

>
> Thanks
> -Vincent
>
> >
> >>
> >> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
> >>
> >> WDYT?
> >>
> >> Thanks
> >> -Vincent
> >>
> >
> >
> > --
> > Thomas Mortagne
>


--
Thomas Mortagne
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Jobs for the multiple configuration testing

Adel Atallah
In reply to this post by vmassol
Hello,

I'm +1 to try having both jobs.

Thanks,
Adel

On Wed, Nov 7, 2018 at 9:20 AM Vincent Massol <[hidden email]> wrote:

>
> Hi Thomas,
>
> > On 7 Nov 2018, at 08:44, Thomas Mortagne <[hidden email]> wrote:
> >
> > On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol <[hidden email]> wrote:
> >>
> >> Hi devs,
> >>
> >> I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
> >>
> >> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
> >
> > That's interesting indeed. +1
> >
> >>
> >> 2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
> >
> > Can't those configs be executed in parallel in different agents ?
>
> Yes this is fully parallelizable. However we have only one agent with docker ATM. Once we migrate more agents then we can have them in parallel.
>
> Thanks
> -Vincent
>
> >
> >>
> >> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
> >>
> >> WDYT?
> >>
> >> Thanks
> >> -Vincent
> >>
> >
> >
> > --
> > Thomas Mortagne
>
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Jobs for the multiple configuration testing

Marius Dumitru Florea
In reply to this post by vmassol
+1 for both but we need to push for more powerful CI agents. ATM the build
is faster on my laptop.

Thanks,
Marius

On Mon, Nov 5, 2018 at 6:22 PM Vincent Massol <[hidden email]> wrote:

> Hi devs,
>
> I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
>
> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on
> the maximum number of configurations, in order to flesh out configs that
> don’t start properly. For example XWiki on Tomcat 9.x would fail (since the
> Tomcat 9.x docker image uses java9+). The job would not send a mail on
> failure but it would update a report page (could even update a page on
> xwiki.org directly or if too complex update some page on maven.xwiki.org
> somewhere). This job would run not very often but say once per week. Note
> that one config takes 3-4 minutes to run, so 50 configs would take 3 hours
> which is acceptable.
>
> 2) Job 2: Execute all functional tests on a subset of supported configs.
> For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome
> if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job
> will take a long time to execute. We’ll start with 3-4 configs and will go
> to about 10 configs when we add more. The tests will take roughly 2 hours
> to execute per config I think. So a total of 20 hours when we have 10
> configs. If we run those once per week it should be fine.
>
> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add
> as part of the platform JenkinsFile.
>
> WDYT?
>
> Thanks
> -Vincent
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Jobs for the multiple configuration testing

vmassol
Administrator
In reply to this post by vmassol
Hi devs,

So we now have 2 dockers jobs:
- one that executes daily and runs the latest version of the supported configs. It runs all docker tests (right now that’s MenuIT but I’ll start converting some other tests and we need to convert them all). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker/
- another one that executes weekly and that runs all supported configs but only on smoke tests (right now that MenuIT only). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/

All are passing FTM :)

Thanks
-Vincent

> On 5 Nov 2018, at 17:22, Vincent Massol <[hidden email]> wrote:
>
> Hi devs,
>
> I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
>
> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
>
> 2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
>
> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
>
> WDYT?
>
> Thanks
> -Vincent
>

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Jobs for the multiple configuration testing

vmassol
Administrator


> On 16 Nov 2018, at 22:03, Vincent Massol <[hidden email]> wrote:
>
> Hi devs,
>
> So we now have 2 dockers jobs:
> - one that executes daily and runs the latest version of the supported configs. It runs all docker tests (right now that’s MenuIT but I’ll start converting some other tests and we need to convert them all). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker/
> - another one that executes weekly and that runs all supported configs but only on smoke tests (right now that MenuIT only). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/

And to see which configs are tested and passing or not:
https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/lastCompletedBuild/testReport/org.xwiki.menu.test.ui/MenuIT/

Thanks
-Vincent

> All are passing FTM :)
>
> Thanks
> -Vincent
>
>> On 5 Nov 2018, at 17:22, Vincent Massol <[hidden email]> wrote:
>>
>> Hi devs,
>>
>> I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
>>
>> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
>>
>> 2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
>>
>> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
>>
>> WDYT?
>>
>> Thanks
>> -Vincent
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Jobs for the multiple configuration testing

vmassol
Administrator
Ok I’ve created a 3rd job for unsupported configs:
https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_unsupported/

So we now have:
* Job 1: run all tests on latest versions of supported configs
* Job 2: run smoke tests (ie a few tests only) on all supported configs
* Job 3: run smoke tests on unsupported configs (MySQL 8.x, Tomcat 9.x, etc)

Definitions are all here:
https://github.com/xwiki/xwiki-jenkins-pipeline/blob/06b679bf511261dbd0125a352a5f9074746aff83/src/org/xwiki/jenkins/DockerTests.groovy

Thanks
-Vincent


> On 18 Nov 2018, at 16:57, Vincent Massol <[hidden email]> wrote:
>
>
>
>> On 16 Nov 2018, at 22:03, Vincent Massol <[hidden email]> wrote:
>>
>> Hi devs,
>>
>> So we now have 2 dockers jobs:
>> - one that executes daily and runs the latest version of the supported configs. It runs all docker tests (right now that’s MenuIT but I’ll start converting some other tests and we need to convert them all). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker/
>> - another one that executes weekly and that runs all supported configs but only on smoke tests (right now that MenuIT only). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/
>
> And to see which configs are tested and passing or not:
> https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/lastCompletedBuild/testReport/org.xwiki.menu.test.ui/MenuIT/
>
> Thanks
> -Vincent
>
>> All are passing FTM :)
>>
>> Thanks
>> -Vincent
>>
>>> On 5 Nov 2018, at 17:22, Vincent Massol <[hidden email]> wrote:
>>>
>>> Hi devs,
>>>
>>> I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
>>>
>>> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
>>>
>>> 2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
>>>
>>> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
>>>
>>> WDYT?
>>>
>>> Thanks
>>> -Vincent
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Jobs for the multiple configuration testing

vmassol
Administrator
Self reminder:
Just realized that I now need to modify the clover pipeline so that we now run docker-based tests too! This means the clover job can only run on agent4. This also means that I’ll need to setup FF on agent4 for non-docker-based functional tests.

Thanks
-Vincent

> On 18 Nov 2018, at 17:49, Vincent Massol <[hidden email]> wrote:
>
> Ok I’ve created a 3rd job for unsupported configs:
> https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_unsupported/
>
> So we now have:
> * Job 1: run all tests on latest versions of supported configs
> * Job 2: run smoke tests (ie a few tests only) on all supported configs
> * Job 3: run smoke tests on unsupported configs (MySQL 8.x, Tomcat 9.x, etc)
>
> Definitions are all here:
> https://github.com/xwiki/xwiki-jenkins-pipeline/blob/06b679bf511261dbd0125a352a5f9074746aff83/src/org/xwiki/jenkins/DockerTests.groovy
>
> Thanks
> -Vincent
>
>
>> On 18 Nov 2018, at 16:57, Vincent Massol <[hidden email]> wrote:
>>
>>
>>
>>> On 16 Nov 2018, at 22:03, Vincent Massol <[hidden email]> wrote:
>>>
>>> Hi devs,
>>>
>>> So we now have 2 dockers jobs:
>>> - one that executes daily and runs the latest version of the supported configs. It runs all docker tests (right now that’s MenuIT but I’ll start converting some other tests and we need to convert them all). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker/
>>> - another one that executes weekly and that runs all supported configs but only on smoke tests (right now that MenuIT only). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/
>>
>> And to see which configs are tested and passing or not:
>> https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/lastCompletedBuild/testReport/org.xwiki.menu.test.ui/MenuIT/
>>
>> Thanks
>> -Vincent
>>
>>> All are passing FTM :)
>>>
>>> Thanks
>>> -Vincent
>>>
>>>> On 5 Nov 2018, at 17:22, Vincent Massol <[hidden email]> wrote:
>>>>
>>>> Hi devs,
>>>>
>>>> I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
>>>>
>>>> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
>>>>
>>>> 2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
>>>>
>>>> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
>>>>
>>>> WDYT?
>>>>
>>>> Thanks
>>>> -Vincent
>>>>
>>>
>>
>