[Proposal] Improve our Test Coverage strategy (TAKE 3)

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Proposal] Improve our Test Coverage strategy (TAKE 3)

vmassol
Administrator
Hi devs,

I remember that we talked about this new global test coverage strategy but I don’t recall where and I couldn’t find an email about it. So I’m reposting the strategy that I’ve just finished implementing.

* Every day the Clover job runs: https://ci.xwiki.org/view/Tools/job/Clover/
* It generates a global coverage report and compares it automatically with the first report generated for the current version
* If the global coverage is reduced, then:
** an email is sent to [hidden email] containing the report
** the job is marked as FAILING
** A failing badge is added to the job history
** A red large text is added to the job page and a link to the report is sent
* Developers MUST fix the global coverage before we can release a version, so the it means the global coverage must not be reduced during a version.
* The RM should check the CI (already part of his chores) and the “Recommended Failing” view on ci.xwiki.org must not have errors or failures (so that includes pitest job and clover job).

Any comments? Does that seem ok to you?

If you confirm it’s ok, I’ll document it on https://dev.xwiki.org/xwiki/bin/view/Community/Testing/TestCoverage/

Note: A few minutes ago an email has been sent since the global coverage has been reduced compared to yesterday.

See http://maven.xwiki.org/site/clover/20190321/XWikiReport-20190320-0128-20190321-1059.html

So we need to fix it now.

Thanks
-Vincent

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Improve our Test Coverage strategy (TAKE 3)

Thomas Mortagne
Administrator
+1

On Thu, Mar 21, 2019 at 5:41 PM Vincent Massol <[hidden email]> wrote:

>
> Hi devs,
>
> I remember that we talked about this new global test coverage strategy but I don’t recall where and I couldn’t find an email about it. So I’m reposting the strategy that I’ve just finished implementing.
>
> * Every day the Clover job runs: https://ci.xwiki.org/view/Tools/job/Clover/
> * It generates a global coverage report and compares it automatically with the first report generated for the current version
> * If the global coverage is reduced, then:
> ** an email is sent to [hidden email] containing the report
> ** the job is marked as FAILING
> ** A failing badge is added to the job history
> ** A red large text is added to the job page and a link to the report is sent
> * Developers MUST fix the global coverage before we can release a version, so the it means the global coverage must not be reduced during a version.
> * The RM should check the CI (already part of his chores) and the “Recommended Failing” view on ci.xwiki.org must not have errors or failures (so that includes pitest job and clover job).
>
> Any comments? Does that seem ok to you?
>
> If you confirm it’s ok, I’ll document it on https://dev.xwiki.org/xwiki/bin/view/Community/Testing/TestCoverage/
>
> Note: A few minutes ago an email has been sent since the global coverage has been reduced compared to yesterday.
>
> See http://maven.xwiki.org/site/clover/20190321/XWikiReport-20190320-0128-20190321-1059.html
>
> So we need to fix it now.
>
> Thanks
> -Vincent
>


--
Thomas Mortagne
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Improve our Test Coverage strategy (TAKE 3)

Marius Dumitru Florea
+1

Thanks,
Marius

On Thu, Mar 21, 2019 at 7:03 PM Thomas Mortagne <[hidden email]>
wrote:

> +1
>
> On Thu, Mar 21, 2019 at 5:41 PM Vincent Massol <[hidden email]> wrote:
> >
> > Hi devs,
> >
> > I remember that we talked about this new global test coverage strategy
> but I don’t recall where and I couldn’t find an email about it. So I’m
> reposting the strategy that I’ve just finished implementing.
> >
> > * Every day the Clover job runs:
> https://ci.xwiki.org/view/Tools/job/Clover/
> > * It generates a global coverage report and compares it automatically
> with the first report generated for the current version
> > * If the global coverage is reduced, then:
> > ** an email is sent to [hidden email] containing the report
> > ** the job is marked as FAILING
> > ** A failing badge is added to the job history
> > ** A red large text is added to the job page and a link to the report is
> sent
> > * Developers MUST fix the global coverage before we can release a
> version, so the it means the global coverage must not be reduced during a
> version.
> > * The RM should check the CI (already part of his chores) and the
> “Recommended Failing” view on ci.xwiki.org must not have errors or
> failures (so that includes pitest job and clover job).
> >
> > Any comments? Does that seem ok to you?
> >
> > If you confirm it’s ok, I’ll document it on
> https://dev.xwiki.org/xwiki/bin/view/Community/Testing/TestCoverage/
> >
> > Note: A few minutes ago an email has been sent since the global coverage
> has been reduced compared to yesterday.
> >
> > See
> http://maven.xwiki.org/site/clover/20190321/XWikiReport-20190320-0128-20190321-1059.html
> >
> > So we need to fix it now.
> >
> > Thanks
> > -Vincent
> >
>
>
> --
> Thomas Mortagne
>