[Proposal] Improve our Test Coverage strategy

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[Proposal] Improve our Test Coverage strategy

vmassol
Administrator
Hi devs,

Our current Test coverage strategy is to fail the build when new code added results in a coverage lower than the threshold for the module, using jacoco.

This has 2 limitations causing our global TPC to go down from time to time (see https://markmail.org/message/hqumkdiz7jm76ya6 ).

Thus I’d like to propose the following addition to our strategy:

* We already have a jenkins pipeline to automatically compute the full TPC using Clover. See http://dev.xwiki.org/xwiki/bin/view/Community/Testing#HUsingClover2BJenkins
* Make it run more often (it’s currently executed once per month, see http://ci.xwiki.org/view/Tools/job/Clover/). Takes about 5-6 hours to execute. Thus we could run it once per week or even once per day during the night.
* Add some groovy logic in the pipeline to perform an analysis after the Clover report has been generated. Perform 2 checks by comparing the previous report with the one that just executed:
** Find new packages introduced that have a TPC < the average computed TPC
** Find packages and/or files having a TPC lower than the previous TPC
** Find removed packages that had a TPC higher than the average computed TPC
* Save a report in the directory for the Clover report at http://ci.xwiki.org/view/Tools/job/Clover/
* For all failures, send an email to [hidden email] with details and a link to the saved report
* Ideally, and if we can do it, call the github API to find the authors of commits for those packages and add them in the report. Examples of APIs we could use:
** https://api.github.com/repos/xwiki/xwiki-platform/commits?since=2018-05-07T00:00:00Z&until=2018-05-10T00:00:00Z (there’s a path parameter that could be used to filter but I don’t think it’ll work
** https://github.com/xwiki/xwiki-platform/compare/master@%7B2018-05-07%7D...master@%7B2018-05-10%7D (the committers/authors need to be extracted from the HTML which is a bit fragile)
* Add a step in the Release process to ensure that the global TPC has not been lowered. This would be a way to ensure we pay attention to that and fix it when we lower it. We would need to tune this to find something that helps keep the TPC increasing while not putting too much pressure at the same time on the release date. It doesn’t have to be in the release process but we need some checkpoint to make sure we look at it and that all devs fix the tests when they lower the global TPC, or at the very least that an analysis is done in case where it’s hard to keep the TPC (for example, removing existing code that had a lot of tests will lower the TPC ;)).

WDYT?

As a dev, would you be willing to pay attention to not lower the global TPC and work to fix it when it happens?

Thanks
-Vincent

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Improve our Test Coverage strategy

Thomas Mortagne
Administrator
On Thu, May 10, 2018 at 9:03 PM, Vincent Massol <[hidden email]> wrote:

> Hi devs,
>
> Our current Test coverage strategy is to fail the build when new code added results in a coverage lower than the threshold for the module, using jacoco.
>
> This has 2 limitations causing our global TPC to go down from time to time (see https://markmail.org/message/hqumkdiz7jm76ya6 ).
>
> Thus I’d like to propose the following addition to our strategy:
>
> * We already have a jenkins pipeline to automatically compute the full TPC using Clover. See http://dev.xwiki.org/xwiki/bin/view/Community/Testing#HUsingClover2BJenkins
> * Make it run more often (it’s currently executed once per month, see http://ci.xwiki.org/view/Tools/job/Clover/). Takes abll out 5-6 hours to execute. Thus we could run it once per week or even once per day during the night.
> * Add some groovy logic in the pipeline to perform an analysis after the Clover report has been generated. Perform 2 checks by comparing the previous report with the one that just executed:
> ** Find new packages introduced that have a TPC < the average computed TPC
> ** Find packages and/or files having a TPC lower than the previous TPC
> ** Find removed packages that had a TPC higher than the average computed TPC

"comparing the previous report" won't work if we run this every days,
it means you just have to wait 24h and the current report will be all
green :)

It should be compared to a specific report (or even several) like
comparing to the current LTS branch report and to the report produced
after the previous master release.

> * Save a report in the directory for the Clover report at http://ci.xwiki.org/view/Tools/job/Clover/
> * For all failures, send an email to [hidden email] with details and a link to the saved report

Will we also see a build in error in the ci.xwiki.org home page ? Not
sure a [hidden email] mail only would be enough :)

> * Ideally, and if we can do it, call the github API to find the authors of commits for those packages and add them in the report. Examples of APIs we could use:
> ** https://api.github.com/repos/xwiki/xwiki-platform/commits?since=2018-05-07T00:00:00Z&until=2018-05-10T00:00:00Z (there’s a path parameter that could be used to filter but I don’t think it’ll work
> ** https://github.com/xwiki/xwiki-platform/compare/master@%7B2018-05-07%7D...master@%7B2018-05-10%7D (the committers/authors need to be extracted from the HTML which is a bit fragile)
> * Add a step in th e Release process to ensure that the global TPC has not been lowered. This would be a way to ensure we pay attention to that and fix it when we lower it. We would need to tune this to find something that helps keep the TPC increasing while not putting too much pressure at the same time on the release date. It doesn’t have to be in the release process but we need some checkpoint to make sure we look at it and that all devs fix the tests when they lower the global TPC, or at the very least that an analysis is done in case where it’s hard to keep the TPC (for example, removing existing code that had a lot of tests will lower the TPC ;)).
>
> WDYT?
>
> As a dev, would you be willing to pay attention to not lower the global TPC and work to fix it when it happens?
>
> Thanks
> -Vincent
>

Sounds OK.

--
Thomas Mortagne
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Improve our Test Coverage strategy

vmassol
Administrator


> On 11 May 2018, at 15:38, Thomas Mortagne <[hidden email]> wrote:
>
> On Thu, May 10, 2018 at 9:03 PM, Vincent Massol <[hidden email]> wrote:
>> Hi devs,
>>
>> Our current Test coverage strategy is to fail the build when new code added results in a coverage lower than the threshold for the module, using jacoco.
>>
>> This has 2 limitations causing our global TPC to go down from time to time (see https://markmail.org/message/hqumkdiz7jm76ya6 ).
>>
>> Thus I’d like to propose the following addition to our strategy:
>>
>> * We already have a jenkins pipeline to automatically compute the full TPC using Clover. See http://dev.xwiki.org/xwiki/bin/view/Community/Testing#HUsingClover2BJenkins
>> * Make it run more often (it’s currently executed once per month, see http://ci.xwiki.org/view/Tools/job/Clover/). Takes abll out 5-6 hours to execute. Thus we could run it once per week or even once per day during the night.
>> * Add some groovy logic in the pipeline to perform an analysis after the Clover report has been generated. Perform 2 checks by comparing the previous report with the one that just executed:
>> ** Find new packages introduced that have a TPC < the average computed TPC
>> ** Find packages and/or files having a TPC lower than the previous TPC
>> ** Find removed packages that had a TPC higher than the average computed TPC
>
> "comparing the previous report" won't work if we run this every days,
> it means you just have to wait 24h and the current report will be all
> green :)
>
> It should be compared to a specific report (or even several) like
> comparing to the current LTS branch report and to the report produced
> after the previous master release.

Yes I thought about that too. It actually works but you need to handle past reports and not discard them, which I agree can be a problem especially if you want to have a failing job (see below).

I thought about having a file that we generate in the pipeline script and put in http://maven.xwiki.org/site/clover/ which will contain the directory name for the last good Clover report (i.e the last report having a global TPC higher than the previous one). It’s not perfect either but maybe good enough to start with. The problem is not reacting fast enough and accumulating too many changes and keeping a failing TPC check forever.

>
>> * Save a report in the directory for the Clover report at http://ci.xwiki.org/view/Tools/job/Clover/
>> * For all failures, send an email to [hidden email] with details and a link to the saved report
>
> Will we also see a build in error in the ci.xwiki.org home page ? Not
> sure a [hidden email] mail only would be enough :)

Yes I agree. It’s easy to make the Clover job fail though so we can do that. Now ATM I don’t think the Clover appears in the “Recommended Failing” view so we would just need to add it and we’ll be good.

>
>> * Ideally, and if we can do it, call the github API to find the authors of commits for those packages and add them in the report. Examples of APIs we could use:
>> ** https://api.github.com/repos/xwiki/xwiki-platform/commits?since=2018-05-07T00:00:00Z&until=2018-05-10T00:00:00Z (there’s a path parameter that could be used to filter but I don’t think it’ll work
>> ** https://github.com/xwiki/xwiki-platform/compare/master@%7B2018-05-07%7D...master@%7B2018-05-10%7D (the committers/authors need to be extracted from the HTML which is a bit fragile)
>> * Add a step in th e Release process to ensure that the global TPC has not been lowered. This would be a way to ensure we pay attention to that and fix it when we lower it. We would need to tune this to find something that helps keep the TPC increasing while not putting too much pressure at the same time on the release date. It doesn’t have to be in the release process but we need some checkpoint to make sure we look at it and that all devs fix the tests when they lower the global TPC, or at the very least that an analysis is done in case where it’s hard to keep the TPC (for example, removing existing code that had a lot of tests will lower the TPC ;)).
>>
>> WDYT?
>>
>> As a dev, would you be willing to pay attention to not lower the global TPC and work to fix it when it happens?
>>
>> Thanks
>> -Vincent
>>
>
> Sounds OK.

Cool.

What about other devs? :)

Thanks
-Vincent
>
> --
> Thomas Mortagne

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Improve our Test Coverage strategy

Marius Dumitru Florea
In reply to this post by vmassol
On Thu, May 10, 2018 at 10:03 PM, Vincent Massol <[hidden email]> wrote:

> Hi devs,
>
> Our current Test coverage strategy is to fail the build when new code
> added results in a coverage lower than the threshold for the module, using
> jacoco.
>
> This has 2 limitations causing our global TPC to go down from time to time
> (see https://markmail.org/message/hqumkdiz7jm76ya6 ).
>
> Thus I’d like to propose the following addition to our strategy:
>
> * We already have a jenkins pipeline to automatically compute the full TPC
> using Clover. See http://dev.xwiki.org/xwiki/bin/view/Community/Testing#
> HUsingClover2BJenkins
> * Make it run more often (it’s currently executed once per month, see
> http://ci.xwiki.org/view/Tools/job/Clover/). Takes about 5-6 hours to
> execute. Thus we could run it once per week or even once per day during the
> night.
> * Add some groovy logic in the pipeline to perform an analysis after the
> Clover report has been generated. Perform 2 checks by comparing the
> previous report with the one that just executed:
>


> ** Find new packages introduced that have a TPC < the average computed TPC
>

What is the average computed TPC currently?


> ** Find packages and/or files having a TPC lower than the previous TPC
> ** Find removed packages that had a TPC higher than the average computed
> TPC
>

I would add:

** find the packages that have the TPC higher than what is declared in the
pom (because we don't always update the TPC value declared in the pom when
we refactor the code or when we add new tests)


> * Save a report in the directory for the Clover report at
> http://ci.xwiki.org/view/Tools/job/Clover/
> * For all failures, send an email to [hidden email] with details
> and a link to the saved report
> * Ideally, and if we can do it, call the github API to find the authors of
> commits for those packages and add them in the report. Examples of APIs we
> could use:
> ** https://api.github.com/repos/xwiki/xwiki-platform/commits?
> since=2018-05-07T00:00:00Z&until=2018-05-10T00:00:00Z (there’s a path
> parameter that could be used to filter but I don’t think it’ll work
> ** https://github.com/xwiki/xwiki-platform/compare/master@
> %7B2018-05-07%7D...master@%7B2018-05-10%7D (the committers/authors need
> to be extracted from the HTML which is a bit fragile)
> * Add a step in the Release process to ensure that the global TPC has not
> been lowered. This would be a way to ensure we pay attention to that and
> fix it when we lower it. We would need to tune this to find something that
> helps keep the TPC increasing while not putting too much pressure at the
> same time on the release date. It doesn’t have to be in the release process
> but we need some checkpoint to make sure we look at it and that all devs
> fix the tests when they lower the global TPC, or at the very least that an
> analysis is done in case where it’s hard to keep the TPC (for example,
> removing existing code that had a lot of tests will lower the TPC ;)).
>
> WDYT?
>
> As a dev, would you be willing to pay attention to not lower the global
> TPC and work to fix it when it happens?
>
> Thanks
> -Vincent
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Improve our Test Coverage strategy

vmassol
Administrator


> On 14 May 2018, at 12:28, Marius Dumitru Florea <[hidden email]> wrote:
>
> On Thu, May 10, 2018 at 10:03 PM, Vincent Massol <[hidden email]> wrote:
>
>> Hi devs,
>>
>> Our current Test coverage strategy is to fail the build when new code
>> added results in a coverage lower than the threshold for the module, using
>> jacoco.
>>
>> This has 2 limitations causing our global TPC to go down from time to time
>> (see https://markmail.org/message/hqumkdiz7jm76ya6 ).
>>
>> Thus I’d like to propose the following addition to our strategy:
>>
>> * We already have a jenkins pipeline to automatically compute the full TPC
>> using Clover. See http://dev.xwiki.org/xwiki/bin/view/Community/Testing#
>> HUsingClover2BJenkins
>> * Make it run more often (it’s currently executed once per month, see
>> http://ci.xwiki.org/view/Tools/job/Clover/). Takes about 5-6 hours to
>> execute. Thus we could run it once per week or even once per day during the
>> night.
>> * Add some groovy logic in the pipeline to perform an analysis after the
>> Clover report has been generated. Perform 2 checks by comparing the
>> previous report with the one that just executed:
>>
>
>
>> ** Find new packages introduced that have a TPC < the average computed TPC
>>
>
> What is the average computed TPC currently?

It’s about 70%, see for example:
http://maven.xwiki.org/site/clover/20180511/clover-commons+rendering+platform-20180511-0147/dashboard.html

>
>
>> ** Find packages and/or files having a TPC lower than the previous TPC
>> ** Find removed packages that had a TPC higher than the average computed
>> TPC
>>
>
> I would add:
>
> ** find the packages that have the TPC higher than what is declared in the
> pom (because we don't always update the TPC value declared in the pom when
> we refactor the code or when we add new tests)

Yes I agree. We should do that but not in the pipeline for the global coverage. We should have another pipeline for this and update the pom.xml files in it.

>
>> * Save a report in the directory for the Clover report at
>> http://ci.xwiki.org/view/Tools/job/Clover/
>> * For all failures, send an email to [hidden email] with details
>> and a link to the saved report
>> * Ideally, and if we can do it, call the github API to find the authors of
>> commits for those packages and add them in the report. Examples of APIs we
>> could use:
>> ** https://api.github.com/repos/xwiki/xwiki-platform/commits?
>> since=2018-05-07T00:00:00Z&until=2018-05-10T00:00:00Z (there’s a path
>> parameter that could be used to filter but I don’t think it’ll work
>> ** https://github.com/xwiki/xwiki-platform/compare/master@
>> %7B2018-05-07%7D...master@%7B2018-05-10%7D (the committers/authors need
>> to be extracted from the HTML which is a bit fragile)
>> * Add a step in the Release process to ensure that the global TPC has not
>> been lowered. This would be a way to ensure we pay attention to that and
>> fix it when we lower it. We would need to tune this to find something that
>> helps keep the TPC increasing while not putting too much pressure at the
>> same time on the release date. It doesn’t have to be in the release process
>> but we need some checkpoint to make sure we look at it and that all devs
>> fix the tests when they lower the global TPC, or at the very least that an
>> analysis is done in case where it’s hard to keep the TPC (for example,
>> removing existing code that had a lot of tests will lower the TPC ;)).
>>
>> WDYT?
>>
>> As a dev, would you be willing to pay attention to not lower the global
>> TPC and work to fix it when it happens?

@Marius: what about this question? Would you be ok with it? :)

Thanks
-Vincent

>>
>> Thanks
>> -Vincent

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Improve our Test Coverage strategy

Marius Dumitru Florea
On Mon, May 14, 2018 at 2:17 PM, Vincent Massol <[hidden email]> wrote:

>
>
> > On 14 May 2018, at 12:28, Marius Dumitru Florea <
> [hidden email]> wrote:
> >
> > On Thu, May 10, 2018 at 10:03 PM, Vincent Massol <[hidden email]>
> wrote:
> >
> >> Hi devs,
> >>
> >> Our current Test coverage strategy is to fail the build when new code
> >> added results in a coverage lower than the threshold for the module,
> using
> >> jacoco.
> >>
> >> This has 2 limitations causing our global TPC to go down from time to
> time
> >> (see https://markmail.org/message/hqumkdiz7jm76ya6 ).
> >>
> >> Thus I’d like to propose the following addition to our strategy:
> >>
> >> * We already have a jenkins pipeline to automatically compute the full
> TPC
> >> using Clover. See http://dev.xwiki.org/xwiki/
> bin/view/Community/Testing#
> >> HUsingClover2BJenkins
> >> * Make it run more often (it’s currently executed once per month, see
> >> http://ci.xwiki.org/view/Tools/job/Clover/). Takes about 5-6 hours to
> >> execute. Thus we could run it once per week or even once per day during
> the
> >> night.
> >> * Add some groovy logic in the pipeline to perform an analysis after the
> >> Clover report has been generated. Perform 2 checks by comparing the
> >> previous report with the one that just executed:
> >>
> >
> >
> >> ** Find new packages introduced that have a TPC < the average computed
> TPC
> >>
> >
> > What is the average computed TPC currently?
>
> It’s about 70%, see for example:
> http://maven.xwiki.org/site/clover/20180511/clover-
> commons+rendering+platform-20180511-0147/dashboard.html
>
>
Requiring 70% TPC for new packages is a nice goal but I find it hard to
achieve in practice.


> >
> >
> >> ** Find packages and/or files having a TPC lower than the previous TPC
> >> ** Find removed packages that had a TPC higher than the average computed
> >> TPC
> >>
> >
> > I would add:
> >
> > ** find the packages that have the TPC higher than what is declared in
> the
> > pom (because we don't always update the TPC value declared in the pom
> when
> > we refactor the code or when we add new tests)
>
> Yes I agree. We should do that but not in the pipeline for the global
> coverage. We should have another pipeline for this and update the pom.xml
> files in it.
>
> >
> >> * Save a report in the directory for the Clover report at
> >> http://ci.xwiki.org/view/Tools/job/Clover/
> >> * For all failures, send an email to [hidden email] with
> details
> >> and a link to the saved report
> >> * Ideally, and if we can do it, call the github API to find the authors
> of
> >> commits for those packages and add them in the report. Examples of APIs
> we
> >> could use:
> >> ** https://api.github.com/repos/xwiki/xwiki-platform/commits?
> >> since=2018-05-07T00:00:00Z&until=2018-05-10T00:00:00Z (there’s a path
> >> parameter that could be used to filter but I don’t think it’ll work
> >> ** https://github.com/xwiki/xwiki-platform/compare/master@
> >> %7B2018-05-07%7D...master@%7B2018-05-10%7D (the committers/authors need
> >> to be extracted from the HTML which is a bit fragile)
> >> * Add a step in the Release process to ensure that the global TPC has
> not
> >> been lowered. This would be a way to ensure we pay attention to that and
> >> fix it when we lower it. We would need to tune this to find something
> that
> >> helps keep the TPC increasing while not putting too much pressure at the
> >> same time on the release date. It doesn’t have to be in the release
> process
> >> but we need some checkpoint to make sure we look at it and that all devs
> >> fix the tests when they lower the global TPC, or at the very least that
> an
> >> analysis is done in case where it’s hard to keep the TPC (for example,
> >> removing existing code that had a lot of tests will lower the TPC ;)).
> >>
> >> WDYT?
> >>
> >> As a dev, would you be willing to pay attention to not lower the global
> >> TPC and work to fix it when it happens?
>
> @Marius: what about this question? Would you be ok with it? :)
>

Sure.


>
> Thanks
> -Vincent
>
> >>
> >> Thanks
> >> -Vincent
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Improve our Test Coverage strategy

vmassol
Administrator


> On 14 May 2018, at 13:30, Marius Dumitru Florea <[hidden email]> wrote:
>
> On Mon, May 14, 2018 at 2:17 PM, Vincent Massol <[hidden email]> wrote:
>
>>
>>
>>> On 14 May 2018, at 12:28, Marius Dumitru Florea <
>> [hidden email]> wrote:
>>>
>>> On Thu, May 10, 2018 at 10:03 PM, Vincent Massol <[hidden email]>
>> wrote:
>>>
>>>> Hi devs,
>>>>
>>>> Our current Test coverage strategy is to fail the build when new code
>>>> added results in a coverage lower than the threshold for the module,
>> using
>>>> jacoco.
>>>>
>>>> This has 2 limitations causing our global TPC to go down from time to
>> time
>>>> (see https://markmail.org/message/hqumkdiz7jm76ya6 ).
>>>>
>>>> Thus I’d like to propose the following addition to our strategy:
>>>>
>>>> * We already have a jenkins pipeline to automatically compute the full
>> TPC
>>>> using Clover. See http://dev.xwiki.org/xwiki/
>> bin/view/Community/Testing#
>>>> HUsingClover2BJenkins
>>>> * Make it run more often (it’s currently executed once per month, see
>>>> http://ci.xwiki.org/view/Tools/job/Clover/). Takes about 5-6 hours to
>>>> execute. Thus we could run it once per week or even once per day during
>> the
>>>> night.
>>>> * Add some groovy logic in the pipeline to perform an analysis after the
>>>> Clover report has been generated. Perform 2 checks by comparing the
>>>> previous report with the one that just executed:
>>>>
>>>
>>>
>>>> ** Find new packages introduced that have a TPC < the average computed
>> TPC
>>>>
>>>
>>> What is the average computed TPC currently?
>>
>> It’s about 70%, see for example:
>> http://maven.xwiki.org/site/clover/20180511/clover-
>> commons+rendering+platform-20180511-0147/dashboard.html
>>
>>
> Requiring 70% TPC for new packages is a nice goal but I find it hard to
> achieve in practice.

Two points on this:
* We should have above 80-90%+ in practice for new modules, not 70%. If we have 70% or less we can be sure you did a bad job on quality and that you’re impacting the overall quality of XWiki.
* For some modules, it makes less sense or it’s harder to have unit tests and integration tests are better. We’re not talking about 70% of unit test coverage but 70%+ coverage of overall testing (unit, integration, functional).

Now we’ll need to put in place to see it in action and verify if there are cases where this can be hard to achieve. But I’d prefer that we consider those cases as exceptional and handle them in an ad-hoc manner.

>
>>>
>>>> ** Find packages and/or files having a TPC lower than the previous TPC
>>>> ** Find removed packages that had a TPC higher than the average computed
>>>> TPC
>>>>
>>>
>>> I would add:
>>>
>>> ** find the packages that have the TPC higher than what is declared in
>> the
>>> pom (because we don't always update the TPC value declared in the pom
>> when
>>> we refactor the code or when we add new tests)
>>
>> Yes I agree. We should do that but not in the pipeline for the global
>> coverage. We should have another pipeline for this and update the pom.xml
>> files in it.
>>
>>>
>>>> * Save a report in the directory for the Clover report at
>>>> http://ci.xwiki.org/view/Tools/job/Clover/
>>>> * For all failures, send an email to [hidden email] with
>> details
>>>> and a link to the saved report
>>>> * Ideally, and if we can do it, call the github API to find the authors
>> of
>>>> commits for those packages and add them in the report. Examples of APIs
>> we
>>>> could use:
>>>> ** https://api.github.com/repos/xwiki/xwiki-platform/commits?
>>>> since=2018-05-07T00:00:00Z&until=2018-05-10T00:00:00Z (there’s a path
>>>> parameter that could be used to filter but I don’t think it’ll work
>>>> ** https://github.com/xwiki/xwiki-platform/compare/master@
>>>> %7B2018-05-07%7D...master@%7B2018-05-10%7D (the committers/authors need
>>>> to be extracted from the HTML which is a bit fragile)
>>>> * Add a step in the Release process to ensure that the global TPC has
>> not
>>>> been lowered. This would be a way to ensure we pay attention to that and
>>>> fix it when we lower it. We would need to tune this to find something
>> that
>>>> helps keep the TPC increasing while not putting too much pressure at the
>>>> same time on the release date. It doesn’t have to be in the release
>> process
>>>> but we need some checkpoint to make sure we look at it and that all devs
>>>> fix the tests when they lower the global TPC, or at the very least that
>> an
>>>> analysis is done in case where it’s hard to keep the TPC (for example,
>>>> removing existing code that had a lot of tests will lower the TPC ;)).
>>>>
>>>> WDYT?
>>>>
>>>> As a dev, would you be willing to pay attention to not lower the global
>>>> TPC and work to fix it when it happens?
>>
>> @Marius: what about this question? Would you be ok with it? :)
>>
>
> Sure.

ok cool

Thanks
-Vincent

>
>
>>
>> Thanks
>> -Vincent
>>
>>>>
>>>> Thanks
>>>> -Vincent

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Improve our Test Coverage strategy

Marius Dumitru Florea
On Mon, May 14, 2018 at 2:41 PM, Vincent Massol <[hidden email]> wrote:

>
>
> > On 14 May 2018, at 13:30, Marius Dumitru Florea <
> [hidden email]> wrote:
> >
> > On Mon, May 14, 2018 at 2:17 PM, Vincent Massol <[hidden email]>
> wrote:
> >
> >>
> >>
> >>> On 14 May 2018, at 12:28, Marius Dumitru Florea <
> >> [hidden email]> wrote:
> >>>
> >>> On Thu, May 10, 2018 at 10:03 PM, Vincent Massol <[hidden email]>
> >> wrote:
> >>>
> >>>> Hi devs,
> >>>>
> >>>> Our current Test coverage strategy is to fail the build when new code
> >>>> added results in a coverage lower than the threshold for the module,
> >> using
> >>>> jacoco.
> >>>>
> >>>> This has 2 limitations causing our global TPC to go down from time to
> >> time
> >>>> (see https://markmail.org/message/hqumkdiz7jm76ya6 ).
> >>>>
> >>>> Thus I’d like to propose the following addition to our strategy:
> >>>>
> >>>> * We already have a jenkins pipeline to automatically compute the full
> >> TPC
> >>>> using Clover. See http://dev.xwiki.org/xwiki/
> >> bin/view/Community/Testing#
> >>>> HUsingClover2BJenkins
> >>>> * Make it run more often (it’s currently executed once per month, see
> >>>> http://ci.xwiki.org/view/Tools/job/Clover/). Takes about 5-6 hours to
> >>>> execute. Thus we could run it once per week or even once per day
> during
> >> the
> >>>> night.
> >>>> * Add some groovy logic in the pipeline to perform an analysis after
> the
> >>>> Clover report has been generated. Perform 2 checks by comparing the
> >>>> previous report with the one that just executed:
> >>>>
> >>>
> >>>
> >>>> ** Find new packages introduced that have a TPC < the average computed
> >> TPC
> >>>>
> >>>
> >>> What is the average computed TPC currently?
> >>
> >> It’s about 70%, see for example:
> >> http://maven.xwiki.org/site/clover/20180511/clover-
> >> commons+rendering+platform-20180511-0147/dashboard.html
> >>
> >>
> > Requiring 70% TPC for new packages is a nice goal but I find it hard to
> > achieve in practice.
>
> Two points on this:
>


> * We should have above 80-90%+ in practice for new modules, not 70%. If we
> have 70% or less we can be sure you did a bad job on quality and that
> you’re impacting the overall quality of XWiki.
>

*should* but I don't think it happens in practice, for various reasons,
mainly due to limited time. Did you check the TPC of the last 10 new
packages? Is it above 70%?


> * For some modules, it makes less sense or it’s harder to have unit tests
> and integration tests are better. We’re not talking about 70% of unit test
> coverage but 70%+ coverage of overall testing (unit, integration,
> functional).
>
> Now we’ll need to put in place to see it in action and verify if there are
> cases where this can be hard to achieve. But I’d prefer that we consider
> those cases as exceptional and handle them in an ad-hoc manner.
>
> >
> >>>
> >>>> ** Find packages and/or files having a TPC lower than the previous TPC
> >>>> ** Find removed packages that had a TPC higher than the average
> computed
> >>>> TPC
> >>>>
> >>>
> >>> I would add:
> >>>
> >>> ** find the packages that have the TPC higher than what is declared in
> >> the
> >>> pom (because we don't always update the TPC value declared in the pom
> >> when
> >>> we refactor the code or when we add new tests)
> >>
> >> Yes I agree. We should do that but not in the pipeline for the global
> >> coverage. We should have another pipeline for this and update the
> pom.xml
> >> files in it.
> >>
> >>>
> >>>> * Save a report in the directory for the Clover report at
> >>>> http://ci.xwiki.org/view/Tools/job/Clover/
> >>>> * For all failures, send an email to [hidden email] with
> >> details
> >>>> and a link to the saved report
> >>>> * Ideally, and if we can do it, call the github API to find the
> authors
> >> of
> >>>> commits for those packages and add them in the report. Examples of
> APIs
> >> we
> >>>> could use:
> >>>> ** https://api.github.com/repos/xwiki/xwiki-platform/commits?
> >>>> since=2018-05-07T00:00:00Z&until=2018-05-10T00:00:00Z (there’s a path
> >>>> parameter that could be used to filter but I don’t think it’ll work
> >>>> ** https://github.com/xwiki/xwiki-platform/compare/master@
> >>>> %7B2018-05-07%7D...master@%7B2018-05-10%7D (the committers/authors
> need
> >>>> to be extracted from the HTML which is a bit fragile)
> >>>> * Add a step in the Release process to ensure that the global TPC has
> >> not
> >>>> been lowered. This would be a way to ensure we pay attention to that
> and
> >>>> fix it when we lower it. We would need to tune this to find something
> >> that
> >>>> helps keep the TPC increasing while not putting too much pressure at
> the
> >>>> same time on the release date. It doesn’t have to be in the release
> >> process
> >>>> but we need some checkpoint to make sure we look at it and that all
> devs
> >>>> fix the tests when they lower the global TPC, or at the very least
> that
> >> an
> >>>> analysis is done in case where it’s hard to keep the TPC (for example,
> >>>> removing existing code that had a lot of tests will lower the TPC ;)).
> >>>>
> >>>> WDYT?
> >>>>
> >>>> As a dev, would you be willing to pay attention to not lower the
> global
> >>>> TPC and work to fix it when it happens?
> >>
> >> @Marius: what about this question? Would you be ok with it? :)
> >>
> >
> > Sure.
>
> ok cool
>
> Thanks
> -Vincent
>
> >
> >
> >>
> >> Thanks
> >> -Vincent
> >>
> >>>>
> >>>> Thanks
> >>>> -Vincent
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Improve our Test Coverage strategy

vmassol
Administrator


> On 14 May 2018, at 13:48, Marius Dumitru Florea <[hidden email]> wrote:

[snip]

>>>>>
>>>>>> ** Find new packages introduced that have a TPC < the average computed
>>>> TPC
>>>>>>
>>>>>
>>>>> What is the average computed TPC currently?
>>>>
>>>> It’s about 70%, see for example:
>>>> http://maven.xwiki.org/site/clover/20180511/clover-
>>>> commons+rendering+platform-20180511-0147/dashboard.html
>>>>
>>>>
>>> Requiring 70% TPC for new packages is a nice goal but I find it hard to
>>> achieve in practice.
>>
>> Two points on this:
>>
>
>
>> * We should have above 80-90%+ in practice for new modules, not 70%. If we
>> have 70% or less we can be sure you did a bad job on quality and that
>> you’re impacting the overall quality of XWiki.
>>
>
> *should* but I don't think it happens in practice, for various reasons,
> mainly due to limited time. Did you check the TPC of the last 10 new
> packages? Is it above 70%?

I’ll check it.

Thanks
-Vincent

>
>
>> * For some modules, it makes less sense or it’s harder to have unit tests
>> and integration tests are better. We’re not talking about 70% of unit test
>> coverage but 70%+ coverage of overall testing (unit, integration,
>> functional).
>>
>> Now we’ll need to put in place to see it in action and verify if there are
>> cases where this can be hard to achieve. But I’d prefer that we consider
>> those cases as exceptional and handle them in an ad-hoc manner.
>>
>>>
>>>>>
>>>>>> ** Find packages and/or files having a TPC lower than the previous TPC
>>>>>> ** Find removed packages that had a TPC higher than the average
>> computed
>>>>>> TPC
>>>>>>
>>>>>
>>>>> I would add:
>>>>>
>>>>> ** find the packages that have the TPC higher than what is declared in
>>>> the
>>>>> pom (because we don't always update the TPC value declared in the pom
>>>> when
>>>>> we refactor the code or when we add new tests)
>>>>
>>>> Yes I agree. We should do that but not in the pipeline for the global
>>>> coverage. We should have another pipeline for this and update the
>> pom.xml
>>>> files in it.
>>>>
>>>>>
>>>>>> * Save a report in the directory for the Clover report at
>>>>>> http://ci.xwiki.org/view/Tools/job/Clover/
>>>>>> * For all failures, send an email to [hidden email] with
>>>> details
>>>>>> and a link to the saved report
>>>>>> * Ideally, and if we can do it, call the github API to find the
>> authors
>>>> of
>>>>>> commits for those packages and add them in the report. Examples of
>> APIs
>>>> we
>>>>>> could use:
>>>>>> ** https://api.github.com/repos/xwiki/xwiki-platform/commits?
>>>>>> since=2018-05-07T00:00:00Z&until=2018-05-10T00:00:00Z (there’s a path
>>>>>> parameter that could be used to filter but I don’t think it’ll work
>>>>>> ** https://github.com/xwiki/xwiki-platform/compare/master@
>>>>>> %7B2018-05-07%7D...master@%7B2018-05-10%7D (the committers/authors
>> need
>>>>>> to be extracted from the HTML which is a bit fragile)
>>>>>> * Add a step in the Release process to ensure that the global TPC has
>>>> not
>>>>>> been lowered. This would be a way to ensure we pay attention to that
>> and
>>>>>> fix it when we lower it. We would need to tune this to find something
>>>> that
>>>>>> helps keep the TPC increasing while not putting too much pressure at
>> the
>>>>>> same time on the release date. It doesn’t have to be in the release
>>>> process
>>>>>> but we need some checkpoint to make sure we look at it and that all
>> devs
>>>>>> fix the tests when they lower the global TPC, or at the very least
>> that
>>>> an
>>>>>> analysis is done in case where it’s hard to keep the TPC (for example,
>>>>>> removing existing code that had a lot of tests will lower the TPC ;)).
>>>>>>
>>>>>> WDYT?
>>>>>>
>>>>>> As a dev, would you be willing to pay attention to not lower the
>> global
>>>>>> TPC and work to fix it when it happens?
>>>>
>>>> @Marius: what about this question? Would you be ok with it? :)
>>>>
>>>
>>> Sure.
>>
>> ok cool
>>
>> Thanks
>> -Vincent
>>
>>>
>>>
>>>>
>>>> Thanks
>>>> -Vincent
>>>>
>>>>>>
>>>>>> Thanks
>>>>>> -Vincent