[Proposal] Improve our Test Coverage strategy (TAKE 3)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Proposal] Improve our Test Coverage strategy (TAKE 3)

vmassol
Administrator
Hi devs,

After TAKE 2 (see http://markmail.org/message/owtyhkmrz4tcbymn ),  and after analyzing several modules (I analyzed about 4-5 in total), I think we should improve a bit the strategy to make it usable and applicable.

Lessons learnt
============

* It takes a lot of time to analyze a single global tpc drop (every time it takes me around 2 hours)
* In general the results of the analysis are not that great. There are often “good enough” reasons for the drop. It’s often a lack of unit tests and code that is exercised by functional tests but the path has changed for various reasons.
* I find the ratio of time to spend on the analysis vs result to be too low.
* In the end what’s important is that our global TPC continues to grow

New Strategy
===========

* We run the Clover Jenkins pipeline every night (between 11PM-8AM)
* The pipeline sends an email whenever the new report has its global TPC going down when compared with the baseline (vs one or more modules had their TPC lowered in TAKE 2)
* The baseline report is the report generated just after each XS release. This means that we keep the same baseline during a XS release
** Technically it means that the pipeline will update the latest.txt file (which contains the clover report timestamp) when it notices a version change
* We add a step in the Release Plan Template to have the report passing before we can release.
* The RM is in charge of a release from day 1 to the release day (already the case), and is also in charge of making sure that the global coverage job failures get addressed before the release day so that we’re ready on the release day.
* Implementation detail: don’t send a failure email when there are failing tests in the build, to avoid false positives

WDYT?

Thanks
-Vincent
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Improve our Test Coverage strategy (TAKE 3)

Thomas Mortagne
Administrator
+1

On Sun, Feb 10, 2019 at 2:50 PM Vincent Massol <[hidden email]> wrote:

>
> Hi devs,
>
> After TAKE 2 (see http://markmail.org/message/owtyhkmrz4tcbymn ),  and after analyzing several modules (I analyzed about 4-5 in total), I think we should improve a bit the strategy to make it usable and applicable.
>
> Lessons learnt
> ============
>
> * It takes a lot of time to analyze a single global tpc drop (every time it takes me around 2 hours)
> * In general the results of the analysis are not that great. There are often “good enough” reasons for the drop. It’s often a lack of unit tests and code that is exercised by functional tests but the path has changed for various reasons.
> * I find the ratio of time to spend on the analysis vs result to be too low.
> * In the end what’s important is that our global TPC continues to grow
>
> New Strategy
> ===========
>
> * We run the Clover Jenkins pipeline every night (between 11PM-8AM)
> * The pipeline sends an email whenever the new report has its global TPC going down when compared with the baseline (vs one or more modules had their TPC lowered in TAKE 2)
> * The baseline report is the report generated just after each XS release. This means that we keep the same baseline during a XS release
> ** Technically it means that the pipeline will update the latest.txt file (which contains the clover report timestamp) when it notices a version change
> * We add a step in the Release Plan Template to have the report passing before we can release.
> * The RM is in charge of a release from day 1 to the release day (already the case), and is also in charge of making sure that the global coverage job failures get addressed before the release day so that we’re ready on the release day.
> * Implementation detail: don’t send a failure email when there are failing tests in the build, to avoid false positives
>
> WDYT?
>
> Thanks
> -Vincent



--
Thomas Mortagne
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Improve our Test Coverage strategy (TAKE 3)

Simon Urli
So if I sum up, we're moving the scope of global coverage from each
modules to the whole project, right?

+1 for getting email/build failing only if the global TPC decrease

Now I assume in case of decrease, we can get the whole report with info
about which modules had a change in their TPC, right?

Simon

On 11/02/2019 14:48, Thomas Mortagne wrote:

> +1
>
> On Sun, Feb 10, 2019 at 2:50 PM Vincent Massol <[hidden email]> wrote:
>>
>> Hi devs,
>>
>> After TAKE 2 (see http://markmail.org/message/owtyhkmrz4tcbymn ),  and after analyzing several modules (I analyzed about 4-5 in total), I think we should improve a bit the strategy to make it usable and applicable.
>>
>> Lessons learnt
>> ============
>>
>> * It takes a lot of time to analyze a single global tpc drop (every time it takes me around 2 hours)
>> * In general the results of the analysis are not that great. There are often “good enough” reasons for the drop. It’s often a lack of unit tests and code that is exercised by functional tests but the path has changed for various reasons.
>> * I find the ratio of time to spend on the analysis vs result to be too low.
>> * In the end what’s important is that our global TPC continues to grow
>>
>> New Strategy
>> ===========
>>
>> * We run the Clover Jenkins pipeline every night (between 11PM-8AM)
>> * The pipeline sends an email whenever the new report has its global TPC going down when compared with the baseline (vs one or more modules had their TPC lowered in TAKE 2)
>> * The baseline report is the report generated just after each XS release. This means that we keep the same baseline during a XS release
>> ** Technically it means that the pipeline will update the latest.txt file (which contains the clover report timestamp) when it notices a version change
>> * We add a step in the Release Plan Template to have the report passing before we can release.
>> * The RM is in charge of a release from day 1 to the release day (already the case), and is also in charge of making sure that the global coverage job failures get addressed before the release day so that we’re ready on the release day.
>> * Implementation detail: don’t send a failure email when there are failing tests in the build, to avoid false positives
>>
>> WDYT?
>>
>> Thanks
>> -Vincent
>
>
>

--
Simon Urli
Software Engineer at XWiki SAS
[hidden email]
More about us at http://www.xwiki.com
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Improve our Test Coverage strategy (TAKE 3)

vmassol
Administrator
Hi Simon,

> On 11 Feb 2019, at 15:43, Simon Urli <[hidden email]> wrote:
>
> So if I sum up, we're moving the scope of global coverage from each modules to the whole project, right?

Correct.

> +1 for getting email/build failing only if the global TPC decrease
>
> Now I assume in case of decrease, we can get the whole report with info about which modules had a change in their TPC, right?

Correct again :) Actually it’s even my goal to improve it and also give details about package level (easy) and maybe even class level (harder), for “failing” modules.

Thanks
-Vincent

> Simon
>
> On 11/02/2019 14:48, Thomas Mortagne wrote:
>> +1
>> On Sun, Feb 10, 2019 at 2:50 PM Vincent Massol <[hidden email]> wrote:
>>>
>>> Hi devs,
>>>
>>> After TAKE 2 (see http://markmail.org/message/owtyhkmrz4tcbymn ),  and after analyzing several modules (I analyzed about 4-5 in total), I think we should improve a bit the strategy to make it usable and applicable.
>>>
>>> Lessons learnt
>>> ============
>>>
>>> * It takes a lot of time to analyze a single global tpc drop (every time it takes me around 2 hours)
>>> * In general the results of the analysis are not that great. There are often “good enough” reasons for the drop. It’s often a lack of unit tests and code that is exercised by functional tests but the path has changed for various reasons.
>>> * I find the ratio of time to spend on the analysis vs result to be too low.
>>> * In the end what’s important is that our global TPC continues to grow
>>>
>>> New Strategy
>>> ===========
>>>
>>> * We run the Clover Jenkins pipeline every night (between 11PM-8AM)
>>> * The pipeline sends an email whenever the new report has its global TPC going down when compared with the baseline (vs one or more modules had their TPC lowered in TAKE 2)
>>> * The baseline report is the report generated just after each XS release. This means that we keep the same baseline during a XS release
>>> ** Technically it means that the pipeline will update the latest.txt file (which contains the clover report timestamp) when it notices a version change
>>> * We add a step in the Release Plan Template to have the report passing before we can release.
>>> * The RM is in charge of a release from day 1 to the release day (already the case), and is also in charge of making sure that the global coverage job failures get addressed before the release day so that we’re ready on the release day.
>>> * Implementation detail: don’t send a failure email when there are failing tests in the build, to avoid false positives
>>>
>>> WDYT?
>>>
>>> Thanks
>>> -Vincent
>
> --
> Simon Urli
> Software Engineer at XWiki SAS
> [hidden email]
> More about us at http://www.xwiki.com

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Improve our Test Coverage strategy (TAKE 3)

Marius Dumitru Florea
In reply to this post by vmassol
+1

Thanks,
Marius

On Sun, Feb 10, 2019 at 3:50 PM Vincent Massol <[hidden email]> wrote:

> Hi devs,
>
> After TAKE 2 (see http://markmail.org/message/owtyhkmrz4tcbymn ),  and
> after analyzing several modules (I analyzed about 4-5 in total), I think we
> should improve a bit the strategy to make it usable and applicable.
>
> Lessons learnt
> ============
>
> * It takes a lot of time to analyze a single global tpc drop (every time
> it takes me around 2 hours)
> * In general the results of the analysis are not that great. There are
> often “good enough” reasons for the drop. It’s often a lack of unit tests
> and code that is exercised by functional tests but the path has changed for
> various reasons.
> * I find the ratio of time to spend on the analysis vs result to be too
> low.
> * In the end what’s important is that our global TPC continues to grow
>
> New Strategy
> ===========
>
> * We run the Clover Jenkins pipeline every night (between 11PM-8AM)
> * The pipeline sends an email whenever the new report has its global TPC
> going down when compared with the baseline (vs one or more modules had
> their TPC lowered in TAKE 2)
> * The baseline report is the report generated just after each XS release.
> This means that we keep the same baseline during a XS release
> ** Technically it means that the pipeline will update the latest.txt file
> (which contains the clover report timestamp) when it notices a version
> change
> * We add a step in the Release Plan Template to have the report passing
> before we can release.
> * The RM is in charge of a release from day 1 to the release day (already
> the case), and is also in charge of making sure that the global coverage
> job failures get addressed before the release day so that we’re ready on
> the release day.
> * Implementation detail: don’t send a failure email when there are failing
> tests in the build, to avoid false positives
>
> WDYT?
>
> Thanks
> -Vincent