[Proposal] Velocity Test Coverage strategy

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[Proposal] Velocity Test Coverage strategy

vmassol
Administrator
Hi devs,

During XWiki SAS’s hackathon last week, Simon and me worked on implementing test coverage computation for velocity code and more precisely to measure the code coverage we get in XWiki XML pages when running our tests.

The rationale is that we know what’s our java test coverage but we have no clue about the velocity one. And we have a lot of code in velocity scripts in wiki pages. Thus we need a strategy for this too if we wish to increase our global code quality.

So we have currently developed 2 mojos (xar:instrument and xar:reportCoverage) in the XAR plugin code and created a JIRA issue, see XCOMMONS-1448.

Here’s the proposal I’d like your opinion on:
* Finish working on this to stabilize it and commit/push it
* Apply the same strategy we have with Jacoco for java test coverage, i.e. introduce a new xar:coverageCheck mojo that will fail the build if we get a global TPC under the threshold mentioned in the POM

Consequences:
* It will mean that whenever we add new velocity scripts (especially when there are branches such as #if) we will need to improve or add XAR page tests. This can be done in 2 ways:
** by writing/improving a functional UI test
** by writing/improving a XAR unit test
* We will find places that have 0% coverage and these will be good candidates to add tests for

My POV:
* We should have the minimum # of functional UI tests since they take very long to execute. We need them but we shouldn’t test the various branches with them IMO. Only one path.
* Instead we should focus on have more of XAR unit tests since they execute fast and are better suited (with mocks) to test the various branches
* The XAR unit test framework we have is still pretty new and it’s probably not to easy to write unit tests for wiki pages in some cases, we will need to work on that as we discover them. I’m happy to help on that.

WDYT?

Personally I’m ok to try it and see what happens.

Thanks
-Vincent
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Velocity Test Coverage strategy

Adel Atallah
On Sun, Jul 29, 2018 at 5:56 PM, Vincent Massol <[hidden email]> wrote:

> Hi devs,
>
> During XWiki SAS’s hackathon last week, Simon and me worked on implementing test coverage computation for velocity code and more precisely to measure the code coverage we get in XWiki XML pages when running our tests.
>
> The rationale is that we know what’s our java test coverage but we have no clue about the velocity one. And we have a lot of code in velocity scripts in wiki pages. Thus we need a strategy for this too if we wish to increase our global code quality.
>
> So we have currently developed 2 mojos (xar:instrument and xar:reportCoverage) in the XAR plugin code and created a JIRA issue, see XCOMMONS-1448.
>
> Here’s the proposal I’d like your opinion on:
> * Finish working on this to stabilize it and commit/push it
> * Apply the same strategy we have with Jacoco for java test coverage, i.e. introduce a new xar:coverageCheck mojo that will fail the build if we get a global TPC under the threshold mentioned in the POM
>
> Consequences:
> * It will mean that whenever we add new velocity scripts (especially when there are branches such as #if) we will need to improve or add XAR page tests. This can be done in 2 ways:
> ** by writing/improving a functional UI test
> ** by writing/improving a XAR unit test
> * We will find places that have 0% coverage and these will be good candidates to add tests for
>
> My POV:
> * We should have the minimum # of functional UI tests since they take very long to execute. We need them but we shouldn’t test the various branches with them IMO. Only one path.
> * Instead we should focus on have more of XAR unit tests since they execute fast and are better suited (with mocks) to test the various branches
> * The XAR unit test framework we have is still pretty new and it’s probably not to easy to write unit tests for wiki pages in some cases, we will need to work on that as we discover them. I’m happy to help on that.
>
> WDYT?
>
> Personally I’m ok to try it and see what happens.
>
> Thanks
> -Vincent

+1
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Velocity Test Coverage strategy

Eduard Moraru
+1

It would also be nice to eventually have some kind of reporting UI, similar
to jacoco that allows you to drill down towards the affected pages/vm
files, preferably ending with some highlighted view of the covered/not
covered lines.

Thanks,
Eduard

On Sun, Jul 29, 2018 at 7:09 PM, Adel Atallah <[hidden email]>
wrote:

> On Sun, Jul 29, 2018 at 5:56 PM, Vincent Massol <[hidden email]>
> wrote:
> > Hi devs,
> >
> > During XWiki SAS’s hackathon last week, Simon and me worked on
> implementing test coverage computation for velocity code and more precisely
> to measure the code coverage we get in XWiki XML pages when running our
> tests.
> >
> > The rationale is that we know what’s our java test coverage but we have
> no clue about the velocity one. And we have a lot of code in velocity
> scripts in wiki pages. Thus we need a strategy for this too if we wish to
> increase our global code quality.
> >
> > So we have currently developed 2 mojos (xar:instrument and
> xar:reportCoverage) in the XAR plugin code and created a JIRA issue, see
> XCOMMONS-1448.
> >
> > Here’s the proposal I’d like your opinion on:
> > * Finish working on this to stabilize it and commit/push it
> > * Apply the same strategy we have with Jacoco for java test coverage,
> i.e. introduce a new xar:coverageCheck mojo that will fail the build if we
> get a global TPC under the threshold mentioned in the POM
> >
> > Consequences:
> > * It will mean that whenever we add new velocity scripts (especially
> when there are branches such as #if) we will need to improve or add XAR
> page tests. This can be done in 2 ways:
> > ** by writing/improving a functional UI test
> > ** by writing/improving a XAR unit test
> > * We will find places that have 0% coverage and these will be good
> candidates to add tests for
> >
> > My POV:
> > * We should have the minimum # of functional UI tests since they take
> very long to execute. We need them but we shouldn’t test the various
> branches with them IMO. Only one path.
> > * Instead we should focus on have more of XAR unit tests since they
> execute fast and are better suited (with mocks) to test the various branches
> > * The XAR unit test framework we have is still pretty new and it’s
> probably not to easy to write unit tests for wiki pages in some cases, we
> will need to work on that as we discover them. I’m happy to help on that.
> >
> > WDYT?
> >
> > Personally I’m ok to try it and see what happens.
> >
> > Thanks
> > -Vincent
>
> +1
>
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Velocity Test Coverage strategy

Thomas Mortagne
Administrator
+1 and same as Edy, I very often use the jacoco report to know exactly
where I can improve the coverage

On Mon, Jul 30, 2018 at 1:02 AM, Eduard Moraru <[hidden email]> wrote:

> +1
>
> It would also be nice to eventually have some kind of reporting UI, similar
> to jacoco that allows you to drill down towards the affected pages/vm
> files, preferably ending with some highlighted view of the covered/not
> covered lines.
>
> Thanks,
> Eduard
>
> On Sun, Jul 29, 2018 at 7:09 PM, Adel Atallah <[hidden email]>
> wrote:
>
>> On Sun, Jul 29, 2018 at 5:56 PM, Vincent Massol <[hidden email]>
>> wrote:
>> > Hi devs,
>> >
>> > During XWiki SAS’s hackathon last week, Simon and me worked on
>> implementing test coverage computation for velocity code and more precisely
>> to measure the code coverage we get in XWiki XML pages when running our
>> tests.
>> >
>> > The rationale is that we know what’s our java test coverage but we have
>> no clue about the velocity one. And we have a lot of code in velocity
>> scripts in wiki pages. Thus we need a strategy for this too if we wish to
>> increase our global code quality.
>> >
>> > So we have currently developed 2 mojos (xar:instrument and
>> xar:reportCoverage) in the XAR plugin code and created a JIRA issue, see
>> XCOMMONS-1448.
>> >
>> > Here’s the proposal I’d like your opinion on:
>> > * Finish working on this to stabilize it and commit/push it
>> > * Apply the same strategy we have with Jacoco for java test coverage,
>> i.e. introduce a new xar:coverageCheck mojo that will fail the build if we
>> get a global TPC under the threshold mentioned in the POM
>> >
>> > Consequences:
>> > * It will mean that whenever we add new velocity scripts (especially
>> when there are branches such as #if) we will need to improve or add XAR
>> page tests. This can be done in 2 ways:
>> > ** by writing/improving a functional UI test
>> > ** by writing/improving a XAR unit test
>> > * We will find places that have 0% coverage and these will be good
>> candidates to add tests for
>> >
>> > My POV:
>> > * We should have the minimum # of functional UI tests since they take
>> very long to execute. We need them but we shouldn’t test the various
>> branches with them IMO. Only one path.
>> > * Instead we should focus on have more of XAR unit tests since they
>> execute fast and are better suited (with mocks) to test the various branches
>> > * The XAR unit test framework we have is still pretty new and it’s
>> probably not to easy to write unit tests for wiki pages in some cases, we
>> will need to work on that as we discover them. I’m happy to help on that.
>> >
>> > WDYT?
>> >
>> > Personally I’m ok to try it and see what happens.
>> >
>> > Thanks
>> > -Vincent
>>
>> +1
>>



--
Thomas Mortagne
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Velocity Test Coverage strategy

vmassol
Administrator
In reply to this post by Eduard Moraru


> On 30 Jul 2018, at 01:02, Eduard Moraru <[hidden email]> wrote:
>
> +1
>
> It would also be nice to eventually have some kind of reporting UI, similar
> to jacoco that allows you to drill down towards the affected pages/vm
> files, preferably ending with some highlighted view of the covered/not
> covered lines.

Yes, we should be able to at least provide the line numbers and file name for lines that have not been called.

Thanks
-vincent

>
> Thanks,
> Eduard
>
> On Sun, Jul 29, 2018 at 7:09 PM, Adel Atallah <[hidden email]>
> wrote:
>
>> On Sun, Jul 29, 2018 at 5:56 PM, Vincent Massol <[hidden email]>
>> wrote:
>>> Hi devs,
>>>
>>> During XWiki SAS’s hackathon last week, Simon and me worked on
>> implementing test coverage computation for velocity code and more precisely
>> to measure the code coverage we get in XWiki XML pages when running our
>> tests.
>>>
>>> The rationale is that we know what’s our java test coverage but we have
>> no clue about the velocity one. And we have a lot of code in velocity
>> scripts in wiki pages. Thus we need a strategy for this too if we wish to
>> increase our global code quality.
>>>
>>> So we have currently developed 2 mojos (xar:instrument and
>> xar:reportCoverage) in the XAR plugin code and created a JIRA issue, see
>> XCOMMONS-1448.
>>>
>>> Here’s the proposal I’d like your opinion on:
>>> * Finish working on this to stabilize it and commit/push it
>>> * Apply the same strategy we have with Jacoco for java test coverage,
>> i.e. introduce a new xar:coverageCheck mojo that will fail the build if we
>> get a global TPC under the threshold mentioned in the POM
>>>
>>> Consequences:
>>> * It will mean that whenever we add new velocity scripts (especially
>> when there are branches such as #if) we will need to improve or add XAR
>> page tests. This can be done in 2 ways:
>>> ** by writing/improving a functional UI test
>>> ** by writing/improving a XAR unit test
>>> * We will find places that have 0% coverage and these will be good
>> candidates to add tests for
>>>
>>> My POV:
>>> * We should have the minimum # of functional UI tests since they take
>> very long to execute. We need them but we shouldn’t test the various
>> branches with them IMO. Only one path.
>>> * Instead we should focus on have more of XAR unit tests since they
>> execute fast and are better suited (with mocks) to test the various branches
>>> * The XAR unit test framework we have is still pretty new and it’s
>> probably not to easy to write unit tests for wiki pages in some cases, we
>> will need to work on that as we discover them. I’m happy to help on that.
>>>
>>> WDYT?
>>>
>>> Personally I’m ok to try it and see what happens.
>>>
>>> Thanks
>>> -Vincent
>>
>> +1
>>

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Velocity Test Coverage strategy

Marius Dumitru Florea
In reply to this post by vmassol
+1

Thanks,
Marius

On Sun, Jul 29, 2018 at 6:56 PM, Vincent Massol <[hidden email]> wrote:

> Hi devs,
>
> During XWiki SAS’s hackathon last week, Simon and me worked on
> implementing test coverage computation for velocity code and more precisely
> to measure the code coverage we get in XWiki XML pages when running our
> tests.
>
> The rationale is that we know what’s our java test coverage but we have no
> clue about the velocity one. And we have a lot of code in velocity scripts
> in wiki pages. Thus we need a strategy for this too if we wish to increase
> our global code quality.
>
> So we have currently developed 2 mojos (xar:instrument and
> xar:reportCoverage) in the XAR plugin code and created a JIRA issue, see
> XCOMMONS-1448.
>
> Here’s the proposal I’d like your opinion on:
> * Finish working on this to stabilize it and commit/push it
> * Apply the same strategy we have with Jacoco for java test coverage, i.e.
> introduce a new xar:coverageCheck mojo that will fail the build if we get a
> global TPC under the threshold mentioned in the POM
>
> Consequences:
> * It will mean that whenever we add new velocity scripts (especially when
> there are branches such as #if) we will need to improve or add XAR page
> tests. This can be done in 2 ways:
> ** by writing/improving a functional UI test
> ** by writing/improving a XAR unit test
> * We will find places that have 0% coverage and these will be good
> candidates to add tests for
>
> My POV:
> * We should have the minimum # of functional UI tests since they take very
> long to execute. We need them but we shouldn’t test the various branches
> with them IMO. Only one path.
> * Instead we should focus on have more of XAR unit tests since they
> execute fast and are better suited (with mocks) to test the various branches
> * The XAR unit test framework we have is still pretty new and it’s
> probably not to easy to write unit tests for wiki pages in some cases, we
> will need to work on that as we discover them. I’m happy to help on that.
>
> WDYT?
>
> Personally I’m ok to try it and see what happens.
>
> Thanks
> -Vincent