[Proposal] New Functional test best practice for docker-based tests

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Proposal] New Functional test best practice for docker-based tests

vmassol
Administrator
Hi,

Now that JUnit 5.4.0 is out we have test ordering. I propose the following best practices for TC-based docker tests:

@UITest
@TestMethodOrder(OrderAnnotation.class)
public class MenuIT
{
    @Test
    @Order(1)
    public void verifyMenuInApplicationsIndex(TestUtils setup)
    {
...
    }

    @Test
    @Order(2)
    public void verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(TestUtils setup)
    {
...
    }

    @Test
    @Order(3)
    public void verifyMenuIsAvailableInAdministration(TestUtils setup) throws Exception
    {

    }

Instead of the current:

    @Test
    public void verifyMenu(TestUtils setup) throws Exception
    {
        verifyMenuInApplicationsIndex(setup);
        verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(setup);
        verifyMenuIsAvailableInAdministration(setup);
    }


Pros:
* Easier to run each test
* Easier to debug and view recorded video for a specific failing test
* More in sync with JUnit’s practices
* It’s still a scenario and thus doesn’t incur penalty of extra test setup

Cons:
* Starts and stop the VNC docker container for each test. It takes between 1 and 2s to start the VNC container and the stop (i.e. video recording save) should be the same as before.

So I think it’s worth it. Out of 3mn, 3-6 more seconds for 3 tests is not too much (between 2% and 3% more).

WDYT?

Thanks
-Vincent








Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] New Functional test best practice for docker-based tests

Thomas Mortagne
Administrator
On Tue, Feb 12, 2019 at 2:45 PM Vincent Massol <[hidden email]> wrote:

>
> Hi,
>
> Now that JUnit 5.4.0 is out we have test ordering. I propose the following best practices for TC-based docker tests:
>
> @UITest
> @TestMethodOrder(OrderAnnotation.class)
> public class MenuIT
> {
>     @Test
>     @Order(1)
>     public void verifyMenuInApplicationsIndex(TestUtils setup)
>     {
> ...
>     }
>
>     @Test
>     @Order(2)
>     public void verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(TestUtils setup)
>     {
> ...
>     }
>
>     @Test
>     @Order(3)
>     public void verifyMenuIsAvailableInAdministration(TestUtils setup) throws Exception
>     {
> …
>     }
>
> Instead of the current:
>
>     @Test
>     public void verifyMenu(TestUtils setup) throws Exception
>     {
>         verifyMenuInApplicationsIndex(setup);
>         verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(setup);
>         verifyMenuIsAvailableInAdministration(setup);
>     }
>
>
> Pros:
> * Easier to run each test
> * Easier to debug and view recorded video for a specific failing test
> * More in sync with JUnit’s practices
> * It’s still a scenario and thus doesn’t incur penalty of extra test setup
>
> Cons:
> * Starts and stop the VNC docker container for each test. It takes between 1 and 2s to start the VNC container and the stop (i.e. video recording save) should be the same as before.
>
> So I think it’s worth it. Out of 3mn, 3-6 more seconds for 3 tests is not too much (between 2% and 3% more).
>
> WDYT?

Why only docker based tests and not all UI tests ?

>
> Thanks
> -Vincent
>
>
>
>
>
>
>
>


--
Thomas Mortagne
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] New Functional test best practice for docker-based tests

vmassol
Administrator


> On 12 Feb 2019, at 14:49, Thomas Mortagne <[hidden email]> wrote:
>
> On Tue, Feb 12, 2019 at 2:45 PM Vincent Massol <[hidden email]> wrote:
>>
>> Hi,
>>
>> Now that JUnit 5.4.0 is out we have test ordering. I propose the following best practices for TC-based docker tests:
>>
>> @UITest
>> @TestMethodOrder(OrderAnnotation.class)
>> public class MenuIT
>> {
>>    @Test
>>    @Order(1)
>>    public void verifyMenuInApplicationsIndex(TestUtils setup)
>>    {
>> ...
>>    }
>>
>>    @Test
>>    @Order(2)
>>    public void verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(TestUtils setup)
>>    {
>> ...
>>    }
>>
>>    @Test
>>    @Order(3)
>>    public void verifyMenuIsAvailableInAdministration(TestUtils setup) throws Exception
>>    {
>> …
>>    }
>>
>> Instead of the current:
>>
>>    @Test
>>    public void verifyMenu(TestUtils setup) throws Exception
>>    {
>>        verifyMenuInApplicationsIndex(setup);
>>        verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(setup);
>>        verifyMenuIsAvailableInAdministration(setup);
>>    }
>>
>>
>> Pros:
>> * Easier to run each test
>> * Easier to debug and view recorded video for a specific failing test
>> * More in sync with JUnit’s practices
>> * It’s still a scenario and thus doesn’t incur penalty of extra test setup
>>
>> Cons:
>> * Starts and stop the VNC docker container for each test. It takes between 1 and 2s to start the VNC container and the stop (i.e. video recording save) should be the same as before.
>>
>> So I think it’s worth it. Out of 3mn, 3-6 more seconds for 3 tests is not too much (between 2% and 3% more).
>>
>> WDYT?
>
> Why only docker based tests and not all UI tests ?

Yes, all UI tests, not just the docker-based ones.

I mentioned the docker-based ones because they are the most controversial ones (with the VNC start/stop), which is the only reason I see for not wanting to do this.

Thanks
-Vincent

>
>>
>> Thanks
>> -Vincent
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> Thomas Mortagne

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] New Functional test best practice for docker-based tests

vmassol
Administrator
In reply to this post by vmassol
Actually to be even more rigorous, we should use:

@UITest
@TestMethodOrder(OrderAnnotation.class)
@TestInstance(Lifecycle.PER_CLASS)
public class MenuIT

See https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-instance-lifecycle for more details.

Thanks
-Vincent

> On 12 Feb 2019, at 14:44, Vincent Massol <[hidden email]> wrote:
>
> Hi,
>
> Now that JUnit 5.4.0 is out we have test ordering. I propose the following best practices for TC-based docker tests:
>
> @UITest
> @TestMethodOrder(OrderAnnotation.class)
> public class MenuIT
> {
>    @Test
>    @Order(1)
>    public void verifyMenuInApplicationsIndex(TestUtils setup)
>    {
> ...
>    }
>
>    @Test
>    @Order(2)
>    public void verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(TestUtils setup)
>    {
> ...
>    }
>
>    @Test
>    @Order(3)
>    public void verifyMenuIsAvailableInAdministration(TestUtils setup) throws Exception
>    {
> …
>    }
>
> Instead of the current:
>
>    @Test
>    public void verifyMenu(TestUtils setup) throws Exception
>    {
>        verifyMenuInApplicationsIndex(setup);
>        verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(setup);
>        verifyMenuIsAvailableInAdministration(setup);
>    }
>
>
> Pros:
> * Easier to run each test
> * Easier to debug and view recorded video for a specific failing test
> * More in sync with JUnit’s practices
> * It’s still a scenario and thus doesn’t incur penalty of extra test setup
>
> Cons:
> * Starts and stop the VNC docker container for each test. It takes between 1 and 2s to start the VNC container and the stop (i.e. video recording save) should be the same as before.
>
> So I think it’s worth it. Out of 3mn, 3-6 more seconds for 3 tests is not too much (between 2% and 3% more).
>
> WDYT?
>
> Thanks
> -Vincent
>
>
>
>
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] New Functional test best practice for docker-based tests

vmassol
Administrator
Improvement thanks to Simon, I’ve now moved the 2 new annotations to the UITest annotation:

@Documented
@Retention(RUNTIME)
@Target({ TYPE, METHOD, ANNOTATION_TYPE })
@ExtendWith(XWikiDockerExtension.class)
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public @interface UITest


Example usage:

@UITest
public class MenuIT
{
    @Test
    @Order(1)


Simpler, thanks Simon!

-Vincent


> On 12 Feb 2019, at 15:04, Vincent Massol <[hidden email]> wrote:
>
> Actually to be even more rigorous, we should use:
>
> @UITest
> @TestMethodOrder(OrderAnnotation.class)
> @TestInstance(Lifecycle.PER_CLASS)
> public class MenuIT
>
> See https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-instance-lifecycle for more details.
>
> Thanks
> -Vincent
>
>> On 12 Feb 2019, at 14:44, Vincent Massol <[hidden email]> wrote:
>>
>> Hi,
>>
>> Now that JUnit 5.4.0 is out we have test ordering. I propose the following best practices for TC-based docker tests:
>>
>> @UITest
>> @TestMethodOrder(OrderAnnotation.class)
>> public class MenuIT
>> {
>>   @Test
>>   @Order(1)
>>   public void verifyMenuInApplicationsIndex(TestUtils setup)
>>   {
>> ...
>>   }
>>
>>   @Test
>>   @Order(2)
>>   public void verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(TestUtils setup)
>>   {
>> ...
>>   }
>>
>>   @Test
>>   @Order(3)
>>   public void verifyMenuIsAvailableInAdministration(TestUtils setup) throws Exception
>>   {
>> …
>>   }
>>
>> Instead of the current:
>>
>>   @Test
>>   public void verifyMenu(TestUtils setup) throws Exception
>>   {
>>       verifyMenuInApplicationsIndex(setup);
>>       verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(setup);
>>       verifyMenuIsAvailableInAdministration(setup);
>>   }
>>
>>
>> Pros:
>> * Easier to run each test
>> * Easier to debug and view recorded video for a specific failing test
>> * More in sync with JUnit’s practices
>> * It’s still a scenario and thus doesn’t incur penalty of extra test setup
>>
>> Cons:
>> * Starts and stop the VNC docker container for each test. It takes between 1 and 2s to start the VNC container and the stop (i.e. video recording save) should be the same as before.
>>
>> So I think it’s worth it. Out of 3mn, 3-6 more seconds for 3 tests is not too much (between 2% and 3% more).
>>
>> WDYT?
>>
>> Thanks
>> -Vincent
>>
>>
>>
>>
>>
>>
>>
>>
>