[Proposal] Cleaning of flickering tests

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Proposal] Cleaning of flickering tests

Simon Urli
Hi everyone,

I was checking our list of flickering tests in JIRA
(https://jira.xwiki.org/issues/?jql=labels%20%3D%20flickering%20AND%20status%20%3D%20Open%20ORDER%20BY%20updated%20DESC)
and I noticed that we had somehow old flickering test issue concerning
test that I've never seen failing.

So I propose we close some of them as inactive: the ones that we don't
remember having seen for a while. The ideal would be to have a mechanism
to update the issue when the CI fails on a flicker, but it takes time to
do properly and it's not a priority.

On the contrary I propose to trust our memory: if we're wrong because we
have closed a flicker that is still happening, it will allow us to
remind that we have this flicker to fix and we can easily reopen the issue.

As Thomas mentioned on the chat, we should also update the release plan
to include the inactive flickers in the list of issue to check.

So for now I propose to close the following list of issues as inactive:

   * XWIKI-14399: AddRemoveTagsTest#addAndDeleteTagFromTagPage is
flickering (https://jira.xwiki.org/browse/XWIKI-14399)
   * XWIKI-14396: AnnotationsTest#addAndDeleteAnnotations is flickering
(https://jira.xwiki.org/browse/XWIKI-14396)
   * XWIKI-14394: SectionTest.testSectionEditInWikiEditorWhenSyntax2x
(xwiki-enterprise-test-ui) is flaky
(https://jira.xwiki.org/browse/XWIKI-14394)
   * XWIKI-14386: appwithinminutes.AppsLiveTableTest.testEditApplication
is possibly flaky (https://jira.xwiki.org/browse/XWIKI-14386)
   * XWIKI-14835:
DeletePageTest#deletePageIsImpossibleWhenNoDeleteRights is flickering
(https://jira.xwiki.org/browse/XWIKI-14835)
   * XWIKI-14860: LoginTest#testDataIsPreservedAfterLogin is flickering
(https://jira.xwiki.org/browse/XWIKI-14860)

And I propose in general to close the flickers we don't remember having
seen after a cycle as inactive.

WDYT?

Simon
--
Simon Urli
Software Engineer at XWiki SAS
[hidden email]
More about us at http://www.xwiki.com
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Cleaning of flickering tests

vmassol
Administrator
Hi Simon and all,

> On 26 Mar 2019, at 10:31, Simon Urli <[hidden email]> wrote:
>
> Hi everyone,
>
> I was checking our list of flickering tests in JIRA (https://jira.xwiki.org/issues/?jql=labels%20%3D%20flickering%20AND%20status%20%3D%20Open%20ORDER%20BY%20updated%20DESC) and I noticed that we had somehow old flickering test issue concerning test that I've never seen failing.
>
> So I propose we close some of them as inactive: the ones that we don't remember having seen for a while. The ideal would be to have a mechanism to update the issue when the CI fails on a flicker, but it takes time to do properly and it's not a priority.
>
> On the contrary I propose to trust our memory: if we're wrong because we have closed a flicker that is still happening, it will allow us to remind that we have this flicker to fix and we can easily reopen the issue.
>
> As Thomas mentioned on the chat, we should also update the release plan to include the inactive flickers in the list of issue to check.
>
> So for now I propose to close the following list of issues as inactive:
>
>  * XWIKI-14399: AddRemoveTagsTest#addAndDeleteTagFromTagPage is flickering (https://jira.xwiki.org/browse/XWIKI-14399)
>  * XWIKI-14396: AnnotationsTest#addAndDeleteAnnotations is flickering (https://jira.xwiki.org/browse/XWIKI-14396)
>  * XWIKI-14394: SectionTest.testSectionEditInWikiEditorWhenSyntax2x (xwiki-enterprise-test-ui) is flaky (https://jira.xwiki.org/browse/XWIKI-14394)
>  * XWIKI-14386: appwithinminutes.AppsLiveTableTest.testEditApplication is possibly flaky (https://jira.xwiki.org/browse/XWIKI-14386)
>  * XWIKI-14835: DeletePageTest#deletePageIsImpossibleWhenNoDeleteRights is flickering (https://jira.xwiki.org/browse/XWIKI-14835)
>  * XWIKI-14860: LoginTest#testDataIsPreservedAfterLogin is flickering (https://jira.xwiki.org/browse/XWIKI-14860)
>
> And I propose in general to close the flickers we don't remember having seen after a cycle as inactive.
>
> WDYT?

Thanks for looking into this. I’d feel more comfortable if we can do some local testing before closing any test issue.

For ex, I’d propose to run it N times (can also be done on a docker container agent, it’s very easy to start a new container on ks* or locally) and if it doesn’t  fail at all then close it if it’s old enough.

Note: To repeat N times a test:
* For junit5: @RepeatedTest(N)
* For junit4: @Intermittent(repetition=N) - see https://dev.xwiki.org/xwiki/bin/view/Community/Testing/
* At the shell level:

for i in {1..50}
do
   mvn test -D…
done

Some rules I propose:
* N = 50 (for a test taking 3 minutes, that means 2.5 hours)
* Last update date of the test jira issue < Now() - 6 months

WDYT? Too hard?

Thanks
-Vincent

>
> Simon
> --
> Simon Urli
> Software Engineer at XWiki SAS
> [hidden email]
> More about us at http://www.xwiki.com

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Cleaning of flickering tests

vmassol
Administrator
In reply to this post by Simon Urli


> On 26 Mar 2019, at 10:31, Simon Urli <[hidden email]> wrote:
>
> Hi everyone,
>
> I was checking our list of flickering tests in JIRA (https://jira.xwiki.org/issues/?jql=labels%20%3D%20flickering%20AND%20status%20%3D%20Open%20ORDER%20BY%20updated%20DESC) and I noticed that we had somehow old flickering test issue concerning test that I've never seen failing.
>
> So I propose we close some of them as inactive: the ones that we don't remember having seen for a while. The ideal would be to have a mechanism to update the issue when the CI fails on a flicker, but it takes time to do properly and it's not a priority.
>
> On the contrary I propose to trust our memory: if we're wrong because we have closed a flicker that is still happening, it will allow us to remind that we have this flicker to fix and we can easily reopen the issue.
>
> As Thomas mentioned on the chat, we should also update the release plan to include the inactive flickers in the list of issue to check.

I should be able to easily create a report when any test fails inside our jenkins pipeline and make it available similar to our clover report. I could indicate if it’s a known flicker or not too in this report. That could compensate for the fact that we only keep 7 days of records in our jobs.

Would need to define the report format, whether it’s the same file updated at each run or a different one. If the same one, then either:
* I’d need to parse it first in memory, add the new tests and overwrite the file
* or add to the bottom of the file which will grow quite large quickly

WDYT?

Thanks
-Vincent

>
> So for now I propose to close the following list of issues as inactive:
>
>  * XWIKI-14399: AddRemoveTagsTest#addAndDeleteTagFromTagPage is flickering (https://jira.xwiki.org/browse/XWIKI-14399)
>  * XWIKI-14396: AnnotationsTest#addAndDeleteAnnotations is flickering (https://jira.xwiki.org/browse/XWIKI-14396)
>  * XWIKI-14394: SectionTest.testSectionEditInWikiEditorWhenSyntax2x (xwiki-enterprise-test-ui) is flaky (https://jira.xwiki.org/browse/XWIKI-14394)
>  * XWIKI-14386: appwithinminutes.AppsLiveTableTest.testEditApplication is possibly flaky (https://jira.xwiki.org/browse/XWIKI-14386)
>  * XWIKI-14835: DeletePageTest#deletePageIsImpossibleWhenNoDeleteRights is flickering (https://jira.xwiki.org/browse/XWIKI-14835)
>  * XWIKI-14860: LoginTest#testDataIsPreservedAfterLogin is flickering (https://jira.xwiki.org/browse/XWIKI-14860)
>
> And I propose in general to close the flickers we don't remember having seen after a cycle as inactive.
>
> WDYT?
>
> Simon
> --
> Simon Urli
> Software Engineer at XWiki SAS
> [hidden email]
> More about us at http://www.xwiki.com