[Proposal] Process for handling flaky tests

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[Proposal] Process for handling flaky tests

vmassol
Administrator
Hi devs,

I’d like to implement this process:

* When we discover a flaky test, we create a jira issue for it and we fill a custom field to indicate in a structured manner the name of the test class and test method (e.g. using the “<class name>#<method name>” format).
* When a test fails, our Jenkins pipeline script verifies if it’s listed in jira and if so, doesn’t send a mail but marks it in the UI (same as when we detect environment-related false positives).

Let me know quickly if you don’t like it since I’m adding the custom field to jira as I’d like to progress on the scrjpt.

Thanks
-Vincent

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Process for handling flaky tests

Thomas Mortagne
Administrator
We already have the rule of creating jira issues for flickering tests
(with label "flickering") so it does not cost much more to add more
structured information about the test in those. A custom field might
make it a easier to both fill and query.

On Mon, Sep 25, 2017 at 1:59 PM, Vincent Massol <[hidden email]> wrote:

> Hi devs,
>
> I’d like to implement this process:
>
> * When we discover a flaky test, we create a jira issue for it and we fill a custom field to indicate in a structured manner the name of the test class and test method (e.g. using the “<class name>#<method name>” format).
> * When a test fails, our Jenkins pipeline script verifies if it’s listed in jira and if so, doesn’t send a mail but marks it in the UI (same as when we detect environment-related false positives).
>
> Let me know quickly if you don’t like it since I’m adding the custom field to jira as I’d like to progress on the scrjpt.
>
> Thanks
> -Vincent
>



--
Thomas Mortagne