[Brainstorming] EvoCrash usage on XWiki?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Brainstorming] EvoCrash usage on XWiki?

vmassol
Administrator
Hi devs,

Context
======

Evocrash is a tool developed as part of the STAMP research project (https://www.stamp-project.eu/). Its goal is to take a stack trace (the use case is production stack traces) and generate a test that, when executed, reproduces the stack trace ;)

It’s using Guided Genetic Algorithm to simplify the search space (which is pretty cool, isn’t it? ;)). More info at https://github.com/STAMP-project/EvoCrash and http://www.evocrash.org/

Results
======

So far evocrash was able to generate tests for the following issues (I’ve pasted the generated test code as comment):
* https://jira.xwiki.org/browse/XRENDERING-422
* https://jira.xwiki.org/browse/XWIKI-14475
* https://jira.xwiki.org/browse/XWIKI-13031
* https://jira.xwiki.org/browse/XWIKI-13196
* https://jira.xwiki.org/browse/XWIKI-13916

Using
=====

The use case I see is the following:
* A user reports a problem when using XWiki and create a jira issue with a stack trace
* An XWiki dev finds it and runs evocrash to generate a test reproducing the problem
* The XWiki dev uses the test to understand the reason (breakpoints can be put in the IDE) of the problem and writes a better test (which can be inspired by the test generated by evocrash or be completely different).

Questions
========

The question we need to answer is whether we think it coud be useful or not. Are there cases where it can be useful?

For example if we imagine someone not knowing the xwiki code base well, maybe this can help him/her understand the bug in a simpler way than just having to read/find where the problem is in the code base?

I’ve started the work of trying to evaluate how useful it could be on the XWiki project by analyzing the examples above. I've only analyzed 2 ATM (see the links in the jira comments) and I’d be interested to have your analysis too.

WDYT?

Thanks
-Vincent
Reply | Threaded
Open this post in threaded view
|

Re: [Brainstorming] EvoCrash usage on XWiki?

Thomas Mortagne
Administrator
Not sure what to think of those examples.

The theory sounds nice and it probably produce nice result when
talking about one method with a few inputs and an output but for
anything more complex (like anything involving components) I feel you
end up with a generated test which is a lot harder to read than simply
looking at the code.

* https://jira.xwiki.org/browse/XWIKI-13031: does not produce the same
error at all. Looks like evocash just try to produce any
ClassCastException but does not care about the actual types in it
which is quite useless (plus reading that test requires a lot of
effort just to conclude that the type of the field in the document is
not String...)

* https://jira.xwiki.org/browse/XWIKI-14475: totally unreadable
* https://jira.xwiki.org/browse/XRENDERING-422: not really useful but
I doubt evocrash can do much about this kind of issue

* https://jira.xwiki.org/browse/XWIKI-13196: this one could help a bit
I guess but not sure it's really faster to execute evocash than
looking at the code :)
* https://jira.xwiki.org/browse/XWIKI-13916: that's a more complex way
than it should to express that we call addTextAreaField for an already
existing field yes it's useful

On Fri, Apr 6, 2018 at 10:44 AM, Vincent Massol <[hidden email]> wrote:

> Hi devs,
>
> Context
> ======
>
> Evocrash is a tool developed as part of the STAMP research project (https://www.stamp-project.eu/). Its goal is to take a stack trace (the use case is production stack traces) and generate a test that, when executed, reproduces the stack trace ;)
>
> It’s using Guided Genetic Algorithm to simplify the search space (which is pretty cool, isn’t it? ;)). More info at https://github.com/STAMP-project/EvoCrash and http://www.evocrash.org/
>
> Results
> ======
>
> So far evocrash was able to generate tests for the following issues (I’ve pasted the generated test code as comment):
> * https://jira.xwiki.org/browse/XRENDERING-422
> * https://jira.xwiki.org/browse/XWIKI-14475
> * https://jira.xwiki.org/browse/XWIKI-13031
> * https://jira.xwiki.org/browse/XWIKI-13196
> * https://jira.xwiki.org/browse/XWIKI-13916
>
> Using
> =====
>
> The use case I see is the following:
> * A user reports a problem when using XWiki and create a jira issue with a stack trace
> * An XWiki dev finds it and runs evocrash to generate a test reproducing the problem
> * The XWiki dev uses the test to understand the reason (breakpoints can be put in the IDE) of the problem and writes a better test (which can be inspired by the test generated by evocrash or be completely different).
>
> Questions
> ========
>
> The question we need to answer is whether we think it coud be useful or not. Are there cases where it can be useful?
>
> For example if we imagine someone not knowing the xwiki code base well, maybe this can help him/her understand the bug in a simpler way than just having to read/find where the problem is in the code base?
>
> I’ve started the work of trying to evaluate how useful it could be on the XWiki project by analyzing the examples above. I've only analyzed 2 ATM (see the links in the jira comments) and I’d be interested to have your analysis too.
>
> WDYT?
>
> Thanks
> -Vincent



--
Thomas Mortagne
Reply | Threaded
Open this post in threaded view
|

Re: [Brainstorming] EvoCrash usage on XWiki?

vmassol
Administrator
Hi Thomas,

> On 6 Apr 2018, at 11:57, Thomas Mortagne <[hidden email]> wrote:
>
> Not sure what to think of those examples.
>
> The theory sounds nice and it probably produce nice result when
> talking about one method with a few inputs and an output but for
> anything more complex (like anything involving components) I feel you
> end up with a generated test which is a lot harder to read than simply
> looking at the code.

As I said in my original mail, I think that really depends about who handles the issue (a long time committer who’s being working for years on the code and knows it very well) or someone new who doesn’t know the code well.

Also, it really depends on the context. Imagine the following example:you have a NPE stack trace in production and the line where the NPE is has several variables that can be null. Just looking at the code will not tell you what the problem is. But the test will and you can even put a breakpoint to know which variable is actually null.

> * https://jira.xwiki.org/browse/XWIKI-13031: does not produce the same
> error at all. Looks like evocash just try to produce any
> ClassCastException but does not care about the actual types in it
> which is quite useless (plus reading that test requires a lot of
> effort just to conclude that the type of the field in the document is
> not String…)

I’ve not tested them all but all the ones I’ve tested do reproduce correctly the stack trace (that’s the point of evocrash so there’s no way it’ll generate a test that doesn’t reproduce the stack trace! ;)).

To give you more info, you run evocrash by giving it a stacktrace and a frame number (the number of stack trace lines you want it to reproduce).

For ex, on:

Caused by: java.lang.ClassCastException: org.apache.lucene.document.Field cannot be cast to java.lang.String
        at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getWikiReference(SolrEntityReferenceResolver.java:93) ~[xwiki-platform-search-solr-api-7.4.jar:na]
        at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getEntityReference(SolrEntityReferenceResolver.java:70) ~[xwiki-platform-search-solr-api-7.4.jar:na]
        at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:63) ~[xwiki-platform-search-solr-api-7.4.jar:na]
        at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:46) ~[xwiki-platform-search-solr-api-7.4.jar:na]
        at org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:48) ~[xwiki-platform-search-solr-api-7.4.jar:na]
        at org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:38) ~[xwiki-platform-search-solr-api-7.4.jar:na]
        […]

If you run evocrash with frame = 5, it’ll generate a test that, when executed lead to:

Caused by: java.lang.ClassCastException: org.apache.lucene.document.Field cannot be cast to java.lang.String
        at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getWikiReference(SolrEntityReferenceResolver.java:93) ~[xwiki-platform-search-solr-api-7.4.jar:na]
        at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getEntityReference(SolrEntityReferenceResolver.java:70) ~[xwiki-platform-search-solr-api-7.4.jar:na]
        at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:63) ~[xwiki-platform-search-solr-api-7.4.jar:na]
        at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:46) ~[xwiki-platform-search-solr-api-7.4.jar:na]
        at org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:48) ~[xwiki-platform-search-solr-api-7.4.jar:na]

> * https://jira.xwiki.org/browse/XWIKI-14475: totally unreadable

Actually this test was not that hard to read for me. It just sets up some mocks. Evocrash can make it even simpler by presenting the mocks in a simpler way but that’s cosmetic and not the focus of evocrash just right now (it’s not the hard part). I’ve also raised something a bit related: https://github.com/STAMP-project/EvoCrash/issues/7

Feel free to raise new issues at https://github.com/STAMP-project/EvoCrash/issues/

> * https://jira.xwiki.org/browse/XRENDERING-422: not really useful but
> I doubt evocrash can do much about this kind of issue

Yes I agree.

> * https://jira.xwiki.org/browse/XWIKI-13196: this one could help a bit
> I guess but not sure it's really faster to execute evocash than
> looking at the code :)

Sure but it also give you the test which is nice. And depending on the line generating the NPE, it’s not always easy to know which variable is null by looking at the code.

> * https://jira.xwiki.org/browse/XWIKI-13916: that's a more complex way
> than it should to express that we call addTextAreaField for an already
> existing field yes it's useful

Thanks for the feedback.

There are plenty of limitations/issues with evocrash ATM. It’s good if we can identify cases where it generates stuff that could be useful. One direction that the devs are pursuing is looking at existing test classes to mimick how they’re written to produce the new tests. I don’t know how they’d be able to pull this off (that seems really hard to me ;)) but I know they’re working in this direction.

Thanks
-Vincent

>
> On Fri, Apr 6, 2018 at 10:44 AM, Vincent Massol <[hidden email]> wrote:
>> Hi devs,
>>
>> Context
>> ======
>>
>> Evocrash is a tool developed as part of the STAMP research project (https://www.stamp-project.eu/). Its goal is to take a stack trace (the use case is production stack traces) and generate a test that, when executed, reproduces the stack trace ;)
>>
>> It’s using Guided Genetic Algorithm to simplify the search space (which is pretty cool, isn’t it? ;)). More info at https://github.com/STAMP-project/EvoCrash and http://www.evocrash.org/
>>
>> Results
>> ======
>>
>> So far evocrash was able to generate tests for the following issues (I’ve pasted the generated test code as comment):
>> * https://jira.xwiki.org/browse/XRENDERING-422
>> * https://jira.xwiki.org/browse/XWIKI-14475
>> * https://jira.xwiki.org/browse/XWIKI-13031
>> * https://jira.xwiki.org/browse/XWIKI-13196
>> * https://jira.xwiki.org/browse/XWIKI-13916
>>
>> Using
>> =====
>>
>> The use case I see is the following:
>> * A user reports a problem when using XWiki and create a jira issue with a stack trace
>> * An XWiki dev finds it and runs evocrash to generate a test reproducing the problem
>> * The XWiki dev uses the test to understand the reason (breakpoints can be put in the IDE) of the problem and writes a better test (which can be inspired by the test generated by evocrash or be completely different).
>>
>> Questions
>> ========
>>
>> The question we need to answer is whether we think it coud be useful or not. Are there cases where it can be useful?
>>
>> For example if we imagine someone not knowing the xwiki code base well, maybe this can help him/her understand the bug in a simpler way than just having to read/find where the problem is in the code base?
>>
>> I’ve started the work of trying to evaluate how useful it could be on the XWiki project by analyzing the examples above. I've only analyzed 2 ATM (see the links in the jira comments) and I’d be interested to have your analysis too.
>>
>> WDYT?
>>
>> Thanks
>> -Vincent
>
>
>
> --
> Thomas Mortagne

Reply | Threaded
Open this post in threaded view
|

Re: [Brainstorming] EvoCrash usage on XWiki?

Thomas Mortagne
Administrator
On Mon, Apr 9, 2018 at 1:38 PM, Vincent Massol <[hidden email]> wrote:

> Hi Thomas,
>
>> On 6 Apr 2018, at 11:57, Thomas Mortagne <[hidden email]> wrote:
>>
>> Not sure what to think of those examples.
>>
>> The theory sounds nice and it probably produce nice result when
>> talking about one method with a few inputs and an output but for
>> anything more complex (like anything involving components) I feel you
>> end up with a generated test which is a lot harder to read than simply
>> looking at the code.
>
> As I said in my original mail, I think that really depends about who handles the issue (a long time committer who’s being working for years on the code and knows it very well) or someone new who doesn’t know the code well.
>
> Also, it really depends on the context. Imagine the following example:you have a NPE stack trace in production and the line where the NPE is has several variables that can be null. Just looking at the code will not tell you what the problem is. But the test will and you can even put a breakpoint to know which variable is actually null.
>
>> * https://jira.xwiki.org/browse/XWIKI-13031: does not produce the same
>> error at all. Looks like evocash just try to produce any
>> ClassCastException but does not care about the actual types in it
>> which is quite useless (plus reading that test requires a lot of
>> effort just to conclude that the type of the field in the document is
>> not String…)
>
> I’ve not tested them all but all the ones I’ve tested do reproduce correctly the stack trace (that’s the point of evocrash so there’s no way it’ll generate a test that doesn’t reproduce the stack trace! ;)).
>
> To give you more info, you run evocrash by giving it a stacktrace and a frame number (the number of stack trace lines you want it to reproduce).
>
> For ex, on:
>
> Caused by: java.lang.ClassCastException: org.apache.lucene.document.Field cannot be cast to java.lang.String
>         at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getWikiReference(SolrEntityReferenceResolver.java:93) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>         at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getEntityReference(SolrEntityReferenceResolver.java:70) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>         at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:63) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>         at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:46) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>         at org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:48) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>         at org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:38) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>         […]
>
> If you run evocrash with frame = 5, it’ll generate a test that, when executed lead to:
>
> Caused by: java.lang.ClassCastException: org.apache.lucene.document.Field cannot be cast to java.lang.String
>         at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getWikiReference(SolrEntityReferenceResolver.java:93) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>         at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getEntityReference(SolrEntityReferenceResolver.java:70) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>         at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:63) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>         at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:46) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>         at org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:48) ~[xwiki-platform-search-solr-api-7.4.jar:na]

If you execute the test indicated in
https://jira.xwiki.org/browse/XWIKI-13031?focusedCommentId=98082&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-98082
you will get the following exception:

java.lang.ClassCastException:
org.xwiki.model.internal.reference.ExplicitStringEntityReferenceResolver
cannot be cast to java.lang.String

Sure you get the same stack trace from Exception type and methods
calls point of view but what the message describe is a very different
use case.

>
>> * https://jira.xwiki.org/browse/XWIKI-14475: totally unreadable
>
> Actually this test was not that hard to read for me. It just sets up some mocks. Evocrash can make it even simpler by presenting the mocks in a simpler way but that’s cosmetic and not the focus of evocrash just right now (it’s not the hard part). I’ve also raised something a bit related: https://github.com/STAMP-project/EvoCrash/issues/7
>
> Feel free to raise new issues at https://github.com/STAMP-project/EvoCrash/issues/
>
>> * https://jira.xwiki.org/browse/XRENDERING-422: not really useful but
>> I doubt evocrash can do much about this kind of issue
>
> Yes I agree.
>
>> * https://jira.xwiki.org/browse/XWIKI-13196: this one could help a bit
>> I guess but not sure it's really faster to execute evocash than
>> looking at the code :)
>
> Sure but it also give you the test which is nice. And depending on the line generating the NPE, it’s not always easy to know which variable is null by looking at the code.
>
>> * https://jira.xwiki.org/browse/XWIKI-13916: that's a more complex way
>> than it should to express that we call addTextAreaField for an already
>> existing field yes it's useful
>
> Thanks for the feedback.
>
> There are plenty of limitations/issues with evocrash ATM. It’s good if we can identify cases where it generates stuff that could be useful. One direction that the devs are pursuing is looking at existing test classes to mimick how they’re written to produce the new tests. I don’t know how they’d be able to pull this off (that seems really hard to me ;)) but I know they’re working in this direction.
>
> Thanks
> -Vincent
>
>>
>> On Fri, Apr 6, 2018 at 10:44 AM, Vincent Massol <[hidden email]> wrote:
>>> Hi devs,
>>>
>>> Context
>>> ======
>>>
>>> Evocrash is a tool developed as part of the STAMP research project (https://www.stamp-project.eu/). Its goal is to take a stack trace (the use case is production stack traces) and generate a test that, when executed, reproduces the stack trace ;)
>>>
>>> It’s using Guided Genetic Algorithm to simplify the search space (which is pretty cool, isn’t it? ;)). More info at https://github.com/STAMP-project/EvoCrash and http://www.evocrash.org/
>>>
>>> Results
>>> ======
>>>
>>> So far evocrash was able to generate tests for the following issues (I’ve pasted the generated test code as comment):
>>> * https://jira.xwiki.org/browse/XRENDERING-422
>>> * https://jira.xwiki.org/browse/XWIKI-14475
>>> * https://jira.xwiki.org/browse/XWIKI-13031
>>> * https://jira.xwiki.org/browse/XWIKI-13196
>>> * https://jira.xwiki.org/browse/XWIKI-13916
>>>
>>> Using
>>> =====
>>>
>>> The use case I see is the following:
>>> * A user reports a problem when using XWiki and create a jira issue with a stack trace
>>> * An XWiki dev finds it and runs evocrash to generate a test reproducing the problem
>>> * The XWiki dev uses the test to understand the reason (breakpoints can be put in the IDE) of the problem and writes a better test (which can be inspired by the test generated by evocrash or be completely different).
>>>
>>> Questions
>>> ========
>>>
>>> The question we need to answer is whether we think it coud be useful or not. Are there cases where it can be useful?
>>>
>>> For example if we imagine someone not knowing the xwiki code base well, maybe this can help him/her understand the bug in a simpler way than just having to read/find where the problem is in the code base?
>>>
>>> I’ve started the work of trying to evaluate how useful it could be on the XWiki project by analyzing the examples above. I've only analyzed 2 ATM (see the links in the jira comments) and I’d be interested to have your analysis too.
>>>
>>> WDYT?
>>>
>>> Thanks
>>> -Vincent
>>
>>
>>
>> --
>> Thomas Mortagne
>



--
Thomas Mortagne
Reply | Threaded
Open this post in threaded view
|

Re: [Brainstorming] EvoCrash usage on XWiki?

vmassol
Administrator


> On 9 Apr 2018, at 15:34, Thomas Mortagne <[hidden email]> wrote:
>
> On Mon, Apr 9, 2018 at 1:38 PM, Vincent Massol <[hidden email]> wrote:
>> Hi Thomas,
>>
>>> On 6 Apr 2018, at 11:57, Thomas Mortagne <[hidden email]> wrote:
>>>
>>> Not sure what to think of those examples.
>>>
>>> The theory sounds nice and it probably produce nice result when
>>> talking about one method with a few inputs and an output but for
>>> anything more complex (like anything involving components) I feel you
>>> end up with a generated test which is a lot harder to read than simply
>>> looking at the code.
>>
>> As I said in my original mail, I think that really depends about who handles the issue (a long time committer who’s being working for years on the code and knows it very well) or someone new who doesn’t know the code well.
>>
>> Also, it really depends on the context. Imagine the following example:you have a NPE stack trace in production and the line where the NPE is has several variables that can be null. Just looking at the code will not tell you what the problem is. But the test will and you can even put a breakpoint to know which variable is actually null.
>>
>>> * https://jira.xwiki.org/browse/XWIKI-13031: does not produce the same
>>> error at all. Looks like evocash just try to produce any
>>> ClassCastException but does not care about the actual types in it
>>> which is quite useless (plus reading that test requires a lot of
>>> effort just to conclude that the type of the field in the document is
>>> not String…)
>>
>> I’ve not tested them all but all the ones I’ve tested do reproduce correctly the stack trace (that’s the point of evocrash so there’s no way it’ll generate a test that doesn’t reproduce the stack trace! ;)).
>>
>> To give you more info, you run evocrash by giving it a stacktrace and a frame number (the number of stack trace lines you want it to reproduce).
>>
>> For ex, on:
>>
>> Caused by: java.lang.ClassCastException: org.apache.lucene.document.Field cannot be cast to java.lang.String
>>        at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getWikiReference(SolrEntityReferenceResolver.java:93) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getEntityReference(SolrEntityReferenceResolver.java:70) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:63) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:46) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:48) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:38) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        […]
>>
>> If you run evocrash with frame = 5, it’ll generate a test that, when executed lead to:
>>
>> Caused by: java.lang.ClassCastException: org.apache.lucene.document.Field cannot be cast to java.lang.String
>>        at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getWikiReference(SolrEntityReferenceResolver.java:93) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getEntityReference(SolrEntityReferenceResolver.java:70) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:63) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:46) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:48) ~[xwiki-platform-search-solr-api-7.4.jar:na]
>
> If you execute the test indicated in
> https://jira.xwiki.org/browse/XWIKI-13031?focusedCommentId=98082&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-98082
> you will get the following exception:
>
> java.lang.ClassCastException:
> org.xwiki.model.internal.reference.ExplicitStringEntityReferenceResolver
> cannot be cast to java.lang.String
>
> Sure you get the same stack trace from Exception type and methods
> calls point of view but what the message describe is a very different
> use case.

As I said evocrash replicates exactly the same stack trace, including the message. If you get something different it’s either because you’re executing on different code or whoever generated the test made a mistake copying it.

Thanks
-Vincent

>
>>
>>> * https://jira.xwiki.org/browse/XWIKI-14475: totally unreadable
>>
>> Actually this test was not that hard to read for me. It just sets up some mocks. Evocrash can make it even simpler by presenting the mocks in a simpler way but that’s cosmetic and not the focus of evocrash just right now (it’s not the hard part). I’ve also raised something a bit related: https://github.com/STAMP-project/EvoCrash/issues/7
>>
>> Feel free to raise new issues at https://github.com/STAMP-project/EvoCrash/issues/
>>
>>> * https://jira.xwiki.org/browse/XRENDERING-422: not really useful but
>>> I doubt evocrash can do much about this kind of issue
>>
>> Yes I agree.
>>
>>> * https://jira.xwiki.org/browse/XWIKI-13196: this one could help a bit
>>> I guess but not sure it's really faster to execute evocash than
>>> looking at the code :)
>>
>> Sure but it also give you the test which is nice. And depending on the line generating the NPE, it’s not always easy to know which variable is null by looking at the code.
>>
>>> * https://jira.xwiki.org/browse/XWIKI-13916: that's a more complex way
>>> than it should to express that we call addTextAreaField for an already
>>> existing field yes it's useful
>>
>> Thanks for the feedback.
>>
>> There are plenty of limitations/issues with evocrash ATM. It’s good if we can identify cases where it generates stuff that could be useful. One direction that the devs are pursuing is looking at existing test classes to mimick how they’re written to produce the new tests. I don’t know how they’d be able to pull this off (that seems really hard to me ;)) but I know they’re working in this direction.
>>
>> Thanks
>> -Vincent
>>
>>>
>>> On Fri, Apr 6, 2018 at 10:44 AM, Vincent Massol <[hidden email]> wrote:
>>>> Hi devs,
>>>>
>>>> Context
>>>> ======
>>>>
>>>> Evocrash is a tool developed as part of the STAMP research project (https://www.stamp-project.eu/). Its goal is to take a stack trace (the use case is production stack traces) and generate a test that, when executed, reproduces the stack trace ;)
>>>>
>>>> It’s using Guided Genetic Algorithm to simplify the search space (which is pretty cool, isn’t it? ;)). More info at https://github.com/STAMP-project/EvoCrash and http://www.evocrash.org/
>>>>
>>>> Results
>>>> ======
>>>>
>>>> So far evocrash was able to generate tests for the following issues (I’ve pasted the generated test code as comment):
>>>> * https://jira.xwiki.org/browse/XRENDERING-422
>>>> * https://jira.xwiki.org/browse/XWIKI-14475
>>>> * https://jira.xwiki.org/browse/XWIKI-13031
>>>> * https://jira.xwiki.org/browse/XWIKI-13196
>>>> * https://jira.xwiki.org/browse/XWIKI-13916
>>>>
>>>> Using
>>>> =====
>>>>
>>>> The use case I see is the following:
>>>> * A user reports a problem when using XWiki and create a jira issue with a stack trace
>>>> * An XWiki dev finds it and runs evocrash to generate a test reproducing the problem
>>>> * The XWiki dev uses the test to understand the reason (breakpoints can be put in the IDE) of the problem and writes a better test (which can be inspired by the test generated by evocrash or be completely different).
>>>>
>>>> Questions
>>>> ========
>>>>
>>>> The question we need to answer is whether we think it coud be useful or not. Are there cases where it can be useful?
>>>>
>>>> For example if we imagine someone not knowing the xwiki code base well, maybe this can help him/her understand the bug in a simpler way than just having to read/find where the problem is in the code base?
>>>>
>>>> I’ve started the work of trying to evaluate how useful it could be on the XWiki project by analyzing the examples above. I've only analyzed 2 ATM (see the links in the jira comments) and I’d be interested to have your analysis too.
>>>>
>>>> WDYT?
>>>>
>>>> Thanks
>>>> -Vincent
>>>
>>>
>>>
>>> --
>>> Thomas Mortagne
>>
>
>
>
> --
> Thomas Mortagne