Measure Precision/Generalization-plugin problem.

ReM

I have used iDHM on the BPI 2013 Incident Log (separated into a test and training set) and let it produce a Petri Net. I exported the Net and used the "Replay a log on a Petri Net for conformance"-plugin (with the test log now and not punishing improper finish) to measure the fitness. Afterwards, I tried to use the "Measure Precision/Generalization"-plugin to get the other two quality dimensions. However, it's been running for over 30 hours by now and still not finished.

On the process model produced by the Alpha-R plugin, it only takes a few seconds. On the gigantic Petri Net produced/translated from ETM, it took roughly 3-10 hours (left it running overnight so I can't say for certain).

Can the plugin "hang up" with some sort of endless loops? I used the It's still working as indicated by the task manager with 12gb of RAM and around 30-40% CPU.

One problem I initially had, was that the replay on the Petri Net for iDHM only worked very haphazardly. It more often than not produced a failed output, but sometimes would work without any error. The Petri Net isn't very complex, so I'm unsure whether there is a problem and I should stop the plugin or give it more time.

Can't attach the picture, so I uploaded it to an image hoster:
https://abload.de/img/heuristicmodellbvjxl.png

I also can't attach the event logs, since they are apparently too big. Can anyone help me in any way or has some information? Is there another way to measure Fitness / Generalisation or a Petri Net that yields the same values? I try to compare various discovery methods, so I need to use the same method for all of them.

I would gladly upload the eventlogs on a file hoster, if someone else would be willing to test it. All I need are both values, so I can complete my bachelor thesis.

Thank you in advance.

JBuijs

Hi ReM,

the 'calculate precision/generalization through escaping edges' (or similar) plug-in is indeed known to block in certain cases.
The 'calculate precision using ETC-1' (or similar, not sure about the exact phrasing), seems to be more robust.

Hope this helps!
And good luck with your bachelor's thesis (feel free to share it here under https://www.win.tue.nl/promforum/categories/research).

ReM

Thank you for the reply.

I assume you mean "Check Precision based on Align-ETConformance" and then using the 1-Align Precision? It yields different values, however. I'm not sure which are preferable. For the ETM it gives me 0.6962 for precision, while the former method calculated 0,434. That seems like a big difference to me. Is there a way to make them comparable?

Also, how can I get the Generalization without the other plugin? The "Anti-Alignment Precision/Generalization"-Plugin fails to output anything. It starts the basic wizard, then calculates and then starts the Set Parameter wizard. When I click continue it just stops without error message or output and never indicates and processing (no green task, it stays black).
edit: I just started the "Anti-Alignment Precision/Generalization"-plugin that also requires a replay of the log. I'm not sure if it's working as the progress bar is stuck at 2 lines. I will leave it on for a few hours and check back.

Can the "Measure Precision/Generalization"-plugin misbehavior maybe manually be fixed or circumvented in some capacity?

I'm thankful for any further answers, as I need help to finish my paper (in due time).

ReM

It didn't complete within 8 hours, leaving me with no option to compute the generalization (at least for iDHM).  I'm quite clueless what to do now.

JBuijs

Hi, the two precision plug-ins use different calculations. Hence one is more robust than the other, but their results indeed differ.

Not sure about the generalization plug-in, you might want to contact the authors directly.

ReM

Using the ProM Light build solved all of my issues.

JBuijs

Thanks for letting us know!!!