Convert CSV to XES using command line

hypermau

Hello, I've use the import tool to convert CSV to XES using the UI.

A couple of question:

1. do you know if there is alredy a script to convert CSV to XES using  command line?

2. where I can find the source code of the converter ? I couldn't find it in the /packages dir.

https://svn.win.tue.nl/repos/prom/Packages/

Thank you

fmannhardt

Hi, I created a command line version of the CSV importer. It is located here:
https://svn.win.tue.nl/repos/prom/Packages/Log/Trunk/src/org/processmining/log/csvimport/CSVConversionCLI.java
The rest of the source code is also there.

Unfortunately, there are some links to the ProM framework, therefore, it is not that easy to call it outside of the ProM framework. Would be cool to have it as a standalone application in the future.

Until then, you can use the two attached files to call it from the command line. Just copy those files in the ProM (nightly builds) installation directory, make sure the current Log package is installed, and call the CSV importer via the CSV.bat file. It will startup ProM in CLI mode and print out help:

\ProM\Installation\prom-nightly>CSV.bat
usage: mpe [CSVFILE] [-complete <completionColumn>] [-event <eventColumn>]
       [-h] [-start <startColumn>] [-trace <traceColumn>] [-xes
       <filename>]
 -complete <completionColumn>
 -event <eventColumn>
 -h                             help
 -start <startColumn>
 -trace <traceColumn>
 -xes <filename>

DStekel

Dear Felix,

I am trying to get this command working, but do not quite know what I should fill in for the parameters. 

What I have done so far:
- Installed Prom 6.7 and added the two files you mentioned in the referenced topic to the install directory: (C:\Users\dst\ProM 6.7).

- Tried to apply the command on one of my test csv files. I assume that the other parameters need to be column names?? Example command:

C:\Users\dst\ProM 6.7>CSV.bat "C:\Workflow logs\AanvraagFacilitaireDienst.csv" -start "Eind" -event "Taak - Omschrijving" -trace "Dos.item" -xes "C:\Workflow logs\AanvraagFacilitaireDienstNew.xes"

However, the resulting xes file in not a valid file (Not xml-like when I open it in notepad )

Furthermore, I retrieve the following error when running: 

[BEGIN CONSOLE OUTPUT]

Starting CLI execution engine...

Parameters: [org.processmining.log.csvimport.CSVConversionCLI, C:\Workflow logs\AanvraagFacilitaireDienst.csv, -start, Eind, -event, Taak - Omschrijving, -trace, Dos.item, -xes, C:\Workflow logs\AanvraagFacilitaireDienstNew.xes]

Sorting CSV file (0,01 MB) by case and time using maximal 1092 MB of memory ...

Finished sorting in 0,27 seconds

Reading cases ...

java.lang.NoSuchMethodException: org.deckfour.xes.factory.XFactoryNaiveImpl.commit()

        at java.lang.Class.getMethod(Unknown Source)

        at org.processmining.log.csvimport.CSVConversion.commitFactoryIfNeeded(CSVConversion.java:406)

        at org.processmining.log.csvimport.CSVConversion.convertCSV(CSVConversion.java:378)

        at org.processmining.log.csvimport.CSVConversion.doConvertCSVToXES(CSVConversion.java:189)

        at org.processmining.log.csvimport.CSVConversionCLI.parseCSV(CSVConversionCLI.java:139)

        at org.processmining.log.csvimport.CSVConversionCLI.main(CSVConversionCLI.java:88)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

        at java.lang.reflect.Method.invoke(Unknown Source)

        at org.processmining.cli.CLISimpleStarter.main(CLISimpleStarter.java:34)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

        at java.lang.reflect.Method.invoke(Unknown Source)

        at org.processmining.framework.boot.Boot.boot(Boot.java:327)

        at org.processmining.framework.boot.Boot.boot(Boot.java:294)

        at org.processmining.cli.CLISimpleStarter.main(CLISimpleStarter.java:41)

Finished reading cases in 0 seconds.

start serializing log to XES.XML

finished serializing log (36 msec.)

Log converted successfully!
[END CONSOLE OUTPUT]

Could you give me some tips on how I should use the command?

fmannhardt

Hi, it seems that you have found a bug. Actually that exception should never have been thrown since it is caught in the latest version on the SVN. Would you mind trying with the nightly build version? 

Not sure if that was broken in ProM 6.7. Sorry about that, but we cannot fix bugs in the released version to ensure results being reproducible (and bugs as well)

DStekel

Thanks for you response!

After using the nightly-night of 29 January, this is my output:
[Begin Console Output]
...
Skipping: /C:/Users/dst/ProM-nightly-20180129-1.7/ProM-nightly-20180129-1.7/lib/lpsolve55j.jar while scanning for plugins.

>>> Scanning for plugins took 21.736 seconds

Starting CLI execution engine...

Parameters: [org.processmining.log.csvimport.CSVConversionCLI, C:\Workflow logs\AanvraagFacilitaireDienst.csv, -start, Eind, -event, Taak - Omschrijving, -trace, Dos.item, -xes, C:\Workflow logs\AanvraagFacilitaireDienstNew.xes]

Sorting CSV file (0,01 MB) by case and time using maximal 1092 MB of memory ...

Finished sorting in 0,29 seconds

Reading cases ...

Finished reading cases in 0 seconds.

start serializing log to XES.XML

finished serializing log (25 msec.)

Log converted successfully!
[End Console Output]

Unfortunately, the resulting .xes file contains gibberish: 

‹       í][sÚ8~Þþ

-Oí,˲±ñ$í¤·™Ìô2³¥»}UŒ0JŒÍØ’þúJ&æÒ:Œ!¾(“É#KGú>}çø¢‹7÷ÌIÑÀ¿lÁŽÚÄw‚!õÝËÖ÷ÁGÅj7¯_\ü­(`0¦Q€1ŽÀ

!>p‰OBÌÈ,(6&àë”ø?>|½ qøÐ×8?

ÂIåÑâÂ?> )öèOÌx@0Z~Á-DûC¯¼Àå‚»ðÃkKìócâ³ÎúàVõÇîÙ€wo]ä·vn•ú÷ùgAÞy<ÇÔÃ7Ü£0˜€1cS»Û],€º'Q'Ýh°8²í\qdD0›…$ºlù$â¾S0c!½™1µ@bh³oNëõ‹¿.È=#¾8

|<!—­OtDœÇ#-0

ɈÞ_¶¼õ¡YH/[MäV•Ä'q;WeÅWÜv«›v’lØgñ™¦E±Vß¾C¦lmØIdÚ~,¹eÞñpÑ%á£ýs>!À,Z{G¢Õ lñý•^®k«/ïÁÊ¿€…˜·^LÐWiöÀÊ•öºäò$Zßu~k˜coÆOz…ýyˆ±û;ԣӐ¼§ÜW¬ãDó¥

nÓ!üÃÖ4M‡z\‹û\ôF|Ú®wå0J€ÞÒÛȇtÄÖm¹£‰G–ç)÷G\ëÿ ¼yÁbUªüÇ„IÁÍ3¼‚Ђ)£·ÃUy5­ä$œžË0¾ã-ú:‰t;ç߯ê½%ÜÚV›öv)XúÔ‹

–^M3’ôí©&joéá ÍÔêT|J‡,½OÉ }‹g½`°½†jÓ¯ÊfѪðÆÜú­Ù«ñ݉ÿ³5Κ‰þ¨RHK

ù’°,%ˆÀx'Óõ4UaOQMµokº

Q‡›øGÕl5u¢¤,±ûòIO²‡)yrd³¼ø8 7ÙpÑ]á-ò>ÎB4^%±(äó%Ø-„ZÏÈ C™w

w÷Œ˜ÛéÛÓïš·v6ã^\oÀ¡¼€³ld5p×ëùÀ2ˆ{ŠEO3Ðy—½'!zF¼írm-Ö¹^NØ!Í6`³awJ…©§=k ;Žø(ªWo¡Û=ÊeZÝôS®nq¤5fÅ

èÕÈ%ËâÃ]3èt¸Þ\r̉ųkúé'˜©¼ÕËÁ4YDuÐ\+ˆ`Žgºõ¤¿è&I³H“Ù™ý¨FöCÓÙ†j«

Ï~|áíøîNÌõâ±ÛH5†£;2«¦0LóoY¥¡±Ö—†kº

!‡[}°–{•Ô3VÉM¢ÞV°Hàø,Q.®Ï±¸®Q¶ÑŸO³ˆ—,èíOê ˜ó¸Ìy»`ø†x› 4)HAØ·

s7ϱ‚°ƒ(¢\D32Y§RõxÒq—¸3ÑÒ‰¼‘àåæh¼Ú‡‹q\£›Üƒõ!´š’ÛºŽÚ†Y„쐩’±Q¡i¯R±ÐA²ê0V‰…òË*$e•”UÏGdz_ʪ¢d•¡×‡ÐjBH‰¬‚yob‘²ªÄl$eUV‰…òÊ*SË™­úNYiLêªjé*ÓÖŒÚ몔KÿE

++NÀCqf83Ï= )Ý]

=æ=˜x+"ÚJJw°ßæo[ºv4ïIÕV:®«’j;ãrË6“ÿJŽkÇY9Ø(‹Ì$3ÇL;Ñ^.¦Ñó3M–S$Ó|"]Í™Ó/)SÉà³ZÁg¯ ÷Jœ8øì+š5€†>µÂn–(†.³‡W

³£‚O£m©½¶ÑïËà³ôï`®“ÁgšQa×$ÇU‚ãdôY>jjBô™íI5ÇEŸýÛ5¡*}V+ú´lXÿ[ÊN}B_úìÙºn돩1XºÌ^©ÌŽRf°­jõS'ªŒ>«Íu2úLs¢ZØn$ÇU‚ãdôY>jjBô™íI5GEŸºjÉÇÄeðùŒw|èõ¿ôyÒç™Dì©ÄýhÈÖúEÅžÏOh5!$¨¶-Óh[Fª„’áaµÙ¨JáᙞgÊöa•X(¯¬2“—p×$”ºªRºJíe¾2¦ºê”I}îÌ¥°B¶Ñ·u½TIý=†·"¢­¤tg´{°×Fº|½ô¢ípª«’h;SN_5l­°Í"$ÅUâP꣚2§ÿ¬ÔÔ€œþN‘TsLð‰ Jî„—/HoVyîøîøK/ã‚Ð0¬Ñ~y‘+_Ú,~½b(jüâC³QÆñj5Ú£ÍÛ>m±R’4’¡ª'I8rã'‰Åk×åN~'C^¯ýZdru¼•i+¿

œgMÚ¨¶¸­ÙV~Ûs¥ì—UŠ­ü´Ô‹Br/¿<ºPCyo#ådQ/Œ3¿†akO¼{S÷Þ>­Òrž“ؐeÉ€·‘UÅ€AqŸR뙃¿èzûúÅ/²H@¦  

I have no idea why the output file looks like this. Any ideas?

DStekel

Nevermind, I discovered that the resulting xes. file is zipped. When extracting through 7-zip, the file looks correct!

Thank you fmannhardt!