fixminer_source/README.md

FixMiner

Code of FixMiner

Reference: FixMiner: Mining Relevant Fix Patterns for Automated Program Repair (Empirical Software Engineering, doi:10.1007/s10664-019-09780-z)

Citing FixMiner

You can cite FixMiner using the following bibtex:

@article{koyuncu2020fixminer,
  title={Fixminer: Mining relevant fix patterns for automated program repair},
  author={Koyuncu, Anil and Liu, Kui and Bissyand{\'e}, Tegawend{\'e} F and Kim, Dongsun and Klein, Jacques and Monperrus, Martin and Le Traon, Yves},
  journal={Empirical Software Engineering},
  pages={1--45},
  year={2020},
  publisher={Springer}
}

FixMiner

• I. Introduction of FixMiner
• II. Environment setup
• III. Replication Data
• IV. Step-by-Step execution

I. Introduction

Fixminer is a systematic and automated approach to mine relevant and actionable fix patterns for automated program repair.

II. Environment setup

• OS: macOS Mojave (10.14.3)

• JDK8: (important!)

• To mine from c code, srcml 1.0.0

• Download and configure Anaconda

• Create an python environment using the environment file

  conda env create -f environment.yml
  

• After creating the environment, activate it. It is containing necessary dependencies for redis, and python.

  source activate fixminerEnv
  

• Update the config.yml file with the corresponding paths in your computer. An example config.yml file could be found under

  fixminer_source/src/main/resources/config.yml
  

IV. Step-by-Step execution

Before running

• Update config file with corresponding user paths.

• Install the project with maven from root. (usage pom.xml)

  mvn clean install
  

• Active the conda environment from shell

  source activate fixminerEnv
  

In order to launch FixMiner, execute fixminer.sh

bash fixminer.sh [JOB] [CONFIG_FILE]
 e.g. bash fixminer.sh  dataset4c /Users/projects/release/fixminer_source/src/main/resources/config.yml

A log file (app.log) is created after every execution of the fixminer.sh. Please check this log file in order to access more information.

Job Types

FixMiner needs to specify a job to run.

1. dataset4j / dataset4c: Create a java/c mining dataset from the projects listed in subjects.csv or datasets.csv for c

2. richEditScript: Calls the jar file produced as the results as maven package to compute Rich edit scripts. This step can be invoke natively from java or using the Launcher with appropriate arguments.

   java -jar FixPatternMiner-1.0.0-jar-with-dependencies.jar  /Users/projects/release/fixminer_source/src/main/resources/config.yml RICHEDITSCRIPT
   

3. shapeSI: Search index creation for shapes. The output of this step is written to pairs folder which will be generated under datapath in config file

4. compare : Calls the jar file produced as the results as maven package to compare the trees. This step can be invoke natively from java or using the Launcher with appropriate arguments.

                    ```powershell
                    java -jar FixPatternMiner-1.0.0-jar-with-dependencies.jar  /Users/projects/release/fixminer_source/src/main/resources/config.yml COMPARE
   
                    ```     
   

5. cluster : Forms clusters of identical trees. The output of this step is written to shapes folder which will be generated under datapath in config file

6. stats: Calculate frequency statistics of the patterns under statsshapes.csv in datapath. The information is also written in app.log file.

7. exportPatterns : Export FixPatterns of APR integration under patterns folder located in datapath/

Data Viewer

The intermediate data provided computed during the steps are listed in directory datapath (see config file)

The data is stored in different formats. (e.g. pickle, redis db, csv, etc..)

Redis Commands

Connect to redis instance

    redis-cli -p 6399

We use 3 databases inside the redis, 0,1,2. DB 0 stores the richedit dumps, comparison indices DB 1 stores the filenames and their corresponding indices DB 2 stores the output of comparison, a.k.a same trees.

In order to switch between these database use the following command

    select 2

In order to trace the status of the stored rich edit scripts, use the following command

    hlen dump

In order to access the rich edit of a single hunk, first locate the key from DB 0. This command returns the exact name of the keys

keys *NAME_OF_THE_HUNK

keys *fuse_67b14b_04e5b1_fabric#fabric-client#src#main#java#org#fusesource#fabric#jolokia#facade#facades#ProfileFacade.java.txt_1

OUTPUT:
1) "MethodDeclaration/40/fuse_67b14b_04e5b1_fabric#fabric-client#src#main#java#org#fusesource#fabric#jolokia#facade#facades#ProfileFacade.java.txt_1"

Then, use the exact key in order to access the rich edit:

hget dump NAME_OF_THE_EXACT_KEY

hget dump MethodDeclaration/40/fuse_67b14b_04e5b1_fabric#fabric-client#src#main#java#org#fusesource#fabric#jolokia#facade#facades#ProfileFacade.java.txt_1

OUTPUT:
"INS MethodDeclaration@@public, void, MethodName:setConfiguration, String pid, Map<String,String> configuration,  @TO@ TypeDeclaration@@[public]ProfileFacade, [Profile, HasId] @AT@ 7279 @LENGTH@ 309\n---INS Modifier@@public @TO@ MethodDeclaration@@public, void, MethodName:setConfiguration, String pid, Map<String,String> configuration,  @AT@ 7279 @LENGTH@ 6\n---INS PrimitiveType@@void @TO@ MethodDeclaration@@public, void, MethodName:setConfiguration, String pid, Map<String,String> configuration,  @AT@ 7286 @LENGTH@ 4\n---INS SimpleName@@MethodName:setConfiguration @TO@ MethodDeclaration@@public, void, MethodName:setConfiguration, String pid, Map<String,String> configuration,  @AT@ 7291 @LENGTH@ 16\n---INS SingleVariableDeclaration@@String pid @TO@ MethodDeclaration@@public, void, MethodName:setConfiguration, String pid, Map<String,String> configuration,  @AT@ 7308 @LENGTH@ 10\n------INS SimpleType@@String @TO@ SingleVariableDeclaration@@String pid @AT@ 7308 @LENGTH@ 6\n------INS SimpleName@@pid @TO@ SingleVariableDeclaration@@String pid @AT@ 7315 @LENGTH@ 3\n---INS SingleVariableDeclaration@@Map<String,String> configuration @TO@ MethodDeclaration@@public, void, MethodName:setConfiguration, String pid, Map<String,String> configuration,  @AT@ 7320 @LENGTH@ 33\n------INS ParameterizedType@@Map<String,String> @TO@ SingleVariableDeclaration@@Map<String,String> configuration @AT@ 7320 @LENGTH@ 19\n---------INS SimpleType@@Map @TO@ ParameterizedType@@Map<String,String> @AT@ 7320 @LENGTH@ 3\n---------INS SimpleType@@String @TO@ ParameterizedType@@Map<String,String> @AT@ 7324 @LENGTH@ 6\n---------INS SimpleType@@String @TO@ ParameterizedType@@Map<String,String> @AT@ 7332 @LENGTH@ 6\n------INS SimpleName@@configuration @TO@ SingleVariableDeclaration@@Map<String,String> configuration @AT@ 7340 @LENGTH@ 13\n---INS VariableDeclarationStatement@@Map<String,Map<String,String>> configurations=getConfigurations(); @TO@ MethodDeclaration@@public, void, MethodName:setConfiguration, String pid, Map<String,String> configuration,  @AT@ 7365 @LENGTH@ 70\n------INS ParameterizedType@@Map<String,Map<String,String>> @TO@ VariableDeclarationStatement@@Map<String,Map<String,String>> configurations=getConfigurations(); @AT@ 7365 @LENGTH@ 32\n---------INS SimpleType@@Map @TO@ ParameterizedType@@Map<String,Map<String,String>> @AT@ 7365 @LENGTH@ 3\n---------INS SimpleType@@String @TO@ ParameterizedType@@Map<String,Map<String,String>> @AT@ 7369 @LENGTH@ 6\n---------INS ParameterizedType@@Map<String,String> @TO@ ParameterizedType@@Map<String,Map<String,String>> @AT@ 7377 @LENGTH@ 19\n------------INS SimpleType@@Map @TO@ ParameterizedType@@Map<String,String> @AT@ 7377 @LENGTH@ 3\n------------INS SimpleType@@String @TO@ ParameterizedType@@Map<String,String> @AT@ 7381 @LENGTH@ 6\n------------INS SimpleType@@String @TO@ ParameterizedType@@Map<String,String> @AT@ 7389 @LENGTH@ 6\n------INS VariableDeclarationFragment@@configurations=getConfigurations() @TO@ VariableDeclarationStatement@@Map<String,Map<String,String>> configurations=getConfigurations(); @AT@ 7398 @LENGTH@ 36\n---------INS SimpleName@@configurations @TO@ VariableDeclarationFragment@@configurations=getConfigurations() @AT@ 7398 @LENGTH@ 14\n---------INS MethodInvocation@@MethodName:getConfigurations:[] @TO@ VariableDeclarationFragment@@configurations=getConfigurations() @AT@ 7415 @LENGTH@ 19\n---INS IfStatement@@if (configurations != null) {  configurations.put(pid,configuration);  setConfigurations(configurations);} @TO@ MethodDeclaration@@public, void, MethodName:setConfiguration, String pid, Map<String,String> configuration,  @AT@ 7444 @LENGTH@ 138\n------INS InfixExpression@@configurations != null @TO@ IfStatement@@if (configurations != null) {  configurations.put(pid,configuration);  setConfigurations(configurations);} @AT@ 7448 @LENGTH@ 22\n---------INS SimpleName@@configurations @TO@ InfixExpression@@configurations != null @AT@ 7448 @LENGTH@ 14\n---------INS Operator@@!= @TO@ InfixExpression@@configurations != null @AT@ 7462 @LENGTH@ 2\n---------INS NullLiteral@@null @TO@ InfixExpression@@configurations != null @AT@ 7466 @LENGTH@ 4\n------INS Block@@ThenBody:{  configurations.put(pid,configuration);  setConfigurations(configurations);} @TO@ IfStatement@@if (configurations != null) {  configurations.put(pid,configuration);  setConfigurations(configurations);} @AT@ 7472 @LENGTH@ 110\n---------INS ExpressionStatement@@MethodInvocation:configurations.put(pid,configuration) @TO@ Block@@ThenBody:{  configurations.put(pid,configuration);  setConfigurations(configurations);} @AT@ 7486 @LENGTH@ 39\n------------INS MethodInvocation@@configurations.put(pid,configuration) @TO@ ExpressionStatement@@MethodInvocation:configurations.put(pid,configuration) @AT@ 7486 @LENGTH@ 38\n---------------INS SimpleName@@Name:configurations @TO@ MethodInvocation@@configurations.put(pid,configuration) @AT@ 7486 @LENGTH@ 14\n---------------INS SimpleName@@MethodName:put:[pid, configuration] @TO@ MethodInvocation@@configurations.put(pid,configuration) @AT@ 7501 @LENGTH@ 23\n------------------INS SimpleName@@pid @TO@ SimpleName@@MethodName:put:[pid, configuration] @AT@ 7505 @LENGTH@ 3\n------------------INS SimpleName@@configuration @TO@ SimpleName@@MethodName:put:[pid, configuration] @AT@ 7510 @LENGTH@ 13\n---------INS ExpressionStatement@@MethodInvocation:setConfigurations(configurations) @TO@ Block@@ThenBody:{  configurations.put(pid,configuration);  setConfigurations(configurations);} @AT@ 7538 @LENGTH@ 34\n------------INS MethodInvocation@@setConfigurations(configurations) @TO@ ExpressionStatement@@MethodInvocation:setConfigurations(configurations) @AT@ 7538 @LENGTH@ 33\n---------------INS SimpleName@@MethodName:setConfigurations:[configurations] @TO@ MethodInvocation@@setConfigurations(configurations) @AT@ 7538 @LENGTH@ 33\n------------------INS SimpleName@@configurations @TO@ SimpleName@@MethodName:setConfigurations:[configurations] @AT@ 7556 @LENGTH@ 14\n"  

Or use the following command to access specialized trees:

hgetall NAME_OF_THE_EXACT_KEY

hgetall MethodDeclaration/40/fuse_67b14b_04e5b1_fabric#fabric-client#src#main#java#org#fusesource#fabric#jolokia#facade#facades#ProfileFacade.java.txt_1

OUTPUT:
1) "targetTree"
2) "[(55@@[(31@@)][(31@@)][(31@@)][(31@@[(44@@)][(44@@)])][(31@@[(44@@[(74@@)][(74@@)][(74@@)])][(44@@)])][(31@@[(60@@[(74@@)][(74@@)][(74@@[(74@@)][(74@@)][(74@@)])])][(60@@[(59@@)][(59@@)])])][(31@@[(25@@[(27@@)][(27@@)][(27@@)])][(25@@[(8@@[(21@@[(32@@)][(32@@[(42@@)][(42@@)])])])][(8@@[(21@@[(32@@[(42@@)])])])])])])]"
3) "actionTree"
4) "[(100@@[(100@@)][(100@@)][(100@@)][(100@@[(100@@)][(100@@)])][(100@@[(100@@[(100@@)][(100@@)][(100@@)])][(100@@)])][(100@@[(100@@[(100@@)][(100@@)][(100@@[(100@@)][(100@@)][(100@@)])])][(100@@[(100@@)][(100@@)])])][(100@@[(100@@[(100@@)][(100@@)][(100@@)])][(100@@[(100@@[(100@@[(100@@)][(100@@[(100@@)][(100@@)])])])][(100@@[(100@@[(100@@[(100@@)])])])])])])]"
5) "shapeTree"
6) "[(31@@[(83@@)][(39@@)][(42@@)][(44@@[(43@@)][(42@@)])][(44@@[(74@@[(43@@)][(43@@)][(43@@)])][(42@@)])][(60@@[(74@@[(43@@)][(43@@)][(74@@[(43@@)][(43@@)][(43@@)])])][(59@@[(42@@)][(32@@)])])][(25@@[(27@@[(42@@)][(-1@@)][(33@@)])][(8@@[(21@@[(32@@[(42@@)][(42@@[(42@@)][(42@@)])])])][(21@@[(32@@[(42@@[(42@@)])])])])])])]"

After executing the shapeSI step, the rich edit scripts to be compared are stored in a key in DB 0. Use the following command to verify number of comparison to be made.

This command can also be used in order to progress the compare step. When the comparison is completed the following command will return 0.

scard compare

Pickle

The see content of the .pickle file the following script could be used.

 import pickle as p
 import gzip
 def load_zipped_pickle(filename):
    with gzip.open(filename, 'rb') as f:
        loaded_object = p.load(f)
        return loaded_object

Usage

result = load_zipped_pickle('code/LANGbugReportsComplete.pickle')
# Result is pandas object which can be exported to several formats
# Details on how to export is listed in offical library documentation
# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html