azalea/fixminer_source
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

New features.

Browse Source
This commit is contained in:
Kui LIU
2017-11-06 20:10:17 +01:00
parent 1218230723
commit c695159c38
22 changed files with 843 additions and 3162 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Dataset/LiveStudy/Patches
+119
View File
@@ -0,0 +1,119 @@
BC_UNCONFIRMED_CAST
BC_UNCONFIRMED_CAST_OF_RETURN_VALUE Comments: no any useful common fix patterns.
EI_EXPOSE_REP
EI_EXPOSE_REP2
IS2_INCONSISTENT_SYNC
NM_METHOD_NAMING_CONVENTION
DE_MIGHT_IGNORE Compile failed.
DLS_DEAD_LOCAL_STORE Compile failed
NP_LOAD_OF_KNOWN_NULL_VALUE
DM_CONVERT_CASE To be improved
DM_BOXED_PRIMITIVE_FOR_PARSING
DM_DEFAULT_ENCODING
DM_FP_NUMBER_CTOR
DM_NUMBER_CTOR
ES_COMPARING_STRINGS_WITH_EQ Comments: ignored
MS_PKGPROTECT
MS_SHOULD_BE_FINAL Comments: failed to checkstyle.
NP_NONNULL_RETURN_VIOLATION
NP_NULL_ON_SOME_PATH
NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE
NP_PARAMETER_MUST_BE_NONNULL_BUT_MARKED_AS_NULLABLE
OBL_UNSATISFIED_OBLIGATION
OBL_UNSATISFIED_OBLIGATION_EXCEPTION_EDGE
ODR_OPEN_DATABASE_RESOURCE
OS_OPEN_STREAM
OS_OPEN_STREAM_EXCEPTION_PATH
PZLA_PREFER_ZERO_LENGTH_ARRAYS
RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE
RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE
REC_CATCH_EXCEPTION
RI_REDUNDANT_INTERFACES
RV_RETURN_VALUE_IGNORED_BAD_PRACTICE
RV_RETURN_VALUE_IGNORED_NO_SIDE_EFFECT
SBSC_USE_STRINGBUFFER_CONCATENATION
SE_NO_SERIALVERSIONID
SF_SWITCH_NO_DEFAULT
SIC_INNER_SHOULD_BE_STATIC
SIC_INNER_SHOULD_BE_STATIC_ANON
SQL_PREPARED_STATEMENT_GENERATED_FROM_NONCONSTANT_STRING
ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD
UC_USELESS_CONDITION
UC_USELESS_OBJECT
UCF_USELESS_CONTROL_FLOW
UPM_UNCALLED_PRIVATE_METHOD
URF_UNREAD_FIELD
URF_UNREAD_PUBLIC_OR_PROTECTED_FIELD
UWF_FIELD_NOT_INITIALIZED_IN_CONSTRUCTOR
UWF_UNWRITTEN_FIELD
VA_FORMAT_STRING_USES_NEWLINE
WMI_WRONG_MAP_ITERATOR
rejected 1 https://github.com/apache/commons-io/pull/47
rejected 1 https://github.com/apache/commons-io/pull/45
rejected 1 https://github.com/apache/commons-lang/pull/298
5 https://github.com/apache/commons-lang/pull/297
improved 1 https://github.com/apache/commons-lang/pull/297 Comments: some patches failed to pass test cases, some patches led to compile failed, some patches led to checkstyle failed.
2 https://github.com/apache/commons-math/pull/69
1 https://github.com/apache/commons-math/pull/68 Comments: inner static class, there is no need to fix the violations in this kind of class.
2 https://github.com/apache/commons-math/pull/67 Comments: improve patches.
1 https://github.com/apache/commons-math/pull/66
rejected 1 https://github.com/apache/ant/pull/47
improved 1 https://github.com/apache/ant/pull/46
merged 2 https://github.com/apache/ant/pull/46
2 https://github.com/apache/ant/pull/45
rejected 3 https://github.com/apache/ant/pull/44
merged 1 https://github.com/apache/ant/pull/43
merged 5 https://github.com/apache/ant/pull/42 Comments: need to import new packages, unless the fixed patches will fail to compile.
merged 1 https://github.com/apache/ant/pull/41
1 https://github.com/apache/aries/pull/80
1 https://github.com/apache/aries/pull/79
1 https://github.com/apache/aries/pull/78
1 https://github.com/apache/aries/pull/77
1 https://github.com/apache/aries/pull/76
rejected 1 https://github.com/apache/camel/pull/2031 ShouldBeFinal
rejected 6 https://github.com/apache/camel/pull/2029 master5
merged 1 https://github.com/apache/camel/pull/2035 master7
merged 2 https://github.com/apache/camel/pull/2032 Useless
merged 2 https://github.com/apache/camel/pull/2025 master6
merged 1 https://github.com/apache/camel/pull/2024 master4
merged 1 https://github.com/apache/camel/pull/2023 master2
rejected 1 https://github.com/apache/camel/pull/2017 master3 Comments: compile failed, the return value of method invocation is not Boolean, it is a specific case.
merged 1 https://github.com/apache/camel/pull/2016 master
merged 6 https://github.com/apache/camel/pull/2038 boxedParsing
3 https://github.com/apache/cassandra/pull/165
4 https://github.com/apache/cassandra/pull/164
2 https://github.com/apache/cassandra/pull/163
Fixed 1 https://github.com/fangyidong/json-simple/pull/126 Comments: fixed in another version. https://github.com/RalleYTN/SimpleJSON
1 https://github.com/fangyidong/json-simple/pull/125
On hold 1 https://github.com/apache/mahout/pull/346
1 https://github.com/apache/mahout/pull/345
1 https://github.com/apache/mahout/pull/344
merged 2 https://github.com/apache/poi/pull/77 5
merged 4 https://github.com/apache/poi/pull/78 6
merged 2 https://github.com/apache/poi/pull/79 t
merged 1 https://github.com/apache/poi/pull/80 4
merged 4 https://github.com/apache/poi/pull/76 2
merged 30 https://github.com/apache/poi/pull/75 master
merged 1 https://github.com/apache/poi/pull/74 3
Some violations never happened in these projects:
Dataset/Patterns/Fix-Patterns
+131
View File
@@ -0,0 +1,131 @@
=====================================
=========Useful Fix Patterns=========
=====================================
==============Detailed===============
DM_CONVERT_CASE
(1) ADD a rule of Locale.ENGLISH into toLowerCase/toUpperCase.
RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE
(1) DEL Null-Check IfStatement,
(2) DEL Null-Check Conditional Expression.
REC_CATCH_EXCEPTION
(1) Change Exception to a specific exception class.
SE_NO_SERIALVERSIONID
(1) Insert a private static final field serialVersionUID.
VA_FORMAT_STRING_USES_NEWLINE
(1) DEL "$backslash$n",
(2) Replace "$backslash$n" with "n"
MS_SHOULD_BE_FINAL
(1) ADD a ``final'' modifier.
WMI_WRONG_MAP_ITERATOR
(1) Replace the KeySet iterating ForStatement with an EntrySet iterating ForStatement.
DM_NUMBER_CTOR
(1) Replace the NumberType ClassInstanceCreation with a NumberType.valueOf() method invocation.
SIC_INNER_SHOULD_BE_STATIC
(1) ADD a "static" modifier,
(2) DEL the buggy statements.
DE_MIGHT_IGNORE
(1) Change Exception to a specific exception class.
DM_BOXED_PRIMITIVE_FOR_PARSING
(1) Replace NumberType.valueOf() with NumberType.parseNumberType().
DM_FP_NUMBER_CTOR
(1) DEL the buggy statement,
(2) Replace Double or Float ClassInstanceCreation with (Double or Float).valueOf(),
(3) Remove the Double or Float ClassInstanceCreation from the buggy expression.
MS_PKGPROTECT
(1) DEL the buggy statement,
(2) DEL the ``public'' or ``protected'' modifier
(3) Replace the ``public'' or ``protected'' modifier with the ``private'' modifier.
(4) DEL the initialization expression.
ES_COMPARING_STRINGS_WITH_EQ
(1) Replace the ``=='' InfixExpression with a stringVar.equals() method invocation.
===============Abstract==============
NP_NULL_ON_SOME_PATH
(1) ADD a Null-check statement or expression.
RI_REDUNDANT_INTERFACES
(1) DEL the implemented Interface type.
NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE
(1) ADD a Null-check statement or expression,
(2) DEL the buggy statement.
UCF_USELESS_CONTROL_FLOW
(1) DEL the buggy statement,
(2) DEL the useless ConditionalExpression.
UC_USELESS_CONDITION
(1) UPD the operator of a ConditionalExpersion,
(2) UPD an sub-expression of a ConditionExpression.
NP_LOAD_OF_KNOWN_NULL_VALUE
(1) DEL Null-check expression or statement,
(2) Replace the null-known variable with NULL.
SBSC_USE_STRINGBUFFER_CONCATENATION
(1) Replace String type with StringBuilder, and replace plus operator of StringVarialbe with the append method of StringBuilder.
UC_USELESS_OBJECT
(1) DEL the buggy statement.
UPM_UNCALLED_PRIVATE_METHOD
(1) DEL the buggy statements.
NM_METHOD_NAMING_CONVENTION
(1) Update Method Name of MethodDeclaration.
URF_UNREAD_FIELD
(1) DEL the buggy statement.
=====================================
NP_NONNULL_RETURN_VIOLATION
(1) DEL the null-returned ReturnStatement.
(2) Replace NullLiteral with an Expression which returns an empty list.
PZLA_PREFER_ZERO_LENGTH_ARRAYS
(1) DEL the buggy statement,
√ (2) Replace NullLiteral with an empty array.
SIC_INNER_SHOULD_BE_STATIC_ANON
(1) DEL a parameter of a ClassInstanceCreation in an AnonymousClassDeclaration,
(2) DEL the statement contains an AnonymousClassDeclaration,
√ (3) Replace the AnonymousClassDeclaration with an defined Static Object of this class.
DLS_DEAD_LOCAL_STORE
(1) DEL the buggy statement.
DM_DEFAULT_ENCODING
√ (1) ADD a parameter (i.e., "UTF-8") into the ClassInstanceCreation of a Stream object.
(2) Replace ClassInstanceCreation with a MethodInvocation.
(3) DEL the buggy statement.
BC_UNCONFIRMED_CAST
(1) DEL the buggy statement,
(2) DEL the Cast Type,
(3) Replace CastExpression with NullLiteral.
URF_UNREAD_PUBLIC_OR_PROTECTED_FIELD
(1) ADD a ``final'' modifier,
√ (2) DEL the buggy statement.
RV_RETURN_VALUE_IGNORED_BAD_PRACTICE
(1) Replace boolean-returned statement with a a new MethodInvocation.
√ (2) ADD an IfStatement.
EI_EXPOSE_REP
(1) DEL the buggy statement,
* (2) Replace the being assigned Expression with a ConditionalExpression.
√ (3) Replace the Expression being assigned with other Expression
EI_EXPOSE_REP2
(1) DEL the buggy statement,
* (2) Replace the being assigned Expression with a ConditionalExpression.
√ (3) Replace the being assigned Expression with other Expression.
=====================================
==========Null Fix Patterns==========
=====================================
UWF_FIELD_NOT_INITIALIZED_IN_CONSTRUCTOR null.
OBL_UNSATISFIED_OBLIGATION null.
OS_OPEN_STREAM_EXCEPTION_PATH null.
SF_SWITCH_NO_DEFAULT null.
UWF_UNWRITTEN_FIELD null.
IS2_INCONSISTENT_SYNC null.
ODR_OPEN_DATABASE_RESOURCE null.
SQL_PREPARED_STATEMENT_GENERATED_FROM_NONCONSTANT_STRING null.
OBL_UNSATISFIED_OBLIGATION_EXCEPTION_EDGE null.
OS_OPEN_STREAM null.
NP_PARAMETER_MUST_BE_NONNULL_BUT_MARKED_AS_NULLABLE null.
=====================================
========Useless Fix Patterns=========
=====================================
BC_UNCONFIRMED_CAST_OF_RETURN_VALUE
(1) UPD T1 v1 = (T1) method_invocation() with T2 v1 = method_invocation(),
(2) DEL the buggy statement,
(3) Replace the buggy CastExpresion with a MethodInvocation.
ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD
(1) DEL the buggy statement.
RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE
(1) ADD a Null-check statement or expression.
Dataset/Patterns/Source-Code-Patterns-Unfixed-Violations
+107
View File
@@ -0,0 +1,107 @@
================================
=========Single Pattern=========
================================
SIC_INNER_SHOULD_BE_STATIC_ANON: new Type() { override some methods(...){...} }. Anonymous inner class declaration. 100%
DE_MIGHT_IGNORE: try{...} catch (Exception e) {...}, Exception is not any specified.
VA_FORMAT_STRING_USES_NEWLINE: "\n", this new line symbol is in a StringLiteral of a format method.
UCF_USELESS_CONTROL_FLOW: if (conditional expression) {empty statements.}.
UC_USELESS_CONDITION: if (conditional expressions) {...}.
UC_USELESS_OBJECT: T1 v1 = new T1().
PZLA_PREFER_ZERO_LENGTH_ARRAYS: return null.
EI_EXPOSE_REP: return variable;
REC_CATCH_EXCEPTION: try{...} catch (Exception e) {...}, Exception is not any specific exception.
SF_SWITCH_NO_DEFAULT: switch(exp) {case1:...case2:...}, without default statement.
UWF_UNWRITTEN_FIELD: return exp / variable.
SBSC_USE_STRINGBUFFER_CONCATENATION: stringVariable += stringExp,
RV_RETURN_VALUE_IGNORED_NO_SIDE_EFFECT: return method_invocation_exp. Method invocation statement returns a value, but does not assign the value to any variable. (High false positives.)
NP_NONNULL_RETURN_VIOLATION: return null. (High false positives.)
================================
========Multiple Patterns=======
================================
NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE: (1) for (File file : filesArray) {...} (2) for (int i =0; i < filesArray.length; i ++) {...}
DM_CONVERT_CASE: (1) stringExp.toLowerCase(), 24,897
(2)stringExp.toUpperCase(). 10,406
MS_PKGPROTECT: (1) public static final T1 v1 = initializationExp, 2442
(2) protected static final T1 v1 = initializationExp. 175
(3) protected static T1 v1 = initializationExp. 158
(4) public static T1 v1 = initializationExp. 262, other 46
SIC_INNER_SHOULD_BE_STATIC: (1) this.v1 = v2; 590
(2) v1 = v2; 170
(3) super(...). 213
BC_UNCONFIRMED_CAST: (1) T1 v1 = (T(1) v2/exp, (2) v1 = (T(1) v2/exp, (3) ((T(1) v2).exp.
ES_COMPARING_STRINGS_WITH_EQ: (1) stringExp1 == stringExp2 IfStatement, ReturnStatement, WhileStatement.
(2) stringExp1 != stringExp2
RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE: (1) if (exp == null ...) {...},
(2) if (exp != null ...) {...},
(3) exp == null ? exp1 : exp2,
(4) exp != null ? exp1 : exp2.
DM_NUMBER_CTOR: (1) new Long(...), (2)new Integer(...), (3)new Short(...), (4)new Byte(...), (5)new Char(...).
DM_FP_NUMBER_CTOR: (1) new Double(exp), (2)new Float(exp).
DM_BOXED_PRIMITIVE_FOR_PARSING: (1) Integer.valueOf(str), (2)Long.valueOf(str). Comments: it contains fewer patterns than findbugs.
EI_EXPOSE_REP2: (1) this.v1 = v2,
(2) v1 = v2. 3700
BC_UNCONFIRMED_CAST_OF_RETURN_VALUE: (1) T1 v1 = (T) method_invocation(),
(2) v1 = (T) method_invocation(),
(3) ((T) method_invocation()).exp.
ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD: (1) static_v1 = instance_exp1.method_invocation(),
(2) v1 = instance_exp1.method_invocation(),
(3) v1 = exp1, other variable assignment statement.
RV_RETURN_VALUE_IGNORED_BAD_PRACTICE: (1) File.mkdirs(),
(2) File.mkdir(),
(3) File.delete(),
(4) File.createNewFile(),
(5) other exp.method_invoation() returns a value.
SQL_PREPARED_STATEMENT_GENERATED_FROM_NONCONSTANT_STRING: (1) sqlExecutionStatement = StringLiteral1 + stringV1 + ... + StringLiteral2 + stringV2.
(2) sqlRelatedVariable = sqlRelatedMethodInvocation();
OS_OPEN_STREAM: (1) StreamType v = new StreamType(...), 2) streamVariable = new StreamType(new StreamType2(...)), e.g., 1. FileOutputStream fos = new FileOutputStream(outFile);
DM_DEFAULT_ENCODING: (1) new BufferedReader(new InputStreamReader(input)), 99+
(2) new StreamWriter(output);
(3) stringExp.getBytes(),
(4) new String(byteExp),
(5) byteExp.toString().
ODR_OPEN_DATABASE_RESOURCE: (1) conn.createStatement(),
(2) conn.createStatement.execute(...),
(3) PreparedStatement st = conn.prepareStatement(),
(4) DataBaseConnection conn = exp.getConnection(...).
NP_PARAMETER_MUST_BE_NONNULL_BUT_MARKED_AS_NULLABLE: (1) return method_invocation(...), 2) return class_instance_creation(...).
========================================
==========Inefficient Patterns==========
========================================
OBL_UNSATISFIED_OBLIGATION: (1) StreamType v = new StreamType(...),
(2) streamVariable = new StreamType(new StreamType2(...)),
(3) sqlStatementExp.executeMethod(). e.g., 1. FileOutputStream fos = new FileOutputStream(outFile); 2. ResultSet rs = sqlStmt.executeQuery().
OS_OPEN_STREAM_EXCEPTION_PATH: (1) StreamType v = new StreamType(...),
(2) streamVariable = new StreamType(...).
IS2_INCONSISTENT_SYNC: (1) return v1,
(2) if(conditionalExp) {...},
(3) sqlStmt = stringLiteral1 + stringExp + stringLiteral2. not very consistent.
OBL_UNSATISFIED_OBLIGATION_EXCEPTION_EDGE: (1) StreamType v = new StreamType(...),
(2) streamVariable = new StreamType(new StreamType2(...)),
(3) sqlStatementExp.executeMethod(). e.g., 1. FileOutputStream fos = new FileOutputStream(outFile); 2. ResultSet rs = sqlStmt.executeQuery().
WMI_WRONG_MAP_ITERATOR: for statement { ... MapVariable.get(keyExp) ... }. or NULL. Lack of enough information.
========================================
=====Redundant Inconsistent Pattern=====
========================================
MS_SHOULD_BE_FINAL: (1) public (28,302) static T1 v1 = exp. e.g., public static int maxStringLength = 65535.
(2) protected (6,719) static T1 v1 = exp.
URF_UNREAD_FIELD: (1) this.v1 = v2, (2)this.v1 = new T(), (3)private Type v = exp. Comments: it contains more patterns than findbugs. only pattern 3 is consistent.
URF_UNREAD_PUBLIC_OR_PROTECTED_FIELD: (1) this.v1 = v2,
(2) public/protected (static) T1 v1 = exp; Comments: only 2 is consistent.
(3) v1 = exp1.
========================================
=========Inconsistent Patterns=========
========================================
RI_REDUNDANT_INTERFACES: super(...). Comments: not very consistent. Locate on the position of super classes.
========================================
==============Null Pattern==============
========================================
===========Inaccurate Position==========
NM_METHOD_NAMING_CONVENTION: null. Comments: violate on title, but position locates on body.
SE_NO_SERIALVERSIONID: null.
UPM_UNCALLED_PRIVATE_METHOD: null. (method body)
UWF_FIELD_NOT_INITIALIZED_IN_CONSTRUCTOR: null. inaccurate positions.
==========Specific Source Code==========
DLS_DEAD_LOCAL_STORE: null. Variable assignment statements. Comments: specific source code.
NP_NULL_ON_SOME_PATH: null.
NP_LOAD_OF_KNOWN_NULL_VALUE: null.
RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE: null.
Dataset/a.txt
-2518
View File
File diff suppressed because it is too large Load Diff
src/main/java/edu/lu/uni/serval/FixPatternParser/violations/FixedViolationHunkParser.java
+2 -1
View File
@@ -6,6 +6,8 @@ import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import com.github.gumtreediff.tree.ITree;
@@ -272,7 +274,6 @@ public class FixedViolationHunkParser extends FixedViolationParser {
}
}
}
if (children.size() == 0) {
List<HierarchicalActionSet> hunkActionSets = violation.getActionSets();
/*
src/main/java/edu/lu/uni/serval/MultipleThreadsParser3/AkkaParser.java
+24 -21
View File
@@ -32,28 +32,31 @@ public class AkkaParser {
for (File violationT : violationTypes) {
if (violationT.isDirectory()) {
violationType = violationT.getName();
if (!violationType.equals("SF_SWITCH_NO_DEFAULT")) continue;
// input data
log.info("Get the input data...");
final List<MessageFile> msgFiles = getMessageFiles(Configuration.GUM_TREE_INPUT, violationType);
log.info("MessageFiles: " + msgFiles.size());
// output path
final String sourceCodeFilesPath = Configuration.ROOT_PATH + "UnfixedViolations_RQ3/" + violationType + "/";
FileHelper.deleteDirectory(sourceCodeFilesPath);
if (//violationType.equals("SF_SWITCH_NO_DEFAULT") ||
// violationType.equals("SE_NO_SERIALVERSIONID") ||
violationType.equals("DM_DEFAULT_ENCODING")) {
// input data
log.info("Get the input data...");
final List<MessageFile> msgFiles = getMessageFiles(Configuration.GUM_TREE_INPUT, violationType);
log.info("MessageFiles: " + msgFiles.size());
// output path
final String sourceCodeFilesPath = Configuration.ROOT_PATH + "UnfixedViolations_RQ3/" + violationType + "/";
FileHelper.deleteDirectory(sourceCodeFilesPath);
ActorSystem system = null;
ActorRef parsingActor = null;
int numberOfWorkers = 200;
final WorkMessage msg = new WorkMessage(0, msgFiles);
try {
log.info("Akka begins...");
system = ActorSystem.create("Mining-Pattern-System-" + violationType);
parsingActor = system.actorOf(ParseFixPatternActor.props(numberOfWorkers, sourceCodeFilesPath, violationType), "mine-pattern-actor-" + violationType);
parsingActor.tell(msg, ActorRef.noSender());
} catch (Exception e) {
system.shutdown();
e.printStackTrace();
ActorSystem system = null;
ActorRef parsingActor = null;
int numberOfWorkers = 200;
final WorkMessage msg = new WorkMessage(0, msgFiles);
try {
log.info("Akka begins...");
system = ActorSystem.create("Mining-Pattern-System-" + violationType);
parsingActor = system.actorOf(ParseFixPatternActor.props(numberOfWorkers, sourceCodeFilesPath, violationType), "mine-pattern-actor-" + violationType);
parsingActor.tell(msg, ActorRef.noSender());
} catch (Exception e) {
system.shutdown();
e.printStackTrace();
}
}
}
}
src/main/java/edu/lu/uni/serval/bugLocalization/BuggyTreeParser.java
-108
View File
@@ -1,108 +0,0 @@
package edu.lu.uni.serval.bugLocalization;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.List;
import com.github.gumtreediff.tree.ITree;
import edu.lu.uni.serval.FixPatternParser.Tokenizer;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.gumtree.regroup.SimpleTree;
import edu.lu.uni.serval.gumtree.regroup.SimplifyTree;
import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.violation.code.parser.ViolationSourceCodeTree;
/**
*
* Testing data
* localized bugs by FindBugs.
* Use their java file, use GumTree to parse it, simplify the buggy code.
* get all buggy code tokens.
*
* training data
*
*
* Word2Vec embeds data.
*
*
* @author kui.liu
*
*/
public class BuggyTreeParser {
public static void main(String[] args) {
List<String> data = readInfo("Dataset/localized-violations1.list");
StringBuilder bugTokens = new StringBuilder();
int i = 0;
for (String bug : data) {
i ++;
String[] info = bug.split(" : ");
String bugId = info[0].trim();
String fileName = info[2].trim();
int startLine = Integer.parseInt(info[3].trim());
int endLine = Integer.parseInt(info[4].trim());
bugId = bugId.substring(0, bugId.lastIndexOf("."));
String[] bugIdInfo = bugId.split("_");
String projectName = bugIdInfo[0];
String bugIdSt = bugIdInfo[1];
String path = "../FPM_Violations/Testing/projects/" + projectName + "/" + bugIdSt + "/buggy/";
if ("Chart".equals(projectName)) {
path += "source/";
} else {
path += "src/";
}
List<File> javaFiles = FileHelper.getAllFiles(path, ".java");
for (File javaFile : javaFiles) {
if (javaFile.getPath().endsWith(fileName)) {
ViolationSourceCodeTree alarmTree = new ViolationSourceCodeTree(javaFile, startLine, endLine);
alarmTree.extract();
List<ITree> matchedTrees = alarmTree.getViolationSourceCodeTrees();
SimpleTree simpleTree = new SimpleTree();
simpleTree.setLabel("Block");
simpleTree.setNodeType("Block");
List<SimpleTree> children = new ArrayList<>();
for (ITree matchedTree : matchedTrees) {
SimpleTree simpleT = new SimplifyTree().canonicalizeSourceCodeTree(matchedTree, null);
children.add(simpleT);
}
simpleTree.setChildren(children);
String tokens = Tokenizer.getTokensDeepFirst(simpleTree);
bugTokens.append(tokens + "\n");
System.out.println(i + "### " + tokens);
}
}
}
FileHelper.outputToFile(Configuration.EMBEDDING_DATA_TOKENS2, bugTokens, false);
}
private static List<String> readInfo(String file) {
List<String> info = new ArrayList<>();
String content = FileHelper.readFile(file);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
try {
while ((line = reader.readLine()) != null) {
info.add(line);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return info;
}
}
src/main/java/edu/lu/uni/serval/bugLocalization/FixedAlarms.java
-100
View File
@@ -1,100 +0,0 @@
package edu.lu.uni.serval.bugLocalization;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.Scanner;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.Exporter;
import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.utils.MapSorter;
public class FixedAlarms {
/**
*
unfixed alarms
321403
5336
MaxLength: 1508
* @param args
*/
public static void main(String[] args) {
String fileName = "unfixedAlarms";
String tokensFile = Configuration.ROOT_PATH + "Alarms_tokens/" + fileName + ".list"; // MaxLength: 1508
FileHelper.deleteFile(Configuration.ROOT_PATH + "Alarms_tokens/" + fileName + "Tokens.list");
int maxLength = 0;
StringBuilder tokensBuilder = new StringBuilder();
StringBuilder sizesBuilder = new StringBuilder();
Map<String, Integer> alarmTypes = new HashMap<>();
FileInputStream fis = null;
Scanner scanner = null;
int counter = 0;
try {
fis = new FileInputStream(tokensFile);
scanner = new Scanner(fis);
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
int colonIndex = line.indexOf(":");
String alarmType = line.substring(0, colonIndex);
if (alarmTypes.containsKey(alarmType)) {
alarmTypes.put(alarmType, alarmTypes.get(alarmType) + 1);
} else {
alarmTypes.put(alarmType, 1);
}
line = line.substring(colonIndex + 1);
colonIndex = line.indexOf(":");
String violationFileName = line.substring(0, colonIndex);
line = line.substring(colonIndex + 1);
colonIndex = line.indexOf(":");
String startLine = line.substring(0, colonIndex);
line = line.substring(colonIndex + 1);
colonIndex = line.indexOf(":");
String endLine = line.substring(0, colonIndex);
String tokens = line.substring(colonIndex + 1);
String[] tokensArray = tokens.split(" ");
int length = tokensArray.length;
if (length > maxLength) maxLength = length;
tokensBuilder.append(tokens + "\n");
sizesBuilder.append(length + "\n");
if (++ counter % 5000 == 0) {
FileHelper.outputToFile(Configuration.ROOT_PATH + "Alarms_tokens/" + fileName + "Tokens.list", tokensBuilder, true);
tokensBuilder.setLength(0);
}
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
try {
scanner.close();
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
System.out.println("MaxLength: " + maxLength);
FileHelper.outputToFile(Configuration.ROOT_PATH + "Alarms_tokens/" + fileName + "Tokens.list", tokensBuilder, true);
tokensBuilder.setLength(0);
FileHelper.outputToFile(Configuration.ROOT_PATH + "Alarms_tokens/" + fileName + "TokenSizes.csv", sizesBuilder, false);
MapSorter<String, Integer> sorter = new MapSorter<String, Integer>();
alarmTypes = sorter.sortByKeyAscending(alarmTypes);
String[] columns = { "Alarm Type", "amount" };
Exporter.exportOutliers(alarmTypes, new File(Configuration.ROOT_PATH + "Alarms_tokens/" + fileName + "TokenTypes.xls"), 1, columns);
// String s = "";
// for (Map.Entry<String, Integer> entry : alarmTypes.entrySet()) {
// s += entry.getKey() + " : " + entry.getValue() + "\n";
//
// }
// FileHelper.outputToFile(Configuration.ROOT_PATH + "Alarms_tokens/fixedAlarmTypesAmount.list", s, false);
}
}
src/main/java/edu/lu/uni/serval/bugLocalization/ProjectScanner.java
-195
View File
@@ -1,195 +0,0 @@
package edu.lu.uni.serval.bugLocalization;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import org.eclipse.jdt.core.dom.CompilationUnit;
import com.github.gumtreediff.tree.ITree;
import edu.lu.uni.serval.FixPattern.utils.ASTNodeMap;
import edu.lu.uni.serval.FixPattern.utils.Checker;
import edu.lu.uni.serval.FixPatternParser.CUCreator;
import edu.lu.uni.serval.FixPatternParser.Tokenizer;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.gumtree.GumTreeGenerator;
import edu.lu.uni.serval.gumtree.GumTreeGenerator.GumTreeType;
import edu.lu.uni.serval.gumtree.regroup.SimpleTree;
import edu.lu.uni.serval.gumtree.regroup.SimplifyTree;
import edu.lu.uni.serval.utils.FileHelper;
/**
* Scan the whole java project.
* Get the source code tokens of all statements.
*
* @author kui.liu
*
*/
public class ProjectScanner {
private int maxSize;
private int numberOfFiles;
private List<SimpleTree> allSimpleTrees = null;
private StringBuilder tokensBuilder;
private StringBuilder localizationsBuilder;
private int counter;
public ProjectScanner() {
maxSize = Integer.parseInt(FileHelper.readFile(Configuration.MAX_TOKEN_VECTORS_SIZE_OF_SOURCE_CODE));
numberOfFiles = 0;
tokensBuilder = new StringBuilder();
localizationsBuilder = new StringBuilder();
counter = 0;
}
public void scanJavaProject(File[] projects, String outputLocalizeFile, String outputTokensFile, int limitation) {
for (File project : projects) {
scanJavaProject(project, outputLocalizeFile, outputTokensFile, limitation);
}
}
public void scanJavaProject(File javaProject, String outputLocalizeFile, String outputTokensFile, int limitation) {
List<File> files = new ArrayList<>();
files.addAll(FileHelper.getAllFiles(javaProject.getPath(), ".java"));
for (File file : files) {
if (file.getPath().toLowerCase().contains("test")) {
continue; // ignore all test-related java files.
}
ITree tree = new GumTreeGenerator().generateITreeForJavaFile(file, GumTreeType.EXP_JDT);
CUCreator cuCreator = new CUCreator();
CompilationUnit cUnit = cuCreator.createCompilationUnit(file);
getTokenVectorOfAllStatements(tree, cUnit, javaProject.getPath(), file.getPath(), outputLocalizeFile, outputTokensFile, limitation);
}
if (localizationsBuilder.length() > 0) {
numberOfFiles ++;
FileHelper.outputToFile(outputLocalizeFile + "Positions_" + numberOfFiles + ".list", localizationsBuilder, true);
FileHelper.outputToFile(outputTokensFile + "Tokens_" + numberOfFiles + ".list", tokensBuilder, true);
localizationsBuilder.setLength(0);
tokensBuilder.setLength(0);
}
}
private void getTokenVectorOfAllStatements(ITree tree, CompilationUnit unit, String projectName, String filePath, String outputLocalizeFile, String outputTokensFile, int limitation) {
String astNodeType = ASTNodeMap.map.get(tree.getType()); //ignore: SwitchCase, SuperConstructorInvocation, ConstructorInvocation
if ((astNodeType.endsWith("Statement") && !astNodeType.equals("TypeDeclarationStatement"))
|| astNodeType.equals("FieldDeclaration")) {
List<ITree> children = new ArrayList<>();
if (Checker.containsBodyBlock(astNodeType)) {
List<ITree> childrenList = tree.getChildren();
for (ITree child : childrenList) {
if (!child.getLabel().endsWith("Body")) {
children.add(child);
}
}
tree.setChildren(children);
} else {
children.addAll(tree.getChildren());
}
if (children.size() > 0) {
SimplifyTree simplifier = new SimplifyTree();
SimpleTree simpleTree = simplifier.canonicalizeSourceCodeTree(tree, null);
// project name: file name: line number
String tokens = Tokenizer.getTokensDeepFirst(simpleTree).trim();
String[] tokensArray = tokens.split(" ");
if (tokensArray.length <= maxSize) {
int position = tree.getPos();
int lineNum = unit.getLineNumber(position);
tokensBuilder.append(tokens).append("\n");
localizationsBuilder.append(projectName + ":" + filePath + "LineNum:" + lineNum + "\n"); //project name: file name: line number
++ counter;
if ( counter % limitation == 0) {
numberOfFiles ++;
FileHelper.outputToFile(outputLocalizeFile + "Positions_" + numberOfFiles + ".list", localizationsBuilder, true);
FileHelper.outputToFile(outputTokensFile + "Tokens_" + numberOfFiles + ".list", tokensBuilder, true);
localizationsBuilder.setLength(0);
tokensBuilder.setLength(0);
}
}
}
} else {
List<ITree> children = tree.getChildren();
for (ITree child : children) {
if (astNodeType.endsWith("Name")) continue;
if (Checker.isExpressionType(astNodeType) && !"LambdaExpression".equals(astNodeType)) continue;
getSimpleTreesOfAllStatements(child);
}
}
}
public void getSimpleTreesOfAllStatements(ITree tree) {
String astNodeType = ASTNodeMap.map.get(tree.getType()); //ignore: SwitchCase, SuperConstructorInvocation, ConstructorInvocation
if ((astNodeType.endsWith("Statement") && !astNodeType.equals("TypeDeclarationStatement"))
|| astNodeType.equals("FieldDeclaration")) {
SimpleTree simpleTree = new SimpleTree();
List<SimpleTree> children = getChildren(tree, astNodeType, simpleTree);
if (children != null) { // Ignore LabeledStatements and TryStatements
simpleTree.setNodeType(astNodeType);
simpleTree.setLabel(astNodeType);
simpleTree.setParent(null);
simpleTree.setChildren(children);
allSimpleTrees.add(simpleTree);
}
} else {
List<ITree> children = tree.getChildren();
for (ITree child : children) {
getSimpleTreesOfAllStatements(child);
}
}
}
private List<SimpleTree> getChildren(ITree tree, String astNodeType, SimpleTree parent) {
List<ITree> children = new ArrayList<>();
if (Checker.containsBodyBlock(astNodeType)) {
List<ITree> childrenList = tree.getChildren();
for (ITree child : childrenList) {
if (!child.getLabel().endsWith("Body")) {
children.add(child);
}
}
} else {
children.addAll(tree.getChildren());
}
if (children.size() == 0) {
return null;
}
List<SimpleTree> childrenSimpleTrees = new ArrayList<>();
for (ITree child : children) {
childrenSimpleTrees.add(getSimpleTree(child, parent));
}
return childrenSimpleTrees;
}
private SimpleTree getSimpleTree(ITree tree, SimpleTree parent) {
String astNodeType = ASTNodeMap.map.get(tree.getType());
SimpleTree simpleTree = new SimpleTree();
simpleTree.setNodeType(astNodeType);
List<ITree> children = tree.getChildren();
if (children.size() > 0) {
List<SimpleTree> subTrees = new ArrayList<>();
for (ITree child : children) {
subTrees.add(getSimpleTree(child, simpleTree));
}
simpleTree.setChildren(subTrees);
simpleTree.setLabel(astNodeType);
} else {
simpleTree.setLabel(tree.getLabel());
}
simpleTree.setParent(parent);
return simpleTree;
}
}
src/main/java/edu/lu/uni/serval/bugLocalization/Step8.java
-43
View File
@@ -1,43 +0,0 @@
package edu.lu.uni.serval.bugLocalization;
import java.io.File;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
/**
* Prepare testing data for evaluation.
*
* Parse java projects to get the token vectors of all statements.
*
* @author kui.liu
*
*/
public class Step8 {
public static void main(String[] args) {
// String outputLocalizeFile = Configuration.TEST_POSITION_FILE;
// String outputTokensFile = Configuration.TEST_DATA_FILE;
// FileHelper.deleteDirectory(outputLocalizeFile);
// FileHelper.deleteDirectory(outputTokensFile);
//
// int limitationOfTestingInstances = Integer.parseInt(FileHelper.readFile(Configuration.NUMBER_OF_TRAINING_DATA).trim()) / 10;
//
// File testProjects = new File(Configuration.TEST_INPUT);
// File[] projects = testProjects.listFiles();
// ProjectScanner scanner = new ProjectScanner();
// scanner.scanJavaProject(projects, outputLocalizeFile, outputTokensFile, limitationOfTestingInstances);
for (int i = 1; i <= 106; i ++) {
System.out.println("cd ../../" + i + "/buggy");
System.out.println("mvn package -DskipTests=true");
System.out.println("mv target/commons-math3-3.3-SNAPSHOT.jar ../../Math" + i + ".jar");
}
}
}
/*
cd ../../4/buggy
mvn package -Dmaven.test.skip=true
mv target/commons-lang3-3.2-SNAPSHOT.jar ../../Lang
*/
src/main/java/edu/lu/uni/serval/bugLocalization/test.java
-108
View File
@@ -1,108 +0,0 @@
package edu.lu.uni.serval.bugLocalization;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.utils.ListSorter;
public class test {
/*
Alarm Type :UPM_UNCALLED_PRIVATE_METHOD : 13
Alarm Type :SS_SHOULD_BE_STATIC : 2
Alarm Type :FE_FLOATING_POINT_EQUALITY : 1
Alarm Type :UWF_UNWRITTEN_FIELD : 2
Alarm Type :CN_IDIOM_NO_SUPER_CALL : 1
Alarm Type :SBSC_USE_STRINGBUFFER_CONCATENATION : 2
Alarm Type :RV_RETURN_VALUE_IGNORED_NO_SIDE_EFFECT : 2
Alarm Type :URF_UNREAD_PUBLIC_OR_PROTECTED_FIELD : 15
Alarm Type :SF_SWITCH_FALLTHROUGH : 1
Alarm Type :EI_EXPOSE_REP2 : 7
Alarm Type :REC_CATCH_EXCEPTION : 1
Alarm Type :MS_MUTABLE_ARRAY : 1
Alarm Type :FI_MISSING_SUPER_CALL : 1
Alarm Type :RI_REDUNDANT_INTERFACES : 1
Alarm Type :RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE : 5
Alarm Type :DB_DUPLICATE_SWITCH_CLAUSES : 1
Alarm Type :UCF_USELESS_CONTROL_FLOW : 6
Alarm Type :NP_DEREFERENCE_OF_READLINE_VALUE : 1
Alarm Type :RV_RETURN_VALUE_IGNORED_BAD_PRACTICE : 4
Alarm Type :DLS_DEAD_LOCAL_STORE : 41
Alarm Type :EI_EXPOSE_REP : 1
Alarm Type :URF_UNREAD_FIELD : 19
Alarm Type :ODR_OPEN_DATABASE_RESOURCE : 1
Alarm Type :DMI_RANDOM_USED_ONLY_ONCE : 1
Alarm Type :NP_LOAD_OF_KNOWN_NULL_VALUE : 1
Alarm Type :OBL_UNSATISFIED_OBLIGATION : 1
Alarm Type :WA_NOT_IN_LOOP : 1
Alarm Type :MS_SHOULD_BE_FINAL : 3
Alarm Type :RV_CHECK_FOR_POSITIVE_INDEXOF : 1
Alarm Type :DM_DEFAULT_ENCODING : 13
Alarm Type :ODR_OPEN_DATABASE_RESOURCE_EXCEPTION_PATH : 1
Alarm Type :DM_GC : 1
Alarm Type :SIC_INNER_SHOULD_BE_STATIC : 3
Alarm Type :ES_COMPARING_STRINGS_WITH_EQ : 3
Alarm Type :ICAST_INTEGER_MULTIPLY_CAST_TO_LONG : 1
Alarm Type :ES_COMPARING_PARAMETER_STRING_WITH_EQ : 3
Alarm Type :UWF_FIELD_NOT_INITIALIZED_IN_CONSTRUCTOR : 3
Alarm Type :WMI_WRONG_MAP_ITERATOR : 1
Alarm Type :BC_UNCONFIRMED_CAST : 1
Alarm Type :MS_PKGPROTECT : 2
Alarm Type :SIC_INNER_SHOULD_BE_STATIC_ANON : 3
Alarm Type :PZLA_PREFER_ZERO_LENGTH_ARRAYS : 2
Alarm Type :SC_START_IN_CTOR : 1
Alarm Type :RR_NOT_CHECKED : 4
Alarm Type :RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE : 2
Alarm Type :DM_CONVERT_CASE : 10
Alarm Type :DE_MIGHT_IGNORE : 1
Alarm Type :SW_SWING_METHODS_INVOKED_IN_SWING_THREAD : 1
Alarm Type :ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD : 14
*/
public static void main(String[] args) throws IOException {
String content = FileHelper.readFile("OUTPUT/DEL_INS.list");
Map<String, Integer> map = new HashMap<>();
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
StringBuilder singleAlarm = new StringBuilder();
List<String> alarms = new ArrayList<>();
while ((line = reader.readLine()) != null) {
if (line.startsWith("Alarm")) {
if (singleAlarm.length() > 0) {
alarms.add(singleAlarm.toString());
singleAlarm = new StringBuilder();
}
}
singleAlarm.append(line + "\n");
// String type = line.trim();
//
// if (map.containsKey(type)) {
// map.put(type, map.get(type) + 1);
// } else {
// map.put(type, 1);
// }
}
reader.close();
alarms.add(singleAlarm.toString());
ListSorter<String> sorter = new ListSorter<String>(alarms);
alarms = sorter.sortAscending();
StringBuilder alarmsBuilder = new StringBuilder();
for (String a : alarms) {
alarmsBuilder.append(a);
}
FileHelper.outputToFile("OUTPUT/DEL.list", alarmsBuilder, false);
for (Map.Entry<String, Integer> entry : map.entrySet()) {
System.out.println(entry.getKey() + " : " + entry.getValue());
}
System.out.println();
}
}
src/main/java/edu/lu/uni/serval/defects4j/Bug.java
+43
View File
@@ -0,0 +1,43 @@
package edu.lu.uni.serval.defects4j;
public class Bug {
private String project;
private String fileName;
private int startLine;
private int endLine;
private String type;
public Bug(String project, String fileName, int startLine, int endLine, String type) {
super();
this.project = project;
this.fileName = fileName;
this.startLine = startLine;
this.endLine = endLine;
this.type = type;
}
public String getProject() {
return project;
}
public String getFileName() {
return fileName;
}
public int getStartLine() {
return startLine;
}
public int getEndLine() {
return endLine;
}
public String getType() {
return type;
}
}
src/main/java/edu/lu/uni/serval/defects4j/BugParser.java
+206
View File
@@ -0,0 +1,206 @@
package edu.lu.uni.serval.defects4j;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;
import com.github.gumtreediff.tree.ITree;
import edu.lu.uni.serval.FixPatternParser.Tokenizer;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.gumtree.regroup.SimpleTree;
import edu.lu.uni.serval.gumtree.regroup.SimplifyTree;
import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.violation.code.parser.ViolationSourceCodeTree;
public class BugParser {
public static void main(String[] args) throws IOException {
List<Bug> bugs = new ArrayList<>();
String bugsFile = "Dataset/Defects4j/Bugs.txt";
String content = FileHelper.readFile(bugsFile);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
StringBuilder tokensBuilder = new StringBuilder();
StringBuilder buggyCodeBuilder = new StringBuilder();
while ((line = reader.readLine()) != null) {
String[] elements = line.split(" : ");
String type = elements[0];
String project = elements[1];
project = project.substring(0, project.lastIndexOf("."));
String fileName = elements[2];
int startLine = Integer.parseInt(elements[3]);
int endLine = Integer.parseInt(elements[4]);
Bug bug = new Bug(project, fileName, startLine, endLine, type);
bugs.add(bug);
}
int i = 0;
for (Bug bug : bugs) {
String type = bug.getType();
String project = bug.getProject();
String fileName = bug.getFileName();
int startLine = bug.getStartLine();
int endLine = bug.getEndLine();
File buggyFile = null;
String[] elements = project.split("_");
if (elements.length < 2) System.out.println(type + "-" + project + "-" + fileName + "-" + startLine + "-" + endLine);
List<File> javaFiles = FileHelper.getAllFiles(Configuration.ROOT_PATH + "Testing/projects1/" + elements[0] + "/" + elements[1] + "-b", ".java");
for (File javaFile : javaFiles) {
if (javaFile.getPath().endsWith(fileName)) {
buggyFile = javaFile;
break;
}
}
if (i == 31) {
i = i + 1 - 1;
}
SimpleTree simpleTree = null;
if ("EQ_DOESNT_OVERRIDE_EQUALS".equals(type)|| "HE_EQUALS_USE_HASHCODE".equals(type) || "HE_INHERITS_EQUALS_USE_HASHCODE".equals(type)||
"SE_NO_SUITABLE_CONSTRUCTOR".equals(type) || "RI_REDUNDANT_INTERFACES".equals(type)
// ||"CN_IDIOM_NO_SUPER_CALL".equals(type)
||"SE_NO_SERIALVERSIONID".equals(type) || "SE_COMPARATOR_SHOULD_BE_SERIALIZABLE".equals(type)) {
ViolationSourceCodeTree parser = new ViolationSourceCodeTree(buggyFile, startLine, endLine);
ITree classNameTree = parser.getClassNameTokens();
simpleTree = new SimplifyTree().canonicalizeSourceCodeTree(classNameTree, null);
startLine = parser.getViolationFinalStartLine();
endLine = parser.getViolationFinalEndLine();
} else {
ViolationSourceCodeTree alarmTree = new ViolationSourceCodeTree(buggyFile, startLine, endLine);
alarmTree.extract();
List<ITree> matchedTrees = alarmTree.getViolationSourceCodeTrees();
if (matchedTrees.size() == 0) {
System.err.println("#Null_Violation_Hunk: " + buggyFile.getName() + ":" + startLine + ":" + endLine);
continue;
}
simpleTree = new SimpleTree();
simpleTree.setLabel("Block");
simpleTree.setNodeType("Block");
List<SimpleTree> children = new ArrayList<>();
for (ITree matchedTree : matchedTrees) {
SimpleTree simpleT = new SimplifyTree().canonicalizeSourceCodeTree(matchedTree, null);
children.add(simpleT);
}
simpleTree.setChildren(children);
startLine = alarmTree.getViolationFinalStartLine();
endLine = alarmTree.getViolationFinalEndLine();
}
String tokens = Tokenizer.getTokensDeepFirst(simpleTree);
String[] tokensArray = tokens.split(" ");
int length = tokensArray.length;
System.out.println((++ i) + "==" + length);
// sizes += length + "\n";
// this.tokens += tokens + "\n";
String sourceCode = readSourceCode(buggyFile, startLine, endLine, type);
// this.sourceCode += sourceCode + "\n";
// tokensBuilder.append(type).append(":").append(tokens).append("\n");
tokensBuilder.append(tokens).append("\n");
buggyCodeBuilder.append(sourceCode).append("\n");
}
FileHelper.outputToFile("Dataset/Defects4j/buggyTokens.list", tokensBuilder, false);
FileHelper.outputToFile("Dataset/Defects4j/buggySourceCode.list", buggyCodeBuilder, false);
}
private static String readSourceCode(File javaFile, int startLine, int endLine, String violationType) {
StringBuilder sourceCode = new StringBuilder("##Source_Code:\n");
sourceCode.append(violationType).append("\n");
sourceCode.append(javaFile.getName().replaceAll("#", "/")).append("\nPosition: ").append(startLine).append(" : ").append(endLine).append("\n");
FileInputStream fis = null;
Scanner scanner = null;
try {
fis = new FileInputStream(javaFile);
scanner = new Scanner(fis);
int counter = 0;
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
counter ++;
if (startLine <= counter && counter <= endLine) {
sourceCode.append(line + "\n");
}
if (counter == endLine) break;
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
try {
scanner.close();
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sourceCode.toString();
}
}
/*
20
32
20
8
8
16
22
10
10
10
12
16
18
10
12
8
14
44
8
26
20
10
12
12
10
4
10
8
8
12
16
1776
8
10
10
12
8
50
18
8
12
38
4
4
14
16
20
26
8
14
12
14
*/
src/main/java/edu/lu/uni/serval/livestudy/Test.java
-19
View File
@@ -1,19 +0,0 @@
package edu.lu.uni.serval.livestudy;
import java.io.File;
import java.util.List;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
public class Test {
public static void main(String[] argus) {
List<File> files = FileHelper.getAllFiles(Configuration.ROOT_PATH + "LiveStudy/projects/camel/", ".java");
for (File file : files ) {
// if (file.getPath().endsWith("/ItemList.java")) {
// System.out.println(file.getPath());
// }
FileHelper.outputToFile("OUTPUT/camel.test", file.getPath() + "\n", true);
}
}
}
src/main/java/edu/lu/uni/serval/statistics/Categories.java
+1 -1
View File
@@ -14,7 +14,7 @@ import edu.lu.uni.serval.utils.FileHelper;
public class Categories {
public static void main(String[] args) throws IOException {
String content = FileHelper.readFile(Configuration.ROOT_PATH + "RQ1/Quantity-per-V-Type.csv");
String content = FileHelper.readFile(Configuration.ROOT_PATH + "RQ1/Quantity-per-Fixed_1.0V-Type.csv");//"RQ1/Quantity-per-V-Type.csv");
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = reader.readLine();
int counter = 0;
src/main/java/edu/lu/uni/serval/statistics/NumberOfFilesAndCodeLines.java
+7 -9
View File
@@ -20,15 +20,13 @@ import edu.lu.uni.serval.utils.FileHelper;
public class NumberOfFilesAndCodeLines {
public static void main(String[] args) {
// String projectsPath = Configuration.ROOT_PATH + "LiveStudy/BugsInfo/";
// File[] projects = new File(projectsPath).listFiles();
// for (File project : projects) {
// if (project.isDirectory()) {
// new NumberOfFilesAndCodeLines().statistic(project.getName());
// }
// }
for (int i = 1; i <= 50; i ++) {
System.out.print(" & " + i);
String projectsPath = Configuration.ROOT_PATH + "LiveStudy/BugsInfo/";
File[] projects = new File(projectsPath).listFiles();
for (File project : projects) {
if (project.isDirectory()) {
if (project.getName().equals("poi"))
new NumberOfFilesAndCodeLines().statistic(project.getName());
}
}
}
src/main/java/edu/lu/uni/serval/statistics/Statistic.java
+78 -7
View File
@@ -12,12 +12,35 @@ import java.util.List;
import java.util.Map;
import java.util.Scanner;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.utils.ListSorter;
import edu.lu.uni.serval.utils.MapSorter;
public class Statistic {
private static Map<String, Integer> map1 = new HashMap<String, Integer>();
private static Map<String, Integer> map2 = new HashMap<>();
public static void main(String[] args) throws IOException {
/*
* DM_DEFAULT_ENCODING
* NP_NONNULL_RETURN_VIOLATION
* NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE
* NP_PARAMETER_MUST_BE_NONNULL_BUT_MARKED_AS_NULLABLE
* ODR_OPEN_DATABASE_RESOURCE
* PZLA_PREFER_ZERO_LENGTH_ARRAYS
* RI_REDUNDANT_INTERFACES
* RV_RETURN_VALUE_IGNORED_BAD_PRACTICE
* RV_RETURN_VALUE_IGNORED_NO_SIDE_EFFECT
* SE_NO_SERIALVERSIONID
* SF_SWITCH_NO_DEFAULT
* SIC_INNER_SHOULD_BE_STATIC
* SQL_PREPARED_STATEMENT_GENERATED_FROM_NONCONSTANT_STRING
* UC_USELESS_CONDITION
* UC_USELESS_OBJECT
* UCF_USELESS_CONTROL_FLOW
* WMI_WRONG_MAP_ITERATOR
*/
// /*
// * Quantities' distribution of all violation types.
// */
@@ -48,12 +71,52 @@ public class Statistic {
/**
* Do statistics from two files:
*/
// statistics("../FPM_Violations/RQ1/all-leafnodes-per-project-vtype.csv", "");
statistics("../FPM_Violations/RQ1/all-leafnodes-per-project-vtype.csv", "", map1, 16918530, 730);
// statistics("../FPM_Violations/RQ1/distinct-fixed-summary-per-project-vtype.csv", "Fixed");
statistics("../FPM_Violations/RQ1/fixedViolations-v-1.0.csv", "Fixed_1.0");
statistics("../FPM_Violations/RQ1/fixedViolations-v-1.0.csv", "Fixed_1.0", map2, 88927, 548);
// fixedVSunfixed();
StringBuilder builder = new StringBuilder();
for (Map.Entry<String, Integer> entry : map1.entrySet()) {
String key = entry.getKey();
builder.append(key + "," + entry.getValue() + "," + (map2.containsKey(key) ? map2.get(key) : 0) + "\n");
}
FileHelper.outputToFile(Configuration.ROOT_PATH + "RQ2/quantity-ratios.csv", builder, false);
statisticsOfFixedViolations();
// statisticsOfFixedViolations();
//rsync -avP gaia-cluster:/work/users/kliu/FixPattern/FPM_Violations/UnfixedViolations/BC_UNCONFIRMED_CAST/Sizes.list Sizes/Sizes1.list
// String s = "rsync -avP gaia-cluster:/work/users/kliu/FixPattern/FPM_Violations/UnfixedViolations_RQ3/";
// File[] files = new File(Configuration.ROOT_PATH + "RQ3_1/UnfixedInstances/").listFiles();
// int i = 0;
// for (File file : files) {
// if (file.getName().endsWith(".list")) {
// i ++;
// System.out.println(s + FileHelper.getFileNameWithoutExtension(file) + "/Sizes.list Sizes/Sizes" + i + ".list");
// }
// }
// tokenSizes();
}
public static void tokenSizes() {
List<Integer> sizes = new ArrayList<>();
List<File> files = FileHelper.getAllFilesInCurrentDiectory(Configuration.ROOT_PATH + "RQ3_1/Sizes/", ".list");
StringBuilder builder = new StringBuilder();
for (File file : files) {
try {
String content = FileHelper.readFile(file);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
while ((line = reader.readLine()) != null) {
sizes.add(Integer.parseInt(line));
builder.append("1,").append(line).append("\n");
}
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
System.out.println(sizes.size());
FileHelper.outputToFile(Configuration.ROOT_PATH + "RQ3_1/Sizes.csv", builder, false);
}
public static void statisticsOfFixedViolations() {
@@ -237,7 +300,7 @@ B: 4682
*/
}
public static void statistics(String fileName, String type) throws IOException {
public static void statistics(String fileName, String type, Map<String, Integer> map, int totalQ, int totalS) throws IOException {
FileInputStream fis = new FileInputStream(fileName);
Scanner scanner = new Scanner(fis);
@@ -332,7 +395,7 @@ B: 4682
Map<String, Integer> quantityOfCategory = new HashMap<>();
Map<String, List<String>> violationTypesOfCategory = new HashMap<>();
Map<String, List<String>> projectsOfCategory = new HashMap<>();
StringBuilder violationsBuilder = new StringBuilder("Type,Identifier,Quantity,Widespread,Category\n");
StringBuilder violationsBuilder = new StringBuilder("Type,Identifier,Quantity,Ratio,Widespread,Ratio2, Categories\n");
int identifier = 0;
for (Map.Entry<String, Integer> entry : violationTypesMap.entrySet()) {
String violationType = entry.getKey();
@@ -342,8 +405,9 @@ B: 4682
if (category == null || category.equals("null")) {
category = "Other";
}
violationsBuilder.append(violationType + "," + identifier + "," + quantity + "," + widespreadViolationsMap.get(violationType).size()
+ "," + category + "\n");
int spread = widespreadViolationsMap.get(violationType).size();
violationsBuilder.append(violationType + "," + identifier + "," + quantity + "," + ((double) quantity / totalQ * 100) + ","
+ spread + "," + ((double) spread / totalS * 100) + "," + category + "\n");
if (quantityOfCategory.containsKey(category)) {
quantityOfCategory.put(category, quantityOfCategory.get(category) + quantity);
@@ -375,6 +439,8 @@ B: 4682
}
sortedViolationTypes.add(violationType);
map.put(violationType, quantity);
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Quantity-per-" + type + "V-Type.csv", violationsBuilder, false);
@@ -411,21 +477,26 @@ B: 4682
// builder.append("," + i);
// }
// builder.append("\n");
StringBuilder bui = new StringBuilder();
for (int i = 0; i < projectNames.size(); i++) {
String projectName = projectNames.get(i);
builder.append(projectName);
for (int j = 0; j < sortedViolationTypes.size(); j ++) {
String violationType = sortedViolationTypes.get(j);
String key = projectName + "," + violationType;
Integer value = perVperProjMap.get(key);
if (value == null) {
value = 0;
} else {
bui.append(projectName).append(",").append(violationType).append(",").append(value).append(",").append(categories.get(violationType) == null ? "Other" : categories.get(violationType)).append("\n");
}
builder.append("," + value);
}
builder.append("\n");
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Distribution-per-project-per-" + type + "type.csv", builder, false);
FileHelper.outputToFile("../FPM_Violations/RQ1/Distribution-per-project-per-" + type + "type___.csv", bui, false);
StringBuilder ssbuilder = new StringBuilder("Type, Identifier, Quantity, Category\n");
Map<String, Integer> perTypePerProj = new HashMap<>();
src/main/java/edu/lu/uni/serval/statistics/UnhappenedTypesLiveStudy.java
+73
View File
@@ -0,0 +1,73 @@
package edu.lu.uni.serval.statistics;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.utils.MapSorter;
public class UnhappenedTypesLiveStudy {
public static void main(String[] args) throws IOException {
List<String> top50Types = readTypes();
String path = Configuration.ROOT_PATH + "RQ3_3/LiveStudy/BugsInfo/";
File[] projects = new File(path).listFiles();
Map<String, Integer> map = new HashMap<>();
int i = 0;
for (File project : projects) {
if (project.isDirectory()) {
File[] types = project.listFiles();
List<String> top50TypesCopy = new ArrayList<>();
top50TypesCopy.addAll(top50Types);
for (File type :types) {
if (type.isDirectory()) {
String typeStr = type.getName();
top50TypesCopy.remove(typeStr);
}
}
System.out.println(project.getName() + "::");
for (String type : top50TypesCopy) {
System.out.println(type);
if (map.containsKey(type)) {
map.put(type, map.get(type) + 1);
} else {
map.put(type, 1);
}
}
i ++;
if (i == 10) break;
}
}
MapSorter<String, Integer> sorter = new MapSorter<>();
map = sorter.sortByValueDescending(map);
for (Map.Entry<String, Integer> entry : map.entrySet()) {
System.out.println(entry.getKey() + "==" + entry.getValue());
}
}
private static List<String> readTypes() throws IOException {
List<String> types = new ArrayList<>();
String content = FileHelper.readFile(Configuration.ROOT_PATH + "RQ1/Quantity-per-Fixed_1.0V-Type.csv");
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
int i = 0;
while ((line = reader.readLine()) != null) {
String type = line.substring(0, line.indexOf(","));
types.add(type);
i ++;
if (i == 51) break;
}
return types;
}
}
src/main/java/edu/lu/uni/serval/violation/code/parser/Test.java
-10
View File
@@ -18,18 +18,8 @@ import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.utils.ListSorter;
public class Test {
/**
* Commons-lang: 54. PZLA_PREFER_ZERO_LENGTH_ARRAYS
* commons-lang: 1, 4, 5 ES_COMPARING_PARAMETER_STRING_WITH_EQ
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException {
boolean b1 = false;
boolean b2 = false;
System.out.println(!b1 || b2 && b1);
System.out.println(b2 && b1);
// testV1();
// testV2();
// testV3("../FPM_Violations/OAR.FPM.4225346.stderr", "OUTPUT/unparsedviolations.txt");
src/main/java/edu/lu/uni/serval/violation/code/parser/ViolationCodeParser.java
+31 -20
View File
@@ -133,32 +133,43 @@ public class ViolationCodeParser {
//// continue;
//// }
// }
ViolationSourceCodeTree alarmTree = new ViolationSourceCodeTree(javaFile, startLine, endLine);
alarmTree.extract();
List<ITree> matchedTrees = alarmTree.getViolationSourceCodeTrees();
if (matchedTrees.size() == 0) {
System.err.println("#Null_Violation_Hunk: " + javaFile.getName() + ":" + startLine + ":" + endLine);
continue;
SimpleTree simpleTree = null;
if ("EQ_DOESNT_OVERRIDE_EQUALS".equals(type)|| "HE_EQUALS_USE_HASHCODE".equals(type) || "HE_INHERITS_EQUALS_USE_HASHCODE".equals(type)||
"SE_NO_SUITABLE_CONSTRUCTOR".equals(type) || "RI_REDUNDANT_INTERFACES".equals(type)
||"CN_IDIOM".equals(type)||"SE_NO_SERIALVERSIONID".equals(type) || "SE_COMPARATOR_SHOULD_BE_SERIALIZABLE".equals(type)) {
ViolationSourceCodeTree parser = new ViolationSourceCodeTree(javaFile, violation.getStartLineNum(), violation.getEndLineNum());
ITree classNameTree = parser.getClassNameTokens();
simpleTree = new SimplifyTree().canonicalizeSourceCodeTree(classNameTree, null);
startLine = parser.getViolationFinalStartLine();
endLine = parser.getViolationFinalEndLine();
} else {
ViolationSourceCodeTree alarmTree = new ViolationSourceCodeTree(javaFile, startLine, endLine);
alarmTree.extract();
List<ITree> matchedTrees = alarmTree.getViolationSourceCodeTrees();
if (matchedTrees.size() == 0) {
System.err.println("#Null_Violation_Hunk: " + javaFile.getName() + ":" + startLine + ":" + endLine);
continue;
}
simpleTree = new SimpleTree();
simpleTree.setLabel("Block");
simpleTree.setNodeType("Block");
List<SimpleTree> children = new ArrayList<>();
for (ITree matchedTree : matchedTrees) {
SimpleTree simpleT = new SimplifyTree().canonicalizeSourceCodeTree(matchedTree, null);
children.add(simpleT);
}
simpleTree.setChildren(children);
startLine = alarmTree.getViolationFinalStartLine();
endLine = alarmTree.getViolationFinalEndLine();
}
SimpleTree simpleTree = new SimpleTree();
simpleTree.setLabel("Block");
simpleTree.setNodeType("Block");
List<SimpleTree> children = new ArrayList<>();
for (ITree matchedTree : matchedTrees) {
SimpleTree simpleT = new SimplifyTree().canonicalizeSourceCodeTree(matchedTree, null);
children.add(simpleT);
}
simpleTree.setChildren(children);
String tokens = Tokenizer.getTokensDeepFirst(simpleTree);
String[] tokensArray = tokens.split(" ");
int length = tokensArray.length;
sizes += length + "\n";
this.tokens += tokens + "\n";
startLine = alarmTree.getViolationFinalStartLine();
endLine = alarmTree.getViolationFinalEndLine();
String sourceCode = readSourceCode(javaFile, startLine, endLine, violation.getViolationType());
this.sourceCode += sourceCode + "\n";
}
src/main/java/edu/lu/uni/serval/violation/code/parser/ViolationSourceCodeTree.java
+16 -1
View File
@@ -98,7 +98,13 @@ public class ViolationSourceCodeTree {
int endPosition = startPosition + tree.getLength();
int endLine = cUnit.getLineNumber(endPosition - 1);
if (endLine < violationStartLine) continue;
/*
violationT ||
violationType.equals("UC_USELESS_CONDITION")
*/
if (type.equals("SE_NO_SERIALVERSIONID")) {
}
// FIXME the violation occurred in the Class Name
matchTrees(tree.getChildren());
}
@@ -387,6 +393,9 @@ public class ViolationSourceCodeTree {
&& type != 76 && type != 84 && type != 87 && type != 88) {
// ArrayType, PrimitiveType, SimpleType, ParameterizedType,
// QualifiedType, WildcardType, UnionType, IntersectionType, NameQualifiedType
if (type == 42 && child.getLabel().startsWith("ClassName:")) {
continue;
}
if (i > 0) {
child = children.get(i - 1);
return child.getPos() + child.getLength() + 1;
@@ -684,10 +693,16 @@ public class ViolationSourceCodeTree {
&& type != 76 && type != 84 && type != 87 && type != 88) {
// ArrayType, PrimitiveType, SimpleType, ParameterizedType,
// QualifiedType, WildcardType, UnionType, IntersectionType, NameQualifiedType
if (type == 42 && child.getLabel().startsWith("ClassName:")) {
newChildren.add(child);
continue;
}
if (i > 0) {
child = children.get(i - 1);
tree.setChildren(newChildren);
return child.getPos() + child.getLength() + 1;
} else {
tree.setChildren(newChildren);
return child.getPos() - 1;
}
} else {
src/main/java/edu/lu/uni/serval/violation/parse/TestViolationParser.java
+5 -1
View File
@@ -28,6 +28,9 @@ public class TestViolationParser {
}
List<String> violationTypes = new ArrayList<>();
/*
* DM_DEFAULT_ENCODING
*/
violationTypes.add("NP_NONNULL_RETURN_VIOLATION");
violationTypes.add("NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE");
violationTypes.add("NP_PARAMETER_MUST_BE_NONNULL_BUT_MARKED_AS_NULLABLE");
@@ -52,7 +55,8 @@ public class TestViolationParser {
List<File> unfixedAlarmFiles = FileHelper.getAllFilesInCurrentDiectory(unfixedViolations, ".list");
for (File file : unfixedAlarmFiles) {
String fileName = FileHelper.getFileNameWithoutExtension(file);
if (!violationTypes.contains(fileName)) continue;
// if (!violationTypes.contains(fileName)) continue;
if (!fileName.equals("DM_DEFAULT_ENCODING")) continue;
FileHelper.createDirectory(unfixedFilesPath + fileName + "/");
FileHelper.createDirectory(unfixedPositionsFilePath + fileName + "/");
ViolationParser parser = new ViolationParser();