azalea/fixminer_source
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #1 from fixminer/TreeDump

Browse Source 
Tree dump
This commit is contained in:
fixminer
2018-09-17 18:35:14 +02:00
committed by GitHub
parent 8907fabbc7 2d12536b93
commit 15aceec7ab
116 changed files with 4599 additions and 116722 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Dataset/LiveStudy/Patches
-119
View File
@@ -1,119 +0,0 @@
BC_UNCONFIRMED_CAST
BC_UNCONFIRMED_CAST_OF_RETURN_VALUE Comments: no any useful common fix patterns.
EI_EXPOSE_REP
EI_EXPOSE_REP2
IS2_INCONSISTENT_SYNC
NM_METHOD_NAMING_CONVENTION
DE_MIGHT_IGNORE Compile failed.
DLS_DEAD_LOCAL_STORE Compile failed
NP_LOAD_OF_KNOWN_NULL_VALUE
DM_CONVERT_CASE To be improved
DM_BOXED_PRIMITIVE_FOR_PARSING
DM_DEFAULT_ENCODING
DM_FP_NUMBER_CTOR
DM_NUMBER_CTOR
ES_COMPARING_STRINGS_WITH_EQ Comments: ignored
MS_PKGPROTECT
MS_SHOULD_BE_FINAL Comments: failed to checkstyle.
NP_NONNULL_RETURN_VIOLATION
NP_NULL_ON_SOME_PATH
NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE
NP_PARAMETER_MUST_BE_NONNULL_BUT_MARKED_AS_NULLABLE
OBL_UNSATISFIED_OBLIGATION
OBL_UNSATISFIED_OBLIGATION_EXCEPTION_EDGE
ODR_OPEN_DATABASE_RESOURCE
OS_OPEN_STREAM
OS_OPEN_STREAM_EXCEPTION_PATH
PZLA_PREFER_ZERO_LENGTH_ARRAYS
RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE
RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE
REC_CATCH_EXCEPTION
RI_REDUNDANT_INTERFACES
RV_RETURN_VALUE_IGNORED_BAD_PRACTICE
RV_RETURN_VALUE_IGNORED_NO_SIDE_EFFECT
SBSC_USE_STRINGBUFFER_CONCATENATION
SE_NO_SERIALVERSIONID
SF_SWITCH_NO_DEFAULT
SIC_INNER_SHOULD_BE_STATIC
SIC_INNER_SHOULD_BE_STATIC_ANON
SQL_PREPARED_STATEMENT_GENERATED_FROM_NONCONSTANT_STRING
ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD
UC_USELESS_CONDITION
UC_USELESS_OBJECT
UCF_USELESS_CONTROL_FLOW
UPM_UNCALLED_PRIVATE_METHOD
URF_UNREAD_FIELD
URF_UNREAD_PUBLIC_OR_PROTECTED_FIELD
UWF_FIELD_NOT_INITIALIZED_IN_CONSTRUCTOR
UWF_UNWRITTEN_FIELD
VA_FORMAT_STRING_USES_NEWLINE
WMI_WRONG_MAP_ITERATOR
rejected 1 https://github.com/apache/commons-io/pull/47
rejected 1 https://github.com/apache/commons-io/pull/45
rejected 1 https://github.com/apache/commons-lang/pull/298
5 https://github.com/apache/commons-lang/pull/297
improved 1 https://github.com/apache/commons-lang/pull/297 Comments: some patches failed to pass test cases, some patches led to compile failed, some patches led to checkstyle failed.
2 https://github.com/apache/commons-math/pull/69
1 https://github.com/apache/commons-math/pull/68 Comments: inner static class, there is no need to fix the violations in this kind of class.
2 https://github.com/apache/commons-math/pull/67 Comments: improve patches.
1 https://github.com/apache/commons-math/pull/66
rejected 1 https://github.com/apache/ant/pull/47
improved 1 https://github.com/apache/ant/pull/46
merged 2 https://github.com/apache/ant/pull/46
2 https://github.com/apache/ant/pull/45
rejected 3 https://github.com/apache/ant/pull/44
merged 1 https://github.com/apache/ant/pull/43
merged 5 https://github.com/apache/ant/pull/42 Comments: need to import new packages, unless the fixed patches will fail to compile.
merged 1 https://github.com/apache/ant/pull/41
1 https://github.com/apache/aries/pull/80
1 https://github.com/apache/aries/pull/79
1 https://github.com/apache/aries/pull/78
1 https://github.com/apache/aries/pull/77
1 https://github.com/apache/aries/pull/76
rejected 1 https://github.com/apache/camel/pull/2031 ShouldBeFinal
rejected 6 https://github.com/apache/camel/pull/2029 master5
merged 1 https://github.com/apache/camel/pull/2035 master7
merged 2 https://github.com/apache/camel/pull/2032 Useless
merged 2 https://github.com/apache/camel/pull/2025 master6
merged 1 https://github.com/apache/camel/pull/2024 master4
merged 1 https://github.com/apache/camel/pull/2023 master2
rejected 1 https://github.com/apache/camel/pull/2017 master3 Comments: compile failed, the return value of method invocation is not Boolean, it is a specific case.
merged 1 https://github.com/apache/camel/pull/2016 master
merged 6 https://github.com/apache/camel/pull/2038 boxedParsing
3 https://github.com/apache/cassandra/pull/165
4 https://github.com/apache/cassandra/pull/164
2 https://github.com/apache/cassandra/pull/163
Fixed 1 https://github.com/fangyidong/json-simple/pull/126 Comments: fixed in another version. https://github.com/RalleYTN/SimpleJSON
1 https://github.com/fangyidong/json-simple/pull/125
On hold 1 https://github.com/apache/mahout/pull/346
1 https://github.com/apache/mahout/pull/345
1 https://github.com/apache/mahout/pull/344
merged 2 https://github.com/apache/poi/pull/77 5
merged 4 https://github.com/apache/poi/pull/78 6
merged 2 https://github.com/apache/poi/pull/79 t
merged 1 https://github.com/apache/poi/pull/80 4
merged 4 https://github.com/apache/poi/pull/76 2
merged 30 https://github.com/apache/poi/pull/75 master
merged 1 https://github.com/apache/poi/pull/74 3
Some violations never happened in these projects:
Dataset/Patterns/Fix-Patterns
-131
View File
@@ -1,131 +0,0 @@
=====================================
=========Useful Fix Patterns=========
=====================================
==============Detailed===============
DM_CONVERT_CASE
(1) ADD a rule of Locale.ENGLISH into toLowerCase/toUpperCase.
RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE
(1) DEL Null-Check IfStatement,
(2) DEL Null-Check Conditional Expression.
REC_CATCH_EXCEPTION
(1) Change Exception to a specific exception class.
SE_NO_SERIALVERSIONID
(1) Insert a private static final field serialVersionUID.
VA_FORMAT_STRING_USES_NEWLINE
(1) DEL "$backslash$n",
(2) Replace "$backslash$n" with "n"
MS_SHOULD_BE_FINAL
(1) ADD a ``final'' modifier.
WMI_WRONG_MAP_ITERATOR
(1) Replace the KeySet iterating ForStatement with an EntrySet iterating ForStatement.
DM_NUMBER_CTOR
(1) Replace the NumberType ClassInstanceCreation with a NumberType.valueOf() method invocation.
SIC_INNER_SHOULD_BE_STATIC
(1) ADD a "static" modifier,
(2) DEL the buggy statements.
DE_MIGHT_IGNORE
(1) Change Exception to a specific exception class.
DM_BOXED_PRIMITIVE_FOR_PARSING
(1) Replace NumberType.valueOf() with NumberType.parseNumberType().
DM_FP_NUMBER_CTOR
(1) DEL the buggy statement,
(2) Replace Double or Float ClassInstanceCreation with (Double or Float).valueOf(),
(3) Remove the Double or Float ClassInstanceCreation from the buggy expression.
MS_PKGPROTECT
(1) DEL the buggy statement,
(2) DEL the ``public'' or ``protected'' modifier
(3) Replace the ``public'' or ``protected'' modifier with the ``private'' modifier.
(4) DEL the initialization expression.
ES_COMPARING_STRINGS_WITH_EQ
(1) Replace the ``=='' InfixExpression with a stringVar.equals() method invocation.
===============Abstract==============
NP_NULL_ON_SOME_PATH
(1) ADD a Null-check statement or expression.
RI_REDUNDANT_INTERFACES
(1) DEL the implemented Interface type.
NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE
(1) ADD a Null-check statement or expression,
(2) DEL the buggy statement.
UCF_USELESS_CONTROL_FLOW
(1) DEL the buggy statement,
(2) DEL the useless ConditionalExpression.
UC_USELESS_CONDITION
(1) UPD the operator of a ConditionalExpersion,
(2) UPD an sub-expression of a ConditionExpression.
NP_LOAD_OF_KNOWN_NULL_VALUE
(1) DEL Null-check expression or statement,
(2) Replace the null-known variable with NULL.
SBSC_USE_STRINGBUFFER_CONCATENATION
(1) Replace String type with StringBuilder, and replace plus operator of StringVarialbe with the append method of StringBuilder.
UC_USELESS_OBJECT
(1) DEL the buggy statement.
UPM_UNCALLED_PRIVATE_METHOD
(1) DEL the buggy statements.
NM_METHOD_NAMING_CONVENTION
(1) Update Method Name of MethodDeclaration.
URF_UNREAD_FIELD
(1) DEL the buggy statement.
=====================================
NP_NONNULL_RETURN_VIOLATION
(1) DEL the null-returned ReturnStatement.
(2) Replace NullLiteral with an Expression which returns an empty list.
PZLA_PREFER_ZERO_LENGTH_ARRAYS
(1) DEL the buggy statement,
√ (2) Replace NullLiteral with an empty array.
SIC_INNER_SHOULD_BE_STATIC_ANON
(1) DEL a parameter of a ClassInstanceCreation in an AnonymousClassDeclaration,
(2) DEL the statement contains an AnonymousClassDeclaration,
√ (3) Replace the AnonymousClassDeclaration with an defined Static Object of this class.
DLS_DEAD_LOCAL_STORE
(1) DEL the buggy statement.
DM_DEFAULT_ENCODING
√ (1) ADD a parameter (i.e., "UTF-8") into the ClassInstanceCreation of a Stream object.
(2) Replace ClassInstanceCreation with a MethodInvocation.
(3) DEL the buggy statement.
BC_UNCONFIRMED_CAST
(1) DEL the buggy statement,
(2) DEL the Cast Type,
(3) Replace CastExpression with NullLiteral.
URF_UNREAD_PUBLIC_OR_PROTECTED_FIELD
(1) ADD a ``final'' modifier,
√ (2) DEL the buggy statement.
RV_RETURN_VALUE_IGNORED_BAD_PRACTICE
(1) Replace boolean-returned statement with a a new MethodInvocation.
√ (2) ADD an IfStatement.
EI_EXPOSE_REP
(1) DEL the buggy statement,
* (2) Replace the being assigned Expression with a ConditionalExpression.
√ (3) Replace the Expression being assigned with other Expression
EI_EXPOSE_REP2
(1) DEL the buggy statement,
* (2) Replace the being assigned Expression with a ConditionalExpression.
√ (3) Replace the being assigned Expression with other Expression.
=====================================
==========Null Fix Patterns==========
=====================================
UWF_FIELD_NOT_INITIALIZED_IN_CONSTRUCTOR null.
OBL_UNSATISFIED_OBLIGATION null.
OS_OPEN_STREAM_EXCEPTION_PATH null.
SF_SWITCH_NO_DEFAULT null.
UWF_UNWRITTEN_FIELD null.
IS2_INCONSISTENT_SYNC null.
ODR_OPEN_DATABASE_RESOURCE null.
SQL_PREPARED_STATEMENT_GENERATED_FROM_NONCONSTANT_STRING null.
OBL_UNSATISFIED_OBLIGATION_EXCEPTION_EDGE null.
OS_OPEN_STREAM null.
NP_PARAMETER_MUST_BE_NONNULL_BUT_MARKED_AS_NULLABLE null.
=====================================
========Useless Fix Patterns=========
=====================================
BC_UNCONFIRMED_CAST_OF_RETURN_VALUE
(1) UPD T1 v1 = (T1) method_invocation() with T2 v1 = method_invocation(),
(2) DEL the buggy statement,
(3) Replace the buggy CastExpresion with a MethodInvocation.
ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD
(1) DEL the buggy statement.
RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE
(1) ADD a Null-check statement or expression.
Dataset/Patterns/Source-Code-Patterns-Unfixed-Violations
-107
View File
@@ -1,107 +0,0 @@
================================
=========Single Pattern=========
================================
SIC_INNER_SHOULD_BE_STATIC_ANON: new Type() { override some methods(...){...} }. Anonymous inner class declaration. 100%
DE_MIGHT_IGNORE: try{...} catch (Exception e) {...}, Exception is not any specified.
VA_FORMAT_STRING_USES_NEWLINE: "\n", this new line symbol is in a StringLiteral of a format method.
UCF_USELESS_CONTROL_FLOW: if (conditional expression) {empty statements.}.
UC_USELESS_CONDITION: if (conditional expressions) {...}.
UC_USELESS_OBJECT: T1 v1 = new T1().
PZLA_PREFER_ZERO_LENGTH_ARRAYS: return null.
EI_EXPOSE_REP: return variable;
REC_CATCH_EXCEPTION: try{...} catch (Exception e) {...}, Exception is not any specific exception.
SF_SWITCH_NO_DEFAULT: switch(exp) {case1:...case2:...}, without default statement.
UWF_UNWRITTEN_FIELD: return exp / variable.
SBSC_USE_STRINGBUFFER_CONCATENATION: stringVariable += stringExp,
RV_RETURN_VALUE_IGNORED_NO_SIDE_EFFECT: return method_invocation_exp. Method invocation statement returns a value, but does not assign the value to any variable. (High false positives.)
NP_NONNULL_RETURN_VIOLATION: return null. (High false positives.)
================================
========Multiple Patterns=======
================================
NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE: (1) for (File file : filesArray) {...} (2) for (int i =0; i < filesArray.length; i ++) {...}
DM_CONVERT_CASE: (1) stringExp.toLowerCase(), 24,897
(2)stringExp.toUpperCase(). 10,406
MS_PKGPROTECT: (1) public static final T1 v1 = initializationExp, 2442
(2) protected static final T1 v1 = initializationExp. 175
(3) protected static T1 v1 = initializationExp. 158
(4) public static T1 v1 = initializationExp. 262, other 46
SIC_INNER_SHOULD_BE_STATIC: (1) this.v1 = v2; 590
(2) v1 = v2; 170
(3) super(...). 213
BC_UNCONFIRMED_CAST: (1) T1 v1 = (T(1) v2/exp, (2) v1 = (T(1) v2/exp, (3) ((T(1) v2).exp.
ES_COMPARING_STRINGS_WITH_EQ: (1) stringExp1 == stringExp2 IfStatement, ReturnStatement, WhileStatement.
(2) stringExp1 != stringExp2
RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE: (1) if (exp == null ...) {...},
(2) if (exp != null ...) {...},
(3) exp == null ? exp1 : exp2,
(4) exp != null ? exp1 : exp2.
DM_NUMBER_CTOR: (1) new Long(...), (2)new Integer(...), (3)new Short(...), (4)new Byte(...), (5)new Char(...).
DM_FP_NUMBER_CTOR: (1) new Double(exp), (2)new Float(exp).
DM_BOXED_PRIMITIVE_FOR_PARSING: (1) Integer.valueOf(str), (2)Long.valueOf(str). Comments: it contains fewer patterns than findbugs.
EI_EXPOSE_REP2: (1) this.v1 = v2,
(2) v1 = v2. 3700
BC_UNCONFIRMED_CAST_OF_RETURN_VALUE: (1) T1 v1 = (T) method_invocation(),
(2) v1 = (T) method_invocation(),
(3) ((T) method_invocation()).exp.
ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD: (1) static_v1 = instance_exp1.method_invocation(),
(2) v1 = instance_exp1.method_invocation(),
(3) v1 = exp1, other variable assignment statement.
RV_RETURN_VALUE_IGNORED_BAD_PRACTICE: (1) File.mkdirs(),
(2) File.mkdir(),
(3) File.delete(),
(4) File.createNewFile(),
(5) other exp.method_invoation() returns a value.
SQL_PREPARED_STATEMENT_GENERATED_FROM_NONCONSTANT_STRING: (1) sqlExecutionStatement = StringLiteral1 + stringV1 + ... + StringLiteral2 + stringV2.
(2) sqlRelatedVariable = sqlRelatedMethodInvocation();
OS_OPEN_STREAM: (1) StreamType v = new StreamType(...), 2) streamVariable = new StreamType(new StreamType2(...)), e.g., 1. FileOutputStream fos = new FileOutputStream(outFile);
DM_DEFAULT_ENCODING: (1) new BufferedReader(new InputStreamReader(input)), 99+
(2) new StreamWriter(output);
(3) stringExp.getBytes(),
(4) new String(byteExp),
(5) byteExp.toString().
ODR_OPEN_DATABASE_RESOURCE: (1) conn.createStatement(),
(2) conn.createStatement.execute(...),
(3) PreparedStatement st = conn.prepareStatement(),
(4) DataBaseConnection conn = exp.getConnection(...).
NP_PARAMETER_MUST_BE_NONNULL_BUT_MARKED_AS_NULLABLE: (1) return method_invocation(...), 2) return class_instance_creation(...).
========================================
==========Inefficient Patterns==========
========================================
OBL_UNSATISFIED_OBLIGATION: (1) StreamType v = new StreamType(...),
(2) streamVariable = new StreamType(new StreamType2(...)),
(3) sqlStatementExp.executeMethod(). e.g., 1. FileOutputStream fos = new FileOutputStream(outFile); 2. ResultSet rs = sqlStmt.executeQuery().
OS_OPEN_STREAM_EXCEPTION_PATH: (1) StreamType v = new StreamType(...),
(2) streamVariable = new StreamType(...).
IS2_INCONSISTENT_SYNC: (1) return v1,
(2) if(conditionalExp) {...},
(3) sqlStmt = stringLiteral1 + stringExp + stringLiteral2. not very consistent.
OBL_UNSATISFIED_OBLIGATION_EXCEPTION_EDGE: (1) StreamType v = new StreamType(...),
(2) streamVariable = new StreamType(new StreamType2(...)),
(3) sqlStatementExp.executeMethod(). e.g., 1. FileOutputStream fos = new FileOutputStream(outFile); 2. ResultSet rs = sqlStmt.executeQuery().
WMI_WRONG_MAP_ITERATOR: for statement { ... MapVariable.get(keyExp) ... }. or NULL. Lack of enough information.
========================================
=====Redundant Inconsistent Pattern=====
========================================
MS_SHOULD_BE_FINAL: (1) public (28,302) static T1 v1 = exp. e.g., public static int maxStringLength = 65535.
(2) protected (6,719) static T1 v1 = exp.
URF_UNREAD_FIELD: (1) this.v1 = v2, (2)this.v1 = new T(), (3)private Type v = exp. Comments: it contains more patterns than findbugs. only pattern 3 is consistent.
URF_UNREAD_PUBLIC_OR_PROTECTED_FIELD: (1) this.v1 = v2,
(2) public/protected (static) T1 v1 = exp; Comments: only 2 is consistent.
(3) v1 = exp1.
========================================
=========Inconsistent Patterns=========
========================================
RI_REDUNDANT_INTERFACES: super(...). Comments: not very consistent. Locate on the position of super classes.
========================================
==============Null Pattern==============
========================================
===========Inaccurate Position==========
NM_METHOD_NAMING_CONVENTION: null. Comments: violate on title, but position locates on body.
SE_NO_SERIALVERSIONID: null.
UPM_UNCALLED_PRIVATE_METHOD: null. (method body)
UWF_FIELD_NOT_INITIALIZED_IN_CONSTRUCTOR: null. inaccurate positions.
==========Specific Source Code==========
DLS_DEAD_LOCAL_STORE: null. Variable assignment statements. Comments: specific source code.
NP_NULL_ON_SOME_PATH: null.
NP_LOAD_OF_KNOWN_NULL_VALUE: null.
RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE: null.
FixPatternMiner.iml
+54
View File
@@ -0,0 +1,54 @@
<?xml version="1.0" encoding="UTF-8"?>
<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
<component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8" inherit-compiler-output="false">
<output url="file://$MODULE_DIR$/target/classes" />
<output-test url="file://$MODULE_DIR$/target/test-classes" />
<content url="file://$MODULE_DIR$">
<sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
<sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
<excludeFolder url="file://$MODULE_DIR$/target" />
</content>
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
<orderEntry type="library" name="Maven: org.javatuples:javatuples:1.2" level="project" />
<orderEntry type="library" name="Maven: com.typesafe.akka:akka-actor_2.11:2.4.11" level="project" />
<orderEntry type="library" name="Maven: org.scala-lang:scala-library:2.11.8" level="project" />
<orderEntry type="library" name="Maven: com.typesafe:config:1.3.0" level="project" />
<orderEntry type="library" name="Maven: org.scala-lang.modules:scala-java8-compat_2.11:0.7.0" level="project" />
<orderEntry type="module" module-name="core" />
<orderEntry type="library" name="Maven: com.github.mpkorstanje:simmetrics-core:3.0.3" level="project" />
<orderEntry type="library" name="Maven: com.google.guava:guava:18.0" level="project" />
<orderEntry type="library" name="Maven: commons-codec:commons-codec:1.10" level="project" />
<orderEntry type="library" name="Maven: net.sf.trove4j:trove4j:3.0.3" level="project" />
<orderEntry type="library" name="Maven: com.google.code.gson:gson:2.3" level="project" />
<orderEntry type="module" module-name="gen.jdt" />
<orderEntry type="library" name="Maven: edu.lu.uni:simple-utils:0.0.1-SNAPSHOT" level="project" />
<orderEntry type="library" name="Maven: org.apache.poi:poi:3.12" level="project" />
<orderEntry type="library" name="Maven: org.apache.poi:poi-ooxml:3.12" level="project" />
<orderEntry type="library" name="Maven: org.apache.poi:poi-ooxml-schemas:3.12" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlbeans:xmlbeans:2.6.0" level="project" />
<orderEntry type="library" name="Maven: stax:stax-api:1.0.1" level="project" />
<orderEntry type="library" name="Maven: net.sourceforge.jexcelapi:jxl:2.6.12" level="project" />
<orderEntry type="library" name="Maven: log4j:log4j:1.2.14" level="project" />
<orderEntry type="library" name="Maven: org.eclipse.core:runtime:3.10.0-v20140318-2214" level="project" />
<orderEntry type="library" name="Maven: org.eclipse:osgi:3.10.0-v20140606-1445" level="project" />
<orderEntry type="library" name="Maven: org.eclipse.equinox:common:3.6.200-v20130402-1505" level="project" />
<orderEntry type="library" name="Maven: org.eclipse.core:jobs:3.6.0-v20140424-0053" level="project" />
<orderEntry type="library" name="Maven: org.eclipse.equinox:registry:3.5.400-v20140428-1507" level="project" />
<orderEntry type="library" name="Maven: org.eclipse.equinox:preferences:3.5.200-v20140224-1527" level="project" />
<orderEntry type="library" name="Maven: org.eclipse.core:contenttype:3.4.200-v20140207-1251" level="project" />
<orderEntry type="library" name="Maven: org.eclipse.equinox:app:1.3.200-v20130910-1609" level="project" />
<orderEntry type="library" name="Maven: org.eclipse.birt.runtime:org.eclipse.core.resources:3.10.0.v20150423-0755" level="project" />
<orderEntry type="library" name="Maven: org.eclipse.tycho:org.eclipse.jdt.core:3.12.2.v20161117-1814" level="project" />
<orderEntry type="library" name="Maven: org.slf4j:slf4j-api:1.7.7" level="project" />
<orderEntry type="library" name="Maven: ch.qos.logback:logback-classic:1.1.2" level="project" />
<orderEntry type="library" name="Maven: ch.qos.logback:logback-core:1.1.2" level="project" />
<orderEntry type="library" name="Maven: junit:junit:4.11" level="project" />
<orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
<orderEntry type="library" name="Maven: redis.clients:jedis:2.8.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.commons:commons-pool2:2.4.2" level="project" />
<orderEntry type="library" name="Maven: org.apache.commons:commons-text:1.3" level="project" />
<orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.7" level="project" />
<orderEntry type="library" name="Maven: com.rabbitmq:amqp-client:4.0.0" level="project" />
</component>
</module>
logs/FixedViolationCodeParseResults.log
-3521
View File
File diff suppressed because it is too large Load Diff
logs/HaveSourceCode.txt
-87622
View File
File diff suppressed because it is too large Load Diff
logs/MultiMethodsOrFieldOrInnerClasses.txt
-3867
View File
File diff suppressed because it is too large Load Diff
logs/MultiV.txt
-4327
View File
File diff suppressed because it is too large Load Diff
logs/Null.txt
-124
View File
@@ -1,124 +0,0 @@
1
BC_VACUOUS_INSTANCEOF Ignore it. no source code, no source code change
CN_IDIOM_NO_SUPER_CALL Ignore it. no source code, no source code change
CN_IMPLEMENTS_CLONE_BUT_NOT_CLONEABLE Ignore it. no source code, no source code change
DLS_DEAD_LOCAL_STORE Ignore it. no source code, no source code change
DM_DEFAULT_ENCODING
DM_NUMBER_CTOR
DM_STRING_CTOR
DM_STRING_VOID_CTOR
DP_DO_INSIDE_DO_PRIVILEGED
EI_EXPOSE_REP
EQ_COMPARETO_USE_OBJECT_EQUALS
EQ_DOESNT_OVERRIDE_EQUALS
EQ_SELF_USE_OBJECT
HE_EQUALS_USE_HASHCODE
HE_HASHCODE_USE_OBJECT_EQUALS
HE_INHERITS_EQUALS_USE_HASHCODE
IS2_INCONSISTENT_SYNC
MS_CANNOT_BE_FINAL
MS_MUTABLE_ARRAY
MS_PKGPROTECT
MS_SHOULD_BE_FINAL
NM_CLASS_NAMING_CONVENTION
NM_CLASS_NOT_EXCEPTION
NM_SAME_SIMPLE_NAME_AS_SUPERCLASS
OBL_UNSATISFIED_OBLIGATION_EXCEPTION_EDGE
OS_OPEN_STREAM
OS_OPEN_STREAM_EXCEPTION_PATH
RI_REDUNDANT_INTERFACES
SA_FIELD_DOUBLE_ASSIGNMENT
SE_COMPARATOR_SHOULD_BE_SERIALIZABLE
SE_NO_SERIALVERSIONID
SIO_SUPERFLUOUS_INSTANCEOF
UCF_USELESS_CONTROL_FLOW
UG_SYNC_SET_UNSYNC_GET
UPM_UNCALLED_PRIVATE_METHOD
URF_UNREAD_FIELD
URF_UNREAD_PUBLIC_OR_PROTECTED_FIELD
UWF_FIELD_NOT_INITIALIZED_IN_CONSTRUCTOR
UWF_UNWRITTEN_FIELD
UWF_UNWRITTEN_PUBLIC_OR_PROTECTED_FIELD
WMI_WRONG_MAP_ITERATOR
-1
CI_CONFUSED_INHERITANCE
CO_ABSTRACT_SELF
DMI_COLLECTION_OF_URLS
EQ_ABSTRACT_SELF
HE_SIGNATURE_DECLARES_HASHING_OF_UNHASHABLE_CLASS
JCIP_FIELD_ISNT_FINAL_IN_IMMUTABLE_CLASS
ME_MUTABLE_ENUM_FIELD
MF_CLASS_MASKS_FIELD
MS_CANNOT_BE_FINAL
MS_PKGPROTECT
MS_SHOULD_BE_FINAL
NM_FIELD_NAMING_CONVENTION
NM_SAME_SIMPLE_NAME_AS_INTERFACE
SE_BAD_FIELD
SE_TRANSIENT_FIELD_NOT_RESTORED
STCAL_STATIC_SIMPLE_DATE_FORMAT_INSTANCE
UPM_UNCALLED_PRIVATE_METHOD
UUF_UNUSED_FIELD
UUF_UNUSED_PUBLIC_OR_PROTECTED_FIELD
UWF_NULL_FIELD
UWF_UNWRITTEN_FIELD
UWF_UNWRITTEN_PUBLIC_OR_PROTECTED_FIELD
VO_VOLATILE_REFERENCE_TO_ARRAY
CI_CONFUSED_INHERITANCE change modifier of fields from protected to private or public, or remove final of class.
CO_ABSTRACT_SELF Interface, remove compareTo() method.
SE_NO_SERIALVERSIONID add a field: serialVersionUID
EQ_ABSTRACT_SELF Interface, remove compareTo() method.
EQ_COMPARETO_USE_OBJECT_EQUALS: Update or Delete compareTo(), Add equals()
EQ_DOESNT_OVERRIDE_EQUALS override equals()
HE_SIGNATURE_DECLARES_HASHING_OF_UNHASHABLE_CLASS remove equals() only one instance
ME_MUTABLE_ENUM_FIELD enum, field add final.
MF_CLASS_MASKS_FIELD, change super class or delete the field with a same name in super class.
MS_SHOULD_BE_FINAL Add final key word
STCAL_STATIC_SIMPLE_DATE_FORMAT_INSTANCE remove public static final DateFormat DATE_FORMAT.... or SimpleDateFormat
UUF_UNUSED_FIELD remove unused fields. not sure
UUF_UNUSED_PUBLIC_OR_PROTECTED_FIELD field
UWF_NULL_FIELD update field, remove field
UWF_UNWRITTEN_FIELD field
UWF_UNWRITTEN_PUBLIC_OR_PROTECTED_FIELD remove field
VO_VOLATILE_REFERENCE_TO_ARRAY field
all GumTree results. class, field, method, //statements.
NM_SAME_SIMPLE_NAME_AS_INTERFACE rename class name or remove super class
NM_SAME_SIMPLE_NAME_AS_SUPERCLASS, class name
NM_METHOD_NAMING_CONVENTION
NM_CLASS_NAMING_CONVENTION class name
NM_CLASS_NOT_EXCEPTION class name
RI_REDUNDANT_INTERFACES class implemented interface
SE_NO_SUITABLE_CONSTRUCTOR Serializable Class level remove Serializable interface and serialVersionUID
SIC_INNER_SHOULD_BE_STATIC add static to inner class, not sure, delete it, 如果是匿名内部类，该如何处理？ ，其他的直接内部类的范围。
SIC_INNER_SHOULD_BE_STATIC_NEEDS_THIS 同上
===
CO_ABSTRACT_SELF inner class
SE_INNER_CLASS inner class
SE_COMPARATOR_SHOULD_BE_SERIALIZABLE, inner class, 读definition
SE_BAD_FIELD_INNER_CLASS inner class
SIC_INNER_SHOULD_BE_STATIC_ANON inner class
HE_INHERITS_EQUALS_USE_HASHCODE inner class or class
CN_IDIOM add clone method. or update clone method
DM_NUMBER_CTOR multi v multi patches
EQ_COMPARETO_USE_OBJECT_EQUALS, the same as before
IC_INIT_CIRCULARITY, multi V, multi patches>
SE_NO_SUITABLE_CONSTRUCTOR_FOR_EXTERNALIZATION, add a constructor or add public to old constructor,
NP_NONNULL_FIELD_NOT_INITIALIZED_IN_CONSTRUCTOR
SE_NO_SERIALVERSIONID change superclass or interface, add field or remove @SuppressWarnings("serial"), some are inner class
logs/NullSourceCode.txt
-1305
View File
File diff suppressed because it is too large Load Diff
logs/NullV.txt
-3130
View File
File diff suppressed because it is too large Load Diff
logs/fixedTypes.txt
-339
View File
@@ -1,339 +0,0 @@
SIC_INNER_SHOULD_BE_STATIC_ANON
DLS_DEAD_LOCAL_STORE
DM_CONVERT_CASE
DM_DEFAULT_ENCODING
UWF_FIELD_NOT_INITIALIZED_IN_CONSTRUCTOR
RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE
NM_METHOD_NAMING_CONVENTION
URF_UNREAD_FIELD
BC_UNCONFIRMED_CAST
REC_CATCH_EXCEPTION
BC_UNCONFIRMED_CAST_OF_RETURN_VALUE
SE_NO_SERIALVERSIONID
UPM_UNCALLED_PRIVATE_METHOD
VA_FORMAT_STRING_USES_NEWLINE
MS_SHOULD_BE_FINAL
RV_RETURN_VALUE_IGNORED_BAD_PRACTICE
ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD
EI_EXPOSE_REP2
URF_UNREAD_PUBLIC_OR_PROTECTED_FIELD
WMI_WRONG_MAP_ITERATOR
OBL_UNSATISFIED_OBLIGATION
EI_EXPOSE_REP
NP_LOAD_OF_KNOWN_NULL_VALUE
DM_NUMBER_CTOR
SIC_INNER_SHOULD_BE_STATIC
SBSC_USE_STRINGBUFFER_CONCATENATION
OS_OPEN_STREAM_EXCEPTION_PATH
NP_NONNULL_RETURN_VIOLATION
SF_SWITCH_NO_DEFAULT
UWF_UNWRITTEN_FIELD
DE_MIGHT_IGNORE
IS2_INCONSISTENT_SYNC
DM_BOXED_PRIMITIVE_FOR_PARSING
RV_RETURN_VALUE_IGNORED_NO_SIDE_EFFECT
ODR_OPEN_DATABASE_RESOURCE
PZLA_PREFER_ZERO_LENGTH_ARRAYS
RI_REDUNDANT_INTERFACES
NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE
UCF_USELESS_CONTROL_FLOW
UC_USELESS_CONDITION
NP_NULL_ON_SOME_PATH
UC_USELESS_OBJECT
DM_FP_NUMBER_CTOR
MS_PKGPROTECT
SQL_PREPARED_STATEMENT_GENERATED_FROM_NONCONSTANT_STRING
OBL_UNSATISFIED_OBLIGATION_EXCEPTION_EDGE
ES_COMPARING_STRINGS_WITH_EQ
OS_OPEN_STREAM
RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE
NP_PARAMETER_MUST_BE_NONNULL_BUT_MARKED_AS_NULLABLE
SS_SHOULD_BE_STATIC
FE_FLOATING_POINT_EQUALITY
HE_EQUALS_USE_HASHCODE
DM_EXIT
NM_CONFUSING
LI_LAZY_INIT_STATIC
BX_UNBOXING_IMMEDIATELY_REBOXED
EQ_COMPARETO_USE_OBJECT_EQUALS
SQL_NONCONSTANT_STRING_PASSED_TO_EXECUTE
NP_NULL_ON_SOME_PATH_EXCEPTION
NP_UNWRITTEN_FIELD
MS_MUTABLE_COLLECTION_PKGPROTECT
UWF_UNWRITTEN_PUBLIC_OR_PROTECTED_FIELD
CN_IMPLEMENTS_CLONE_BUT_NOT_CLONEABLE
DMI_HARDCODED_ABSOLUTE_FILENAME
SE_BAD_FIELD
DM_STRING_TOSTRING
UUF_UNUSED_FIELD
ICAST_INTEGER_MULTIPLY_CAST_TO_LONG
EC_UNRELATED_TYPES
BC_VACUOUS_INSTANCEOF
ICAST_IDIV_CAST_TO_DOUBLE
RCN_REDUNDANT_NULLCHECK_OF_NULL_VALUE
ES_COMPARING_PARAMETER_STRING_WITH_EQ
RV_RETURN_VALUE_IGNORED
CI_CONFUSED_INHERITANCE
DB_DUPLICATE_SWITCH_CLAUSES
ODR_OPEN_DATABASE_RESOURCE_EXCEPTION_PATH
NP_ALWAYS_NULL
RpC_REPEATED_CONDITIONAL_TEST
SE_BAD_FIELD_STORE
NP_NULL_PARAM_DEREF
GC_UNRELATED_TYPES
DLS_DEAD_LOCAL_STORE_OF_NULL
RC_REF_COMPARISON_BAD_PRACTICE
MS_MUTABLE_ARRAY
SE_INNER_CLASS
UL_UNRELEASED_LOCK_EXCEPTION_PATH
SE_COMPARATOR_SHOULD_BE_SERIALIZABLE
CN_IDIOM_NO_SUPER_CALL
NP_UNWRITTEN_PUBLIC_OR_PROTECTED_FIELD
UC_USELESS_VOID_METHOD
NP_NONNULL_PARAM_VIOLATION
DMI_INVOKING_TOSTRING_ON_ARRAY
DP_DO_INSIDE_DO_PRIVILEGED
SF_SWITCH_FALLTHROUGH
NP_BOOLEAN_RETURN_NULL
NM_CLASS_NOT_EXCEPTION
SIO_SUPERFLUOUS_INSTANCEOF
NP_GUARANTEED_DEREF
UL_UNRELEASED_LOCK
IP_PARAMETER_IS_DEAD_BUT_OVERWRITTEN
HE_INHERITS_EQUALS_USE_HASHCODE
MS_CANNOT_BE_FINAL
DB_DUPLICATE_BRANCHES
STCAL_INVOKE_ON_STATIC_DATE_FORMAT_INSTANCE
MS_MUTABLE_COLLECTION
IL_INFINITE_RECURSIVE_LOOP
VO_VOLATILE_INCREMENT
JLM_JSR166_UTILCONCURRENT_MONITORENTER
NP_NULL_PARAM_DEREF_NONVIRTUAL
DM_STRING_CTOR
UR_UNINIT_READ
XFB_XML_FACTORY_BYPASS
RV_EXCEPTION_NOT_THROWN
UI_INHERITANCE_UNSAFE_GETRESOURCE
VA_FORMAT_STRING_EXTRA_ARGUMENTS_PASSED
IT_NO_SUCH_ELEMENT
NM_CLASS_NAMING_CONVENTION
RC_REF_COMPARISON
RC_REF_COMPARISON_BAD_PRACTICE_BOOLEAN
DLS_DEAD_LOCAL_STORE_SHADOWS_FIELD
HE_EQUALS_NO_HASHCODE
NS_NON_SHORT_CIRCUIT
NP_DEREFERENCE_OF_READLINE_VALUE
NP_GUARANTEED_DEREF_ON_EXCEPTION_PATH
EQ_DOESNT_OVERRIDE_EQUALS
DM_STRING_VOID_CTOR
DP_CREATE_CLASSLOADER_INSIDE_DO_PRIVILEGED
BIT_SIGNED_CHECK
ME_ENUM_FIELD_SETTER
BC_EQUALS_METHOD_SHOULD_WORK_FOR_ALL_OBJECTS
NS_DANGEROUS_NON_SHORT_CIRCUIT
NP_EQUALS_SHOULD_HANDLE_NULL_ARGUMENT
SR_NOT_CHECKED
RR_NOT_CHECKED
UUF_UNUSED_PUBLIC_OR_PROTECTED_FIELD
VA_FORMAT_STRING_MISSING_ARGUMENT
EC_BAD_ARRAY_COMPARE
NP_NONNULL_FIELD_NOT_INITIALIZED_IN_CONSTRUCTOR
DM_BOXED_PRIMITIVE_FOR_COMPARE
NP_NULL_ON_SOME_PATH_MIGHT_BE_INFEASIBLE
UMAC_UNCALLABLE_METHOD_OF_ANONYMOUS_CLASS
UWF_NULL_FIELD
CO_COMPARETO_INCORRECT_FLOATING
NP_NULL_PARAM_DEREF_ALL_TARGETS_DANGEROUS
NO_NOTIFY_NOT_NOTIFYALL
SC_START_IN_CTOR
RV_RETURN_VALUE_IGNORED_INFERRED
UM_UNNECESSARY_MATH
SA_FIELD_SELF_ASSIGNMENT
BC_IMPOSSIBLE_CAST
LI_LAZY_INIT_UPDATE_STATIC
NP_TOSTRING_COULD_RETURN_NULL
SIC_INNER_SHOULD_BE_STATIC_NEEDS_THIS
AT_OPERATION_SEQUENCE_ON_CONCURRENT_ABSTRACTION
EQ_UNUSUAL
SE_NO_SUITABLE_CONSTRUCTOR
UW_UNCOND_WAIT
MF_CLASS_MASKS_FIELD
FE_TEST_IF_EQUAL_TO_NOT_A_NUMBER
IA_AMBIGUOUS_INVOCATION_OF_INHERITED_OR_OUTER_METHOD
SE_BAD_FIELD_INNER_CLASS
RV_NEGATING_RESULT_OF_COMPARETO
ISC_INSTANTIATE_STATIC_CLASS
HE_HASHCODE_USE_OBJECT_EQUALS
DM_BOOLEAN_CTOR
DMI_RANDOM_USED_ONLY_ONCE
EQ_CHECK_FOR_OPERAND_NOT_COMPATIBLE_WITH_THIS
DM_GC
NM_SAME_SIMPLE_NAME_AS_SUPERCLASS
EC_NULL_ARG
EQ_SELF_USE_OBJECT
RV_ABSOLUTE_VALUE_OF_RANDOM_INT
SE_NO_SUITABLE_CONSTRUCTOR_FOR_EXTERNALIZATION
MS_EXPOSE_REP
SE_TRANSIENT_FIELD_NOT_RESTORED
IM_AVERAGE_COMPUTATION_COULD_OVERFLOW
NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION
NP_METHOD_RETURN_RELAXING_ANNOTATION
UG_SYNC_SET_UNSYNC_GET
NM_FIELD_NAMING_CONVENTION
MS_OOI_PKGPROTECT
UCF_USELESS_CONTROL_FLOW_NEXT_LINE
VA_FORMAT_STRING_BAD_CONVERSION
BX_BOXING_IMMEDIATELY_UNBOXED_TO_PERFORM_COERCION
SA_FIELD_DOUBLE_ASSIGNMENT
BC_BAD_CAST_TO_CONCRETE_COLLECTION
SF_DEAD_STORE_DUE_TO_SWITCH_FALLTHROUGH
MS_FINAL_PKGPROTECT
DMI_COLLECTION_OF_URLS
INT_BAD_COMPARISON_WITH_SIGNED_BYTE
RV_CHECK_FOR_POSITIVE_INDEXOF
RE_CANT_USE_FILE_SEPARATOR_AS_REGULAR_EXPRESSION
RU_INVOKE_RUN
BC_IMPOSSIBLE_DOWNCAST_OF_TOARRAY
ICAST_BAD_SHIFT_AMOUNT
ICAST_INT_CAST_TO_FLOAT_PASSED_TO_ROUND
DM_BOXED_PRIMITIVE_TOSTRING
UR_UNINIT_READ_CALLED_FROM_SUPER_CONSTRUCTOR
WA_NOT_IN_LOOP
CN_IDIOM
DMI_BLOCKING_METHODS_ON_URL
EC_UNRELATED_CLASS_AND_INTERFACE
SWL_SLEEP_WITH_LOCK_HELD
IM_BAD_CHECK_FOR_ODD
RE_POSSIBLE_UNINTENDED_PATTERN
INT_BAD_COMPARISON_WITH_NONNEGATIVE_VALUE
ICAST_INT_CAST_TO_DOUBLE_PASSED_TO_CEIL
EQ_OVERRIDING_EQUALS_NOT_SYMMETRIC
LG_LOST_LOGGER_DUE_TO_WEAK_REFERENCE
DLS_DEAD_LOCAL_STORE_IN_RETURN
J2EE_STORE_OF_NON_SERIALIZABLE_OBJECT_INTO_SESSION
NP_STORE_INTO_NONNULL_FIELD
RANGE_ARRAY_INDEX
IL_INFINITE_LOOP
DC_DOUBLECHECK
NN_NAKED_NOTIFY
EC_UNRELATED_TYPES_USING_POINTER_EQUALITY
SE_PRIVATE_READ_RESOLVE_NOT_INHERITED
INT_VACUOUS_BIT_OPERATION
DM_NEXTINT_VIA_NEXTDOUBLE
VA_FORMAT_STRING_ILLEGAL
DMI_INVOKING_HASHCODE_ON_ARRAY
RV_DONT_JUST_NULL_CHECK_READLINE
RE_BAD_SYNTAX_FOR_REGULAR_EXPRESSION
SA_LOCAL_SELF_ASSIGNMENT_INSTEAD_OF_FIELD
NM_VERY_CONFUSING
MSF_MUTABLE_SERVLET_FIELD
BC_IMPOSSIBLE_INSTANCEOF
SA_LOCAL_SELF_COMPARISON
SA_LOCAL_SELF_ASSIGNMENT
IC_SUPERCLASS_USES_SUBCLASS_DURING_INITIALIZATION
CNT_ROUGH_CONSTANT_VALUE
SI_INSTANCE_BEFORE_FINALS_ASSIGNED
DLS_DEAD_STORE_OF_CLASS_LITERAL
RV_ABSOLUTE_VALUE_OF_HASHCODE
NM_SAME_SIMPLE_NAME_AS_INTERFACE
HE_HASHCODE_NO_EQUALS
DM_NEW_FOR_GETCLASS
SA_FIELD_SELF_COMPARISON
NM_METHOD_CONSTRUCTOR_CONFUSION
EQ_COMPARING_CLASS_NAMES
UC_USELESS_CONDITION_TYPE
RV_01_TO_INT
SA_LOCAL_DOUBLE_ASSIGNMENT
SQL_BAD_RESULTSET_ACCESS
EQ_GETCLASS_AND_CLASS_CONSTANT
BSHIFT_WRONG_ADD_PRIORITY
HRS_REQUEST_PARAMETER_TO_HTTP_HEADER
DMI_USING_REMOVEALL_TO_CLEAR_COLLECTION
STCAL_STATIC_SIMPLE_DATE_FORMAT_INSTANCE
FI_FINALIZER_NULLS_FIELDS
ML_SYNC_ON_FIELD_TO_GUARD_CHANGING_THAT_FIELD
JCIP_FIELD_ISNT_FINAL_IN_IMMUTABLE_CLASS
CO_COMPARETO_RESULTS_MIN_VALUE
IMSE_DONT_CATCH_IMSE
QF_QUESTIONABLE_FOR_LOOP
MWN_MISMATCHED_NOTIFY
RV_RETURN_VALUE_OF_PUTIFABSENT_IGNORED
DMI_NONSERIALIZABLE_OBJECT_WRITTEN
BC_BAD_CAST_TO_ABSTRACT_COLLECTION
SW_SWING_METHODS_INVOKED_IN_SWING_THREAD
EC_ARRAY_AND_NONARRAY
NP_NULL_INSTANCEOF
MTIA_SUSPECT_SERVLET_INSTANCE_FIELD
EQ_ALWAYS_FALSE
INT_BAD_COMPARISON_WITH_INT_VALUE
NP_OPTIONAL_RETURN_NULL
BX_BOXING_IMMEDIATELY_UNBOXED
VA_FORMAT_STRING_EXPECTED_MESSAGE_FORMAT_SUPPLIED
BIT_IOR
FI_MISSING_SUPER_CALL
VO_VOLATILE_REFERENCE_TO_ARRAY
SF_DEAD_STORE_DUE_TO_SWITCH_FALLTHROUGH_TO_THROW
DMI_CONSTANT_DB_PASSWORD
NP_IMMEDIATE_DEREFERENCE_OF_READLINE
ML_SYNC_ON_UPDATED_FIELD
HE_USE_OF_UNHASHABLE_CLASS
FI_PUBLIC_SHOULD_BE_PROTECTED
DMI_BIGDECIMAL_CONSTRUCTED_FROM_DOUBLE
TQ_NEVER_VALUE_USED_WHERE_ALWAYS_REQUIRED
XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER
NP_ALWAYS_NULL_EXCEPTION
DM_INVALID_MIN_MAX
NM_WRONG_PACKAGE
MS_SHOULD_BE_REFACTORED_TO_BE_FINAL
DC_PARTIALLY_CONSTRUCTED
WA_AWAIT_NOT_IN_LOOP
INT_VACUOUS_COMPARISON
RV_CHECK_COMPARETO_FOR_SPECIFIC_RETURN_VALUE
RCN_REDUNDANT_COMPARISON_OF_NULL_AND_NONNULL_VALUE
DL_SYNCHRONIZATION_ON_SHARED_CONSTANT
EQ_ALWAYS_TRUE
DMI_EMPTY_DB_PASSWORD
DMI_ENTRY_SETS_MAY_REUSE_ENTRY_OBJECTS
WL_USING_GETCLASS_RATHER_THAN_CLASS_LITERAL
FI_USELESS
VA_FORMAT_STRING_BAD_CONVERSION_FROM_ARRAY
SP_SPIN_ON_FIELD
BOA_BADLY_OVERRIDDEN_ADAPTER
MWN_MISMATCHED_WAIT
NP_CLONE_COULD_RETURN_NULL
CO_ABSTRACT_SELF
DMI_COLLECTIONS_SHOULD_NOT_CONTAIN_THEMSELVES
QBA_QUESTIONABLE_BOOLEAN_ASSIGNMENT
IC_INIT_CIRCULARITY
ICAST_INT_2_LONG_AS_INSTANT
FI_EXPLICIT_INVOCATION
ME_MUTABLE_ENUM_FIELD
RCN_REDUNDANT_COMPARISON_TWO_NULL_VALUES
HE_SIGNATURE_DECLARES_HASHING_OF_UNHASHABLE_CLASS
RS_READOBJECT_SYNC
SQL_BAD_PREPARED_STATEMENT_ACCESS
VA_FORMAT_STRING_BAD_CONVERSION_TO_BOOLEAN
EQ_ABSTRACT_SELF
NP_CLOSING_NULL
RV_REM_OF_HASHCODE
PT_RELATIVE_PATH_TRAVERSAL
TLW_TWO_LOCK_WAIT
INT_BAD_REM_BY_1
ICAST_QUESTIONABLE_UNSIGNED_RIGHT_SHIFT
VA_PRIMITIVE_ARRAY_PASSED_TO_OBJECT_VARARG
DM_USELESS_THREAD
DMI_DOH
IJU_ASSERT_METHOD_INVOKED_FROM_RUN_METHOD
BIT_IOR_OF_SIGNED_BYTE
DMI_THREAD_PASSED_WHERE_RUNNABLE_EXPECTED
SA_LOCAL_SELF_COMPUTATION
This method performs a nonsensical computation of a local variable with another reference to the same variable (e.g., x&x or x-x).
Because of the nature of the computation, this operation doesn't seem to make sense, and may indicate a typo or a logic error. Double check the computation.
IM_MULTIPLYING_RESULT_OF_IREM
The code multiplies the result of an integer remaining by an integer constant.
Be sure you don't have your operator precedence confused. For example i % 60 * 1000 is (i % 60) * 1000, not i % (60 * 1000).
MS_MUTABLE_HASHTABLE
Field is a mutable Hashtable (MS_MUTABLE_HASHTABLE)
A final static field references a Hashtable and can be accessed by malicious code or by accident from another package.
This code can freely modify the contents of the Hashtable.
logs/myLog.log
View File
pom.xml
+87 -14
View File
@@ -13,15 +13,23 @@
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<akka.version>2.4.11</akka.version>
<logback.version>1.1.2</logback.version>
<slf4j-api.version>1.7.7</slf4j-api.version>
</properties>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.javatuples/javatuples -->
<dependency>
<groupId>edu.lu.uni</groupId>
<artifactId>simple-utils</artifactId>
<version>0.0.1-SNAPSHOT</version>
<groupId>org.javatuples</groupId>
<artifactId>javatuples</artifactId>
<version>1.2</version>
</dependency>
<!--<dependency>-->
<!--<groupId>edu.lu.uni</groupId>-->
<!--<artifactId>simple-utils</artifactId>-->
<!--<version>0.0.1-SNAPSHOT</version>-->
<!--</dependency>-->
<!-- akka -->
<dependency>
@@ -41,18 +49,32 @@
<artifactId>gen.jdt</artifactId>
<version>2.0.0-SNAPSHOT</version>
</dependency>
<!-- SLF4J - API -->
<dependency>
<groupId>edu.lu.uni.serval</groupId>
<artifactId>SourceCodeParser</artifactId>
<version>0.0.1-SNAPSHOT</version>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j-api.version}</version>
</dependency>
<!-- logback -->
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>${logback.version}</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
<version>${logback.version}</version>
</dependency>
<dependency>
<groupId>edu.lu.uni.serval</groupId>
<artifactId>GitTraveller</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
<!--<dependency>-->
<!--<groupId>edu.lu.uni.serval</groupId>-->
<!--<artifactId>GitTraveller</artifactId>-->
<!--<version>0.0.1-SNAPSHOT</version>-->
<!--</dependency>-->
<dependency>
<groupId>junit</groupId>
@@ -60,7 +82,33 @@
<version>4.12</version>
<scope>test</scope>
</dependency>
</dependencies>
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>2.8.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.commons/commons-text -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-text</artifactId>
<version>1.3</version>
</dependency>
<dependency>
<groupId>com.rabbitmq</groupId>
<artifactId>amqp-client</artifactId>
<version>4.0.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
</dependency>
</dependencies>
<build>
<plugins>
@@ -73,6 +121,31 @@
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<archive>
<manifest>
<mainClass>
edu.lu.uni.serval.fixminer.Launcher
</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
src/main/java/edu/lu/uni/serval/FixPattern/utils/ASTNodeMap.java
+8 -1
View File
@@ -103,7 +103,14 @@ public class ASTNodeMap {
map.put(88, "NameQualifiedType");
map.put(89, "CreationReference");
map.put(90, "ExpressionMethodReference");
map.put(91, "SuperMethhodReference");
map.put(91, "SuperMethodReference");
map.put(92, "TypeMethodReference");
map.put(100,"Insert");
map.put(101,"Update");
map.put(102,"Delete");
map.put(103,"Move");
map.put(104,"NoChange");
}
}
src/main/java/edu/lu/uni/serval/FixPattern/utils/EDiff.java
+98
View File
@@ -0,0 +1,98 @@
package edu.lu.uni.serval.FixPattern.utils;
import com.github.gumtreediff.actions.model.*;
import com.github.gumtreediff.tree.ITree;
import com.github.gumtreediff.tree.TreeContext;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalActionSet;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import java.io.IOException;
import java.util.List;
/**
* Created by anilkoyuncu on 17/09/2018.
*/
public class EDiff {
private static Logger log = LoggerFactory.getLogger(EDiff.class);
public static ITree getSimpliedTree(String fn, JedisPool outerPool) {
ITree tree = null;
Jedis inner = null;
try {
inner = outerPool.getResource();
while (!inner.ping().equals("PONG")){
log.info("wait");
}
inner.select(1);
String dist2load = inner.get(fn);
inner.select(0);
String s = inner.get(dist2load);
HierarchicalActionSet actionSet = (HierarchicalActionSet) EDiffHelper.fromString(s);
ITree parent = null;
ITree children =null;
TreeContext tc = new TreeContext();
tree = EDiffHelper.getASTTree(actionSet, parent, children,tc);
tree.setParent(null);
tc.validate();
} catch (IOException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
}finally {
if (inner != null) {
inner.close();
}
}
return tree;
}
public static ITree getActionTree(HierarchicalActionSet actionSet, ITree parent, ITree children,TreeContext tc){
int newType = 0;
Action action = actionSet.getAction();
if (action instanceof Update){
newType = 101;
}else if(action instanceof Insert){
newType =100;
}else if(action instanceof Move){
newType = 102;
}else if(action instanceof Delete){
newType=103;
}else{
new Exception("unknow action");
}
if(actionSet.getParent() == null){
//root
parent = tc.createTree(newType, "", null);
tc.setRoot(parent);
// parent = new Tree(newType,"");
}else{
children = tc.createTree(newType, "", null);
children.setParentAndUpdateChildren(parent);
// children = new Tree(newType,"");
// parent.addChild(children);
}
List<HierarchicalActionSet> subActions = actionSet.getSubActions();
if (subActions.size() != 0){
for (HierarchicalActionSet subAction : subActions) {
if(actionSet.getParent() == null){
children = parent;
}
getActionTree(subAction,children,null,tc);
}
}
return parent;
}
}
src/main/java/edu/lu/uni/serval/FixPattern/utils/EDiffHelper.java
+88
View File
@@ -0,0 +1,88 @@
package edu.lu.uni.serval.FixPattern.utils;
import com.github.gumtreediff.tree.ITree;
import com.github.gumtreediff.tree.TreeContext;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalActionSet;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.*;
import java.util.Base64;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.stream.Collectors;
/**
* Created by anilkoyuncu on 17/09/2018.
*/
public class EDiffHelper {
private static Logger log = LoggerFactory.getLogger(EDiffHelper.class);
/** Read the object from Base64 string. */
public static Object fromString( String s ) throws IOException,
ClassNotFoundException {
byte [] data = Base64.getDecoder().decode( s );
ObjectInputStream ois = new ObjectInputStream(
new ByteArrayInputStream( data ) );
Object o = ois.readObject();
ois.close();
return o;
}
/** Write the object to a Base64 string. */
public static String toString( Serializable o ) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream( baos );
oos.writeObject( o );
oos.close();
return Base64.getEncoder().encodeToString(baos.toByteArray());
}
public static ITree getASTTree(HierarchicalActionSet actionSet, ITree parent, ITree children, TreeContext tc){
int newType = 0;
String astNodeType = actionSet.getAstNodeType();
List<Integer> keysByValue = getKeysByValue(ASTNodeMap.map, astNodeType);
if(keysByValue.size() != 1){
log.error("More than 1");
}
newType = keysByValue.get(0);
if(actionSet.getParent() == null){
//root
// parent = new Tree(newType,"");
parent = tc.createTree(newType, "", null);
tc.setRoot(parent);
}else{
// children = new Tree(newType,"");
// parent.addChild(children);
children = tc.createTree(newType, "", null);
children.setParentAndUpdateChildren(parent);
}
List<HierarchicalActionSet> subActions = actionSet.getSubActions();
if (subActions.size() != 0){
for (HierarchicalActionSet subAction : subActions) {
if(actionSet.getParent() == null){
children = parent;
}
getASTTree(subAction,children,null,tc);
}
}
return parent;
}
public static <T, E> List<T> getKeysByValue(Map<T, E> map, E value) {
return map.entrySet()
.stream()
.filter(entry -> Objects.equals(entry.getValue(), value))
.map(Map.Entry::getKey)
.collect(Collectors.toList());
}
}
src/main/java/edu/lu/uni/serval/FixPattern/utils/PoolBuilder.java
+33
View File
@@ -0,0 +1,33 @@
package edu.lu.uni.serval.FixPattern.utils;
import redis.clients.jedis.JedisPoolConfig;
import java.time.Duration;
/**
* Created by anilkoyuncu on 17/09/2018.
*/
public class PoolBuilder {
public static JedisPoolConfig getPoolConfig() {
return poolConfig;
}
static final JedisPoolConfig poolConfig = buildPoolConfig();
private static JedisPoolConfig buildPoolConfig() {
final JedisPoolConfig poolConfig = new JedisPoolConfig();
poolConfig.setMaxTotal(128);
poolConfig.setMaxIdle(128);
poolConfig.setMinIdle(16);
poolConfig.setTestOnBorrow(true);
poolConfig.setTestOnReturn(true);
poolConfig.setTestWhileIdle(true);
poolConfig.setMinEvictableIdleTimeMillis(Duration.ofMinutes(60).toMillis());
poolConfig.setTimeBetweenEvictionRunsMillis(Duration.ofHours(30).toMillis());
poolConfig.setNumTestsPerEvictionRun(3);
poolConfig.setBlockWhenExhausted(true);
return poolConfig;
}
}
src/main/java/edu/lu/uni/serval/FixPatternParser/CUCreator.java
+2 -6
View File
@@ -1,15 +1,11 @@
package edu.lu.uni.serval.FixPatternParser;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import org.eclipse.jdt.core.dom.AST;
import org.eclipse.jdt.core.dom.ASTParser;
import org.eclipse.jdt.core.dom.CompilationUnit;
import java.io.*;
/**
* Creator of a CompilationUnit.
*
src/main/java/edu/lu/uni/serval/FixPatternParser/Parser.java
+14 -454
View File
@@ -1,20 +1,6 @@
package edu.lu.uni.serval.FixPatternParser;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import com.github.gumtreediff.actions.model.Action;
import com.github.gumtreediff.actions.model.Move;
import com.github.gumtreediff.tree.ITree;
import edu.lu.uni.serval.gumtree.GumTreeComparer;
import edu.lu.uni.serval.gumtree.regroup.ActionFilter;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalActionSet;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalRegrouper;
import edu.lu.uni.serval.gumtree.regroup.SimpleTree;
import edu.lu.uni.serval.gumtree.regroup.SimplifyTree;
/**
* Parse fix patterns with GumTree.
@@ -32,435 +18,9 @@ public abstract class Parser implements ParserInterface {
protected String originalTree = ""; // Guide of generating patches.
protected String actionSets = ""; // Guide of generating patches.
public abstract void parseFixPatterns(File prevFile, File revFile, File diffEntryFile);
public abstract void parseFixPatterns(File prevFile, File revFile, File diffEntryFile,String project);
protected List<HierarchicalActionSet> parseChangedSourceCodeWithGumTree(File prevFile, File revFile) {
List<HierarchicalActionSet> actionSets = new ArrayList<>();
// GumTree results
List<Action> gumTreeResults = new GumTreeComparer().compareTwoFilesWithGumTree(prevFile, revFile);
if (gumTreeResults != null && gumTreeResults.size() > 0) {
// Regroup GumTre results.
List<HierarchicalActionSet> allActionSets = new HierarchicalRegrouper().regroupGumTreeResults(gumTreeResults);
// Filter out modified actions of changing method names, method parameters, variable names and field names in declaration part.
// TODO: variable effects range, sub-actions are these kinds of modification?
actionSets.addAll(new ActionFilter().filterOutUselessActions(allActionSets));
}
return actionSets;
}
protected List<Move> getFirstAndLastMoveAction(HierarchicalActionSet gumTreeResult) {
List<Move> firstAndLastMoveActions = new ArrayList<>();
List<HierarchicalActionSet> actions = gumTreeResult.getSubActions();
if (actions.size() == 0) {
return null;
}
Move firstMoveAction = null;
Move lastMoveAction = null;
while (actions.size() > 0) {
List<HierarchicalActionSet> subActions = new ArrayList<>();
for (HierarchicalActionSet action : actions) {
subActions.addAll(action.getSubActions());
if (action.toString().startsWith("MOV")) {
if (firstMoveAction == null) {
firstMoveAction = (Move) action.getAction();
lastMoveAction = (Move) action.getAction();
} else {
int startPosition = action.getStartPosition();
int length = action.getLength();
int startPositionFirst = firstMoveAction.getPosition();
int startPositionLast = lastMoveAction.getPosition();
int lengthLast = lastMoveAction.getNode().getLength();
if (startPosition < startPositionFirst || (startPosition == startPositionFirst && length > firstMoveAction.getLength())) {
firstMoveAction = (Move) action.getAction();
}
if ((startPosition + length) > (startPositionLast + lengthLast)) {
lastMoveAction = (Move) action.getAction();
}
}
}
}
actions.clear();
actions.addAll(subActions);
}
if (firstMoveAction == null) {
return null;
}
firstAndLastMoveActions.add(firstMoveAction);
firstAndLastMoveActions.add(lastMoveAction);
return firstAndLastMoveActions;
}
protected String getSemiSourceCodeEditScripts(HierarchicalActionSet actionSet) {
// TODO Auto-generated method stub
return null;
}
protected String getAbstractIdentifiersEditScripts(HierarchicalActionSet actionSet) {
// TODO Auto-generated method stub
return null;
}
protected String getRawTokenEditScripts(HierarchicalActionSet actionSet) {
// TODO Auto-generated method stub
return null;
}
protected int getEndPosition(List<ITree> children) {
int endPosition = 0;
for (ITree child : children) {
if (child.getLabel().endsWith("Body")) {
endPosition = child.getPos() - 1;
break;
}
}
return endPosition;
}
/**
* Get the AST node based edit script of patches in terms of breadth first.
*
* @param actionSet
* @return
*/
protected String getASTEditScriptsBreadthFirst(HierarchicalActionSet actionSet) {
String editScript = "";
List<HierarchicalActionSet> actionSets = new ArrayList<>();
actionSets.add(actionSet);
while (actionSets.size() != 0) {
List<HierarchicalActionSet> subSets = new ArrayList<>();
for (HierarchicalActionSet set : actionSets) {
subSets.addAll(set.getSubActions());
String actionStr = set.getActionString();
int index = actionStr.indexOf("@@");
String singleEdit = actionStr.substring(0, index).replace(" ", "");
singleEdit = handleSimpleNameNode(singleEdit, actionStr, index);
editScript += singleEdit + " ";
}
actionSets.clear();
actionSets.addAll(subSets);
}
return editScript;
}
private String handleSimpleNameNode(String singleEdit, String actionStr, int index) {
if (singleEdit.endsWith("SimpleName")) {
actionStr = actionStr.substring(index + 2);
if (actionStr.startsWith("MethodName")) {
singleEdit = singleEdit.replace("SimpleName", "MethodName");
} else if (actionStr.startsWith("ClassName")) {
singleEdit = singleEdit.replace("SimpleName", "ClassName");
} else {
if (actionStr.startsWith("Name")) {
char c = actionStr.charAt(5);
if (Character.isUpperCase(c)) {
singleEdit = singleEdit.replace("SimpleName", "Name");
} else {
singleEdit = singleEdit.replace("SimpleName", "Variable");
}
} else {
singleEdit = singleEdit.replace("SimpleName", "Variable");
}
}
}
return singleEdit;
}
/**
* Get the AST node based edit script of patches in terms of deep first.
*
* @param actionSet
* @return
*/
protected String getASTEditScriptsDeepFirst(HierarchicalActionSet actionSet) {
String editScript = "";
String actionStr = actionSet.getActionString();
int index = actionStr.indexOf("@@");
String singleEdit = actionStr.substring(0, index).replace(" ", "");
singleEdit = handleSimpleNameNode(singleEdit, actionStr, index);
editScript = singleEdit + " ";
List<HierarchicalActionSet> subActionSets = actionSet.getSubActions();;
for (HierarchicalActionSet subActionSet : subActionSets) {
editScript += getASTEditScriptsDeepFirst(subActionSet);
}
return editScript;
}
/**
*
* @param hunkActionSets
* @param bugStartLine
* @param bugEndLine
* @param fixStartLine
* @param fixEndLine
* @return
*/
protected String getASTEditScriptsBreadthFirst(List<HierarchicalActionSet> hunkActionSets, int bugEndPosition, int fixEndPosition) {
String editScript = "";
for (HierarchicalActionSet hunkActionSet : hunkActionSets) {
editScript += getASTEditScriptsBreadthFirst(hunkActionSet, bugEndPosition, fixEndPosition);
}
return editScript;
}
private String getASTEditScriptsBreadthFirst(HierarchicalActionSet hunkActionSet, int bugEndPosition, int fixEndPosition) {
String editScript = "";
List<HierarchicalActionSet> actionSets = new ArrayList<>();
actionSets.add(hunkActionSet);
while (actionSets.size() != 0) {
List<HierarchicalActionSet> subSets = new ArrayList<>();
for (HierarchicalActionSet set : actionSets) {
int position = set.getAction().getPosition();
String actionStr = set.getActionString();
if (isOutofPosition(actionStr, position, bugEndPosition, fixEndPosition)) {
continue;
}
subSets.addAll(set.getSubActions());
int index = actionStr.indexOf("@@");
String singleEdit = actionStr.substring(0, index).replace(" ", "");
singleEdit = handleSimpleNameNode(singleEdit, actionStr, index);
editScript += singleEdit + " ";
}
actionSets.clear();
actionSets.addAll(subSets);
}
return editScript;
}
private boolean isOutofPosition(String actionStr, int actionPosition, int bugEndPosition, int fixEndPosition) {
if (actionStr.startsWith("INS")) {
if (actionPosition > fixEndPosition) {
return true;
}
} else if (!actionStr.startsWith("MOV") && actionPosition > bugEndPosition) {
return true;
}
return false;
}
protected String getASTEditScriptsDeepFirst(List<HierarchicalActionSet> hunkActionSets, int bugEndPosition, int fixEndPosition) {
StringBuilder editScript = new StringBuilder();
for (HierarchicalActionSet hunkActionSet : hunkActionSets) {
editScript.append(getASTEditScriptsDeepFirst2(hunkActionSet, bugEndPosition, fixEndPosition));
}
return editScript.toString();
}
private String getASTEditScriptsDeepFirst(HierarchicalActionSet actionSet, int bugEndPosition, int fixEndPosition) {
String editScripts = "";
String actionStr = actionSet.getActionString();
int index = actionStr.indexOf("@@");
String singleEdit = actionStr.substring(0, index).replace(" ", "");
singleEdit = handleSimpleNameNode(singleEdit, actionStr, index);
editScripts += singleEdit + " ";
for (HierarchicalActionSet subActionSet : actionSet.getSubActions()) {
int position = subActionSet.getAction().getPosition();
actionStr = subActionSet.getActionString();
if (isOutofPosition(actionStr, position, bugEndPosition, fixEndPosition)) {
continue;
}
editScripts += getASTEditScriptsDeepFirst(subActionSet, bugEndPosition, fixEndPosition);
}
return editScripts;
}
private String getASTEditScriptsDeepFirst2(HierarchicalActionSet actionSet, int bugEndPosition, int fixEndPosition) {
StringBuilder editScripts = new StringBuilder();
String actionStr = actionSet.getActionString();
int index = actionStr.indexOf("@@");
String singleEdit = actionStr.substring(0, index);
if (singleEdit.endsWith("Statement")) {
singleEdit = singleEdit + " " + singleEdit.substring(4, singleEdit.indexOf("Statement")).toLowerCase(Locale.ENGLISH);
} else {
singleEdit = handleSimpleNameNode2(singleEdit, actionStr, index, actionSet);
}
editScripts.append(singleEdit).append(" ");
for (HierarchicalActionSet subActionSet : actionSet.getSubActions()) {
int position = subActionSet.getAction().getPosition();
actionStr = subActionSet.getActionString();
if (isOutofPosition(actionStr, position, bugEndPosition, fixEndPosition)) {
continue;
}
editScripts.append(getASTEditScriptsDeepFirst2(subActionSet, bugEndPosition, fixEndPosition));
}
return editScripts.toString();
}
private String handleSimpleNameNode2(String singleEdit, String actionStr, int index, HierarchicalActionSet actionSet) {
if (singleEdit.endsWith("SimpleName")) {
actionStr = actionStr.substring(index + 2);
if (actionStr.startsWith("MethodName")) {
singleEdit = singleEdit.replace("SimpleName", "MethodName");
String methodName = actionStr.substring(actionStr.indexOf("MethodName:") + 11);
int index1 = methodName.indexOf(":");
int index2 = methodName.indexOf(" ");
index = (index1 < 0 || index1 > index2) ? index2 : index1;
methodName = methodName.substring(0, index);
singleEdit += " " + methodName;
} else if (actionStr.startsWith("ClassName")) {
singleEdit = singleEdit.replace("SimpleName", "ClassName");
String className = actionStr.substring(actionStr.indexOf("ClassName:") + 10);
int index1 = className.indexOf(" ");
index = index1 < 0 ? className.length() : index1;
className = className.substring(0, index);
singleEdit += " " + className;
} else {
if (actionStr.startsWith("Name")) {
char c = actionStr.charAt(5);
if (Character.isUpperCase(c)) {
singleEdit = singleEdit.replace("SimpleName", "Name");
String name = actionStr.substring(actionStr.indexOf("Name:") + 5);
int index1 = name.indexOf(" ");
index = index1 < 0 ? name.length() : index1;
name = name.substring(0, index);
singleEdit += " " + name;
} else {
singleEdit = singleEdit.replace("SimpleName", "Variable");
int index1 = actionStr.indexOf(" ");
index = index1 < 0 ? actionStr.length() : index1;
String var = actionStr.substring(0, index);
var = new SimplifyTree().canonicalVariableName(var, actionSet.getAction().getNode());
singleEdit += " " + var.replaceAll(" ", "");
}
} else {
singleEdit = singleEdit.replace("SimpleName", "Variable");
String var = actionStr.substring(0, (actionStr.indexOf(" ") < 0 ? actionStr.length() : actionStr.indexOf(" ")));
var = new SimplifyTree().canonicalVariableName(var, actionSet.getAction().getNode());
singleEdit += " " + var.replaceAll(" ", "");
}
}
} else {
if (actionSet.getSubActions() != null && actionSet.getSubActions().size() > 0) {
singleEdit += " " + actionSet.getAstNodeType()+ "exp";
} else {
if (singleEdit.endsWith("CharacterLiteral")) {
singleEdit += " charLiteral";
} else if (singleEdit.endsWith("NumberLiteral")) {
singleEdit += " numLiteral";
} else if (singleEdit.endsWith("StringLiteral")) {
singleEdit += " strLiteral";
} else if (singleEdit.endsWith("BooleanLiteral") || singleEdit.endsWith("NullLiteral") || singleEdit.endsWith("Operator") ||
singleEdit.endsWith("ThisExpression") || singleEdit.endsWith("TypeLiteral") || singleEdit.endsWith("Instanceof") ||
singleEdit.endsWith("New") || singleEdit.endsWith("WildcardType") || singleEdit.endsWith("SimpleType") ||
singleEdit.endsWith("QualifiedType") || singleEdit.endsWith("PrimitiveType") || singleEdit.endsWith("NameQualifiedType")) {
singleEdit += " " + actionSet.getNode().getLabel();
} else if (singleEdit.endsWith("SuperConstructorInvocation")) {
singleEdit += " super";
} else if (singleEdit.endsWith("ConstructorInvocation")) {
singleEdit += " this";
} else if (singleEdit.endsWith("SimpleName")) {
actionStr = actionStr.substring(index + 2);
if (actionStr.startsWith("MethodName")) {
singleEdit = singleEdit.replace("SimpleName", "MethodName");
String methodName = actionStr.substring(actionStr.indexOf("MethodName:") + 11);
int index1 = methodName.indexOf(":");
int index2 = methodName.indexOf(" ");
index = (index1 < 0 || index1 > index2) ? index2 : index1;
methodName = methodName.substring(0, index);
singleEdit += " " + methodName;
} else if (actionStr.startsWith("ClassName")) {
singleEdit = singleEdit.replace("SimpleName", "ClassName");
String className = actionStr.substring(actionStr.indexOf("ClassName:") + 10);
int index1 = className.indexOf(" ");
index = index1 < 0 ? className.length() : index1;
className = className.substring(0, index);
singleEdit += " " + className;
} else {
if (actionStr.startsWith("Name")) {
char c = actionStr.charAt(5);
if (Character.isUpperCase(c)) {
singleEdit = singleEdit.replace("SimpleName", "Name");
String name = actionStr.substring(actionStr.indexOf("Name:") + 5);
int index1 = name.indexOf(" ");
index = index1 < 0 ? name.length() : index1;
name = name.substring(0, index);
singleEdit += " " + name;
} else {
singleEdit = singleEdit.replace("SimpleName", "Variable");
int index1 = actionStr.indexOf(" ");
index = index1 < 0 ? actionStr.length() : index1;
String var = actionStr.substring(0, index);
var = new SimplifyTree().canonicalVariableName(var, actionSet.getAction().getNode());
singleEdit += " " + var.replaceAll(" ", "");
}
} else {
singleEdit = singleEdit.replace("SimpleName", "Variable");
String var = actionStr.substring(0, (actionStr.indexOf(" ") < 0 ? actionStr.length() : actionStr.indexOf(" ")));
var = new SimplifyTree().canonicalVariableName(var, actionSet.getAction().getNode());
singleEdit += " " + var.replaceAll(" ", "");
}
}
} else {
ITree nodeTree = actionSet.getNode();
if (nodeTree.getChildren().size() > 0) {
singleEdit += " " + actionSet.getAstNodeType()+ "exp";
} else {
singleEdit += " " + nodeTree.getLabel().replaceAll(" ", "");
}
}
}
}
return singleEdit;
}
protected String getAstEditScripts(List<HierarchicalActionSet> actionSets, int bugEndLine, int fixEndLine) {
String editScripts = "";
for (HierarchicalActionSet actionSet : actionSets) {
editScripts += getActionString(actionSet, bugEndLine, fixEndLine, "");
}
return editScripts;
}
private String getActionString(HierarchicalActionSet actionSet, int bugEndPosition, int fixEndPosition, String startStr) {
String editScripts = "";
String actionStr = actionSet.getActionString();
int index = actionStr.indexOf("@@");
String singleEdit = actionStr.substring(0, index);
singleEdit = handleSimpleNameNode(singleEdit, actionStr, index);
editScripts += startStr + singleEdit + "\n";
for (HierarchicalActionSet subActionSet : actionSet.getSubActions()) {
int position = subActionSet.getAction().getPosition();
actionStr = subActionSet.getActionString();
if (isOutofPosition(actionStr, position, bugEndPosition, fixEndPosition)) {
continue;
}
editScripts += getActionString(subActionSet, bugEndPosition, fixEndPosition, startStr + "---");
}
return editScripts;
}
protected void clearITree(HierarchicalActionSet actionSet) {
actionSet.getAction().setNode(null);
for (HierarchicalActionSet subActionSet : actionSet.getSubActions()) {
clearITree(subActionSet);
}
}
@Override
public String getAstEditScripts() {
return astEditScripts;
@@ -471,10 +31,10 @@ public abstract class Parser implements ParserInterface {
return patchesSourceCode;
}
@Override
public String getBuggyTrees() {
return buggyTrees;
}
// @Override
// public String getBuggyTrees() {
// return buggyTrees;
// }
@Override
public String getSizes() {
@@ -486,13 +46,13 @@ public abstract class Parser implements ParserInterface {
return tokensOfSourceCode;
}
@Override
public String getOriginalTree() {
return originalTree;
}
@Override
public String getActionSets() {
return actionSets;
}
// @Override
// public String getOriginalTree() {
// return originalTree;
// }
//
// @Override
// public String getActionSets() {
// return actionSets;
// }
}
src/main/java/edu/lu/uni/serval/FixPatternParser/ParserInterface.java
+4 -6
View File
@@ -1,23 +1,21 @@
package edu.lu.uni.serval.FixPatternParser;
import java.io.File;
public interface ParserInterface {
public void parseFixPatterns(File prevFile, File revFile, File diffEntryFile);
// public void parseFixPatterns(File prevFile, File revFile, File diffEntryFile);
public String getAstEditScripts();
public String getPatchesSourceCode();
public String getBuggyTrees();
// public String getBuggyTrees();
public String getSizes();
public String getTokensOfSourceCode();
public String getOriginalTree();
// public String getOriginalTree();
public String getActionSets();
// public String getActionSets();
}
src/main/java/edu/lu/uni/serval/FixPatternParser/RunnableParser.java
+10 -1
View File
@@ -8,6 +8,7 @@ public class RunnableParser implements Runnable {
private File revFile;
private File diffentryFile;
private Parser parser;
private String project;
public RunnableParser(File prevFile, File revFile, File diffentryFile, Parser parser) {
this.prevFile = prevFile;
@@ -16,8 +17,16 @@ public class RunnableParser implements Runnable {
this.parser = parser;
}
public RunnableParser(File prevFile, File revFile, File diffentryFile, Parser parser,String project) {
this.prevFile = prevFile;
this.revFile = revFile;
this.diffentryFile = diffentryFile;
this.parser = parser;
this.project = project;
}
@Override
public void run() {
parser.parseFixPatterns(prevFile, revFile, diffentryFile);
parser.parseFixPatterns(prevFile, revFile, diffentryFile,project);
}
}
src/main/java/edu/lu/uni/serval/FixPatternParser/Tokenizer.java
-75
View File
@@ -1,75 +0,0 @@
package edu.lu.uni.serval.FixPatternParser;
import java.util.List;
import edu.lu.uni.serval.gumtree.regroup.SimpleTree;
public class Tokenizer {
public static String getTokensDeepFirst(SimpleTree simpleTree) {
String tokens = "";
List<SimpleTree> children = simpleTree.getChildren();
String astNodeType = simpleTree.getNodeType();
if (children.isEmpty()) { // BreakStatement, ContinueStatement, ReturnStatement, TryStatement
if (astNodeType.endsWith("Statement")) {
String label = astNodeType;
label = label.substring(0, label.lastIndexOf("S")).toLowerCase();
tokens += astNodeType + " " + label + " ";
} else if ("SuperConstructorInvocation".equals(astNodeType)) {
tokens += astNodeType + " super ";
} else if ("ConstructorInvocation".equals(astNodeType)) {
tokens += astNodeType + " this ";
} else if ("StringLiteral".equals(astNodeType)) {
tokens += astNodeType + " stringLiteral ";
} else if ("CharacterLiteral".equals(astNodeType)) {
tokens += astNodeType + " charLiteral ";
} else if ("ArrayInitializer".equals(astNodeType)) {
tokens += astNodeType + " arrayInitializer ";
} else if ("LambdaExpression".equals(astNodeType)) {
tokens += astNodeType + " lambda ";
} else if ("NumberLiteral".equals(astNodeType)) {
tokens += astNodeType + " numberLiteral ";
} else {
tokens += astNodeType + " " + simpleTree.getLabel() + " ";
}
} else {
if ("AssertStatement".equals(astNodeType) || "DoStatement".equals(astNodeType)
|| "ForStatement".equals(astNodeType) || "IfStatement".equals(astNodeType)
|| "ReturnStatement".equals(astNodeType) || "SwitchStatement".equals(astNodeType)
|| "SynchronizedStatement".equals(astNodeType) || "ThrowStatement".equals(astNodeType)
|| "TryStatement".equals(astNodeType) || "WhileStatement".equals(astNodeType)) {
String label = astNodeType;
label = label.substring(0, label.lastIndexOf("S")).toLowerCase();
tokens += astNodeType + " " + label + " ";
} else if ("EnhancedForStatement".equals(astNodeType)) {
tokens += astNodeType + " " + "for ";
} else if ("CatchClause".equals(astNodeType)) {
tokens += astNodeType + " " + "catch ";
} else if ("SwitchCase".equals(astNodeType)) {
tokens += astNodeType + " case ";
} else if ("SuperConstructorInvocation".equals(astNodeType)) {
tokens += astNodeType + " super ";
} else if ("ConstructorInvocation".equals(astNodeType)) {
tokens += astNodeType + " this ";
} else if ("FinallyBody".equals(astNodeType)) {
tokens += astNodeType + " finally ";
} else if ("LabeledStatement".equals(astNodeType)) {
tokens += "LabeledStatement " + simpleTree.getLabel();
} else if ("SuperMethodInvocation".equals(astNodeType)) {
tokens += astNodeType + " super ";
} else if ("MethodName".equals(astNodeType)) {
tokens += "MethodName " + simpleTree.getLabel() + " ";
}
if ("ArrayInitializer".equals(astNodeType)) {
tokens += astNodeType + " arrayInitializer ";
} else {
for (SimpleTree child : children) {
tokens += getTokensDeepFirst(child);
}
}
}
return tokens;
}
}
src/main/java/edu/lu/uni/serval/FixPatternParser/patch/CommitPatchHunkParser.java
-113
View File
@@ -1,113 +0,0 @@
package edu.lu.uni.serval.FixPatternParser.patch;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import edu.lu.uni.serval.FixPatternParser.Tokenizer;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.diffentry.DiffEntryHunk;
import edu.lu.uni.serval.diffentry.DiffEntryReader;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalActionSet;
import edu.lu.uni.serval.gumtree.regroup.HunkActionFilter;
import edu.lu.uni.serval.gumtree.regroup.HunkFixPattern;
import edu.lu.uni.serval.gumtree.regroup.SimpleTree;
import edu.lu.uni.serval.gumtree.regroup.SimplifyTree;
/**
* Parse fix patterns with GumTree.
*
* Multiple statements bugs.
*
* @author kui.liu
*
*/
public class CommitPatchHunkParser extends CommitPatchParser {
@Override
public void parseFixPatterns(File prevFile, File revFile, File diffEntryFile) {
// GumTree results
List<HierarchicalActionSet> actionSets = parseChangedSourceCodeWithGumTree(prevFile, revFile);
if (actionSets.size() > 0) {
// DiffEntry size: filter out big hunks.
List<DiffEntryHunk> diffentryHunks = new DiffEntryReader().readHunks(diffEntryFile);
//Filter out the modify actions, which are not in the DiffEntry hunks.
HunkActionFilter hunkFilter = new HunkActionFilter();
List<HunkFixPattern> allHunkFixPatterns = hunkFilter.filterActionsByDiffEntryHunk2(diffentryHunks, actionSets, revFile, prevFile);
for (HunkFixPattern hunkFixPattern : allHunkFixPatterns) {
// Range of buggy source code
int startLine = 0;
int endLine = 0;
// Range of fixing source code
int startLine2 = 0;
int endLine2 = 0;
/*
* Convert the ITree of buggy code to a simple tree.
* It will be used to compute the similarity.
*/
List<HierarchicalActionSet> hunkActionSets = hunkFixPattern.getHunkActionSets();
SimpleTree simpleTree = new SimpleTree();
simpleTree.setLabel("Block");
simpleTree.setNodeType("Block");
List<SimpleTree> children = new ArrayList<>();
String astEditScripts = "";
for (HierarchicalActionSet hunkActionSet : hunkActionSets) {
SimplifyTree abstractIdentifier = new SimplifyTree();
abstractIdentifier.abstractTree(hunkActionSet);
SimpleTree simpleT = hunkActionSet.getSimpleTree();
if (simpleT == null) { // Failed to get the simple tree for INS actions.
continue;
}
children.add(simpleT);
/**
* Select edit scripts for deep learning.
* Edit scripts will be used to mine common fix patterns.
*/
// 1. First level: AST node type.
astEditScripts += getASTEditScriptsBreadthFirst(hunkActionSet);
// 2. source code: raw tokens
// 3. abstract identifiers:
// 4. semi-source code:
if (startLine == 0) {
startLine = hunkActionSet.getBugStartLineNum();
endLine = hunkActionSet.getBugEndLineNum();
startLine2 = hunkActionSet.getFixStartLineNum();
endLine2 = hunkActionSet.getFixEndLineNum();
} else {
if (startLine > hunkActionSet.getBugStartLineNum()) startLine = hunkActionSet.getBugStartLineNum();
if (startLine2 > hunkActionSet.getFixStartLineNum()) startLine2 = hunkActionSet.getFixStartLineNum();
if (endLine < hunkActionSet.getBugEndLineNum()) endLine = hunkActionSet.getBugEndLineNum();
if (endLine2 < hunkActionSet.getFixEndLineNum()) endLine2 = hunkActionSet.getFixEndLineNum();
}
}
if (children.size() == 0) continue;
if (endLine - startLine >= Configuration.HUNK_SIZE - 2 || endLine2 - startLine2 >= Configuration.HUNK_SIZE - 2 ) continue;
simpleTree.setChildren(children);
simpleTree.setParent(null);
// Source Code of patches.
String patchSourceCode = getPatchSourceCode(hunkFixPattern.getHunk(), startLine, endLine, startLine2, endLine2);
if ("".equals(patchSourceCode)) continue;
this.patchesSourceCode += "PATCH###\n" + patchSourceCode + "\n";
int size = astEditScripts.split(" ").length;
this.sizes += size + "\n";
this.astEditScripts += astEditScripts + "\n";
// this.buggyTrees += Configuration.BUGGY_TREE_TOKEN + "\n" + simpleTree.toString() + "\n";
this.tokensOfSourceCode += Tokenizer.getTokensDeepFirst(simpleTree).trim() + "\n";
// this.actionSets += Configuration.BUGGY_TREE_TOKEN + "\n" + readActionSet(actionSet, "") + "\n";
// this.originalTree += Configuration.BUGGY_TREE_TOKEN + "\n" + actionSet.getOriginalTree().toString() + "\n";
}
}
}
}
src/main/java/edu/lu/uni/serval/FixPatternParser/patch/CommitPatchParser.java
-101
View File
@@ -1,101 +0,0 @@
package edu.lu.uni.serval.FixPatternParser.patch;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.StringReader;
import java.util.List;
import edu.lu.uni.serval.FixPatternParser.Parser;
import edu.lu.uni.serval.diffentry.DiffEntryHunk;
/**
* Parse fix patterns with GumTree.
*
* @author kui.liu
*
*/
public class CommitPatchParser extends Parser{
@Override
public void parseFixPatterns(File prevFile, File revFile, File diffEntryFile) {
}
protected DiffEntryHunk matchHunk(int startLine, int endLine, int startLine2, int endLine2, String actionStr, List<DiffEntryHunk> hunks) {
for (DiffEntryHunk hunk : hunks) {
int bugStartLine = hunk.getBugLineStartNum();
int bugRange = hunk.getBugRange();
int fixStartLine = hunk.getFixLineStartNum();
int fixRange = hunk.getFixRange();
if (actionStr.startsWith("INS")) {
if (fixStartLine + fixRange < startLine2) {
continue;
}
if (endLine2 < fixStartLine ) {
return null;
}
return hunk;
} else {
if (bugStartLine + bugRange < startLine) {
continue;
}
if (endLine < bugStartLine ) {
return null;
}
return hunk;
}
}
return null;
}
protected String getPatchSourceCode(DiffEntryHunk hunk, int startLineNum, int endLineNum, int startLineNum2, int endLineNum2) {
String sourceCode = hunk.getHunk();
int bugStartLine = hunk.getBugLineStartNum();
int fixStartLine = hunk.getFixLineStartNum();
String buggyStatements = "";
String fixedStatements = "";
BufferedReader reader = null;
try {
reader = new BufferedReader(new StringReader(sourceCode));
String line = null;
int bugLines = 0;
int fixLines = 0;
int contextLines = 0; // counter of non-buggy code line.
while ((line = reader.readLine()) != null) {
int bugLineIndex = bugLines + contextLines;
int fixLineIndex = fixLines + contextLines;
if (line.startsWith("-")) {
if (bugStartLine + bugLineIndex >= startLineNum && bugStartLine + bugLineIndex <= endLineNum) {
buggyStatements += line + "\n";
}
bugLines ++;
} else if (line.startsWith("+")) {
if (fixStartLine + fixLineIndex >= startLineNum2 && fixStartLine + fixLineIndex <= endLineNum2) {
fixedStatements += line + "\n";
}
fixLines ++;
} else {
contextLines ++;
}
if (bugStartLine + bugLineIndex > endLineNum && fixStartLine + fixLineIndex > endLineNum2) {
break;
}
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (reader != null) {
reader.close();
reader = null;
}
} catch (IOException e) {
e.printStackTrace();
}
}
return buggyStatements + fixedStatements;
}
}
src/main/java/edu/lu/uni/serval/FixPatternParser/patch/CommitPatchSingleStatementParser.java
-158
View File
@@ -1,158 +0,0 @@
package edu.lu.uni.serval.FixPatternParser.patch;
import java.io.File;
import java.util.List;
import org.eclipse.jdt.core.dom.CompilationUnit;
import com.github.gumtreediff.actions.model.Move;
import com.github.gumtreediff.actions.model.Update;
import com.github.gumtreediff.tree.ITree;
import edu.lu.uni.serval.FixPattern.utils.Checker;
import edu.lu.uni.serval.FixPatternParser.CUCreator;
import edu.lu.uni.serval.FixPatternParser.Tokenizer;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.diffentry.DiffEntryHunk;
import edu.lu.uni.serval.diffentry.DiffEntryReader;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalActionSet;
import edu.lu.uni.serval.gumtree.regroup.SimpleTree;
import edu.lu.uni.serval.gumtree.regroup.SimplifyTree;
/**
* Parse fix patterns with GumTree.
*
* Single Statement bugs.
*
* @author kui.liu
*
*/
public class CommitPatchSingleStatementParser extends CommitPatchParser {
@Override
public void parseFixPatterns(File prevFile, File revFile, File diffEntryFile) {
// GumTree results
List<HierarchicalActionSet> actionSets = parseChangedSourceCodeWithGumTree(prevFile, revFile);
if (actionSets.size() > 0) {
// DiffEntry Hunks: filter out big hunks.
List<DiffEntryHunk> diffentryHunks = new DiffEntryReader().readHunks(diffEntryFile);
CUCreator cuCreator = new CUCreator();
CompilationUnit prevUnit = cuCreator.createCompilationUnit(prevFile);
CompilationUnit revUnit = cuCreator.createCompilationUnit(revFile);
if (prevUnit == null || revUnit == null) {
return;
}
for (HierarchicalActionSet actionSet : actionSets) {
// position of buggy statements
int startPosition = 0;
int endPosition = 0;
// position of fixed statements
int startPosition2 = 0;
int endPosition2 = 0;
String actionStr = actionSet.getActionString();
String astNodeType = actionSet.getAstNodeType();
if (actionStr.startsWith("INS")) {
startPosition2 = actionSet.getStartPosition();
endPosition2 = startPosition2 + actionSet.getLength();
List<Move> firstAndLastMov = getFirstAndLastMoveAction(actionSet);
if (firstAndLastMov != null) {
startPosition = firstAndLastMov.get(0).getNode().getPos();
ITree lastTree = firstAndLastMov.get(1).getNode();
endPosition = lastTree.getPos() + lastTree.getLength();
} else { // Ignore the pure insert actions without any move actions.
continue;
}
} else if (actionStr.startsWith("UPD")) {
startPosition = actionSet.getStartPosition();
endPosition = startPosition + actionSet.getLength();
Update update = (Update) actionSet.getAction();
ITree newNode = update.getNewNode();
startPosition2 = newNode.getPos();
endPosition2 = startPosition2 + newNode.getLength();
if (Checker.containsBodyBlock(astNodeType)) {
List<ITree> children = update.getNode().getChildren();
endPosition = getEndPosition(children);
List<ITree> newChildren = newNode.getChildren();
endPosition2 = getEndPosition(newChildren);
if (endPosition == 0) {
endPosition = startPosition + actionSet.getLength();
}
if (endPosition2 == 0) {
endPosition2 = startPosition2 + newNode.getLength();
}
}
} else {// DEL actions and MOV actions: we don't need these actions, as for now.
continue;
}
if (startPosition == 0 || startPosition2 == 0) {
continue;
}
// Get line numbers.
int startLine = prevUnit.getLineNumber(startPosition);
int endLine = prevUnit.getLineNumber(endPosition);
int startLine2 = revUnit.getLineNumber(startPosition2);
int endLine2 = revUnit.getLineNumber(endPosition2);
if (endLine - startLine >= Configuration.HUNK_SIZE - 2 || endLine2 - startLine2 >= Configuration.HUNK_SIZE - 2)
continue;
// Filter out the modify actions, which are not in the DiffEntry hunks.
DiffEntryHunk hunk = matchHunk(startLine, endLine, startLine2, endLine2, actionStr, diffentryHunks);
if (hunk == null) {
continue;
}
/*
* Convert the ITree of buggy code to a simple tree. It will be
* used to compute the similarity.
*/
SimplifyTree abstractIdentifier = new SimplifyTree();
abstractIdentifier.abstractTree(actionSet);
SimpleTree simpleTree = actionSet.getSimpleTree();
if (simpleTree == null) { // Failed to get the simple tree for INS actions.
continue;
}
/**
* Select edit scripts for deep learning. Edit scripts will be
* used to mine common fix patterns.
*/
// 1. First level: AST node type.
String astEditScripts = getASTEditScriptsBreadthFirst(actionSet);
int size = astEditScripts.split(" ").length;
if (size < 2) {
// System.out.println(actionSet);
continue;
}
// Source Code of patches.
String patchSourceCode = getPatchSourceCode(hunk, startLine, endLine, startLine2, endLine2);
if ("".equals(patchSourceCode))
continue;
this.patchesSourceCode += Configuration.PATCH_SIGNAL + "\n" + patchSourceCode + "\n";
this.sizes += size + "\n";
this.astEditScripts += astEditScripts + "\n";
// // 2. source code: raw tokens
// String rawTokenEditScripts = getRawTokenEditScripts(actionSet);
// // 3. abstract identifiers:
// String abstractIdentifiersEditScripts = getAbstractIdentifiersEditScripts(actionSet);
// // 4. semi-source code:
// String semiSourceCodeEditScripts = getSemiSourceCodeEditScripts(actionSet);
// this.buggyTrees += Configuration.BUGGY_TREE_TOKEN + "\n" +
// simpleTree.toString() + "\n";
this.tokensOfSourceCode += Tokenizer.getTokensDeepFirst(simpleTree).trim() + "\n";
// this.actionSets += Configuration.BUGGY_TREE_TOKEN + "\n" +
// readActionSet(actionSet, "") + "\n";
// this.originalTree += Configuration.BUGGY_TREE_TOKEN + "\n" +
// actionSet.getOriginalTree().toString() + "\n";
}
actionSets.clear();
}
}
}
src/main/java/edu/lu/uni/serval/FixPatternParser/violations/FieldViolation.java
-121
View File
@@ -1,121 +0,0 @@
package edu.lu.uni.serval.FixPatternParser.violations;
public enum FieldViolation {
UWF_FIELD_NOT_INITIALIZED_IN_CONSTRUCTOR,
ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD,
LI_LAZY_INIT_STATIC,
URF_UNREAD_FIELD,
MS_SHOULD_BE_FINAL,
URF_UNREAD_PUBLIC_OR_PROTECTED_FIELD,
UWF_UNWRITTEN_FIELD,
MS_PKGPROTECT,
SS_SHOULD_BE_STATIC,
NP_UNWRITTEN_FIELD,
UWF_UNWRITTEN_PUBLIC_OR_PROTECTED_FIELD,
SE_BAD_FIELD,
UUF_UNUSED_FIELD,
SE_BAD_FIELD_STORE,
MS_MUTABLE_ARRAY,
NP_UNWRITTEN_PUBLIC_OR_PROTECTED_FIELD,
}
enum ClassViolation{
SIC_INNER_SHOULD_BE_STATIC_ANON, //inner class. ClassInstanceCreation by overriding some methods.
SIC_INNER_SHOULD_BE_STATIC,
RI_REDUNDANT_INTERFACES,
SE_COMPARATOR_SHOULD_BE_SERIALIZABLE,
SE_NO_SERIALVERSIONID,
CI_CONFUSED_INHERITANCE,
SE_INNER_CLASS,
NM_CLASS_NOT_EXCEPTION,
}
enum MethodViolation {
NM_METHOD_NAMING_CONVENTION,
UPM_UNCALLED_PRIVATE_METHOD,
DE_MIGHT_IGNORE,
NM_CONFUSING,
CN_IMPLEMENTS_CLONE_BUT_NOT_CLONEABLE,
UC_USELESS_VOID_METHOD,
}
enum StmtViolation {
DLS_DEAD_LOCAL_STORE, //Assignment statement. false positives may be high.
//This instruction assigns a value to a local variable, but the value is not read or used in any subsequent instruction.
DM_BOXED_PRIMITIVE_FOR_PARSING,
DM_CONVERT_CASE,
DM_DEFAULT_ENCODING,
IS2_INCONSISTENT_SYNC,
NP_GUARANTEED_DEREF,
SIO_SUPERFLUOUS_INSTANCEOF,
NP_BOOLEAN_RETURN_NULL,
SF_SWITCH_FALLTHROUGH,
DP_DO_INSIDE_DO_PRIVILEGED,
DMI_INVOKING_TOSTRING_ON_ARRAY,
NP_NONNULL_PARAM_VIOLATION,
CN_IDIOM_NO_SUPER_CALL,
UL_UNRELEASED_LOCK_EXCEPTION_PATH,
RC_REF_COMPARISON_BAD_PRACTICE,
NP_ALWAYS_NULL,
NP_NULL_PARAM_DEREF,
GC_UNRELATED_TYPES,
DLS_DEAD_LOCAL_STORE_OF_NULL,
RpC_REPEATED_CONDITIONAL_TEST,
ODR_OPEN_DATABASE_RESOURCE_EXCEPTION_PATH,
DB_DUPLICATE_SWITCH_CLAUSES,
RV_RETURN_VALUE_IGNORED,
ES_COMPARING_PARAMETER_STRING_WITH_EQ,
RCN_REDUNDANT_NULLCHECK_OF_NULL_VALUE,
ICAST_IDIV_CAST_TO_DOUBLE,
BC_VACUOUS_INSTANCEOF,
EC_UNRELATED_TYPES,
ICAST_INTEGER_MULTIPLY_CAST_TO_LONG,
DM_STRING_TOSTRING,
DMI_HARDCODED_ABSOLUTE_FILENAME,
MS_MUTABLE_COLLECTION_PKGPROTECT,
NP_NULL_ON_SOME_PATH_EXCEPTION,
SQL_NONCONSTANT_STRING_PASSED_TO_EXECUTE,
EQ_COMPARETO_USE_OBJECT_EQUALS,
BX_UNBOXING_IMMEDIATELY_REBOXED,
HE_EQUALS_USE_HASHCODE,
DM_EXIT,
FE_FLOATING_POINT_EQUALITY,
RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE,
BC_UNCONFIRMED_CAST,
REC_CATCH_EXCEPTION,
BC_UNCONFIRMED_CAST_OF_RETURN_VALUE,
VA_FORMAT_STRING_USES_NEWLINE,
RV_RETURN_VALUE_IGNORED_BAD_PRACTICE,
EI_EXPOSE_REP2,
WMI_WRONG_MAP_ITERATOR,
EI_EXPOSE_REP,
NP_LOAD_OF_KNOWN_NULL_VALUE,
DM_NUMBER_CTOR,
SBSC_USE_STRINGBUFFER_CONCATENATION,
OS_OPEN_STREAM_EXCEPTION_PATH,
NP_NONNULL_RETURN_VIOLATION,
SF_SWITCH_NO_DEFAULT,
RV_RETURN_VALUE_IGNORED_NO_SIDE_EFFECT,
ODR_OPEN_DATABASE_RESOURCE,
PZLA_PREFER_ZERO_LENGTH_ARRAYS,
NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE,
UCF_USELESS_CONTROL_FLOW,
UC_USELESS_CONDITION,
NP_NULL_ON_SOME_PATH,
DM_FP_NUMBER_CTOR,
SQL_PREPARED_STATEMENT_GENERATED_FROM_NONCONSTANT_STRING,
OBL_UNSATISFIED_OBLIGATION_EXCEPTION_EDGE,
ES_COMPARING_STRINGS_WITH_EQ,
OS_OPEN_STREAM,
RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE,
NP_PARAMETER_MUST_BE_NONNULL_BUT_MARKED_AS_NULLABLE,
UC_USELESS_OBJECT,
OBL_UNSATISFIED_OBLIGATION,
}
src/main/java/edu/lu/uni/serval/FixPatternParser/violations/FixedViolationHunkParser.java
+46 -472
View File
@@ -1,25 +1,11 @@
package edu.lu.uni.serval.FixPatternParser.violations;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalActionSet;
import java.io.*;
import java.util.List;
import com.github.gumtreediff.tree.ITree;
import edu.lu.uni.serval.FixPatternParser.Tokenizer;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.diffentry.DiffEntryHunk;
import edu.lu.uni.serval.diffentry.DiffEntryReader;
import edu.lu.uni.serval.gumtree.GumTreeGenerator;
import edu.lu.uni.serval.gumtree.GumTreeGenerator.GumTreeType;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalActionSet;
import edu.lu.uni.serval.gumtree.regroup.HunkActionFilter;
import edu.lu.uni.serval.gumtree.regroup.SimpleTree;
import edu.lu.uni.serval.gumtree.regroup.SimplifyTree;
import edu.lu.uni.serval.violation.code.parser.ViolationSourceCodeTree;
/**
* Parse fix violations with GumTree in terms of multiple statements.
@@ -29,8 +15,6 @@ import edu.lu.uni.serval.violation.code.parser.ViolationSourceCodeTree;
*/
public class FixedViolationHunkParser extends FixedViolationParser {
private static final int THRESHOLD_LINE = 3;
public String testingInfo = "";
public int nullMappingGumTreeResult = 0;
@@ -39,475 +23,65 @@ public class FixedViolationHunkParser extends FixedViolationParser {
public int nullSourceCode = 0;
public int nullMatchedDiffEntry = 0;
public int testInfos = 0;
public FixedViolationHunkParser() {
}
public FixedViolationHunkParser(File positionFile) {
setPositionFile(positionFile);
}
public String unfixedViolations = "";
@Override
public void parseFixPatterns(File prevFile, File revFile, File diffentryFile) {
List<Violation> allViolations = readViolations(prevFile, revFile);
// if (allViolations.size() >= 0) return;
// GumTree results
List<HierarchicalActionSet> actionSets = parseChangedSourceCodeWithGumTree2(prevFile, revFile);
if (this.resultType != 0) {
String type = "";
if (this.resultType == 1) {
type = "#NullGumTreeResult:";
} else if (this.resultType == 2) {
type = "#NoSourceCodeChange:";
} else if (this.resultType == 3) {
type = "#NoStatementChange:";
}
for (Violation v : allViolations) {
this.unfixedViolations += type + revFile.getName() + ":" + v.getStartLineNum() + ":" + v.getEndLineNum() + ":" + v.getViolationType() + "\n";
}
} else {
List<DiffEntryHunk> diffentryHunks = new DiffEntryReader().readHunks2(diffentryFile);
// Identify DiffEntry hunks by positions of violations.
List<Violation> violations = identifyFixRangeHeuristically(allViolations, diffentryHunks, revFile);
//Filter out the modify actions, which are not in the DiffEntry hunks.
HunkActionFilter hunkFilter = new HunkActionFilter();
List<Violation> selectedViolations = hunkFilter.filterActionsByModifiedRange2(violations, actionSets, revFile, prevFile);
this.unfixedViolations += hunkFilter.unfixedViolations;
this.nullMappingGumTreeResult += violations.size() - selectedViolations.size();
for (Violation violation : selectedViolations) {
List<HierarchicalActionSet> hunkActionSets = violation.getActionSets();
// multiple UPD, and some UPD contain other UPD.
removeOverlapperdUPD(hunkActionSets);
// Range of buggy source code
int bugStartLine = 0;
int bugEndLine = 0;
// Range of fixing source code
int fixStartLine = 0;
int fixEndLine = 0;
int bugEndPosition = 0;
int fixEndPosition = 0;
for (HierarchicalActionSet hunkActionSet : hunkActionSets) {
int actionBugStart = hunkActionSet.getBugStartLineNum();
int actionBugEnd = hunkActionSet.getBugEndLineNum();
int actionFixStart = hunkActionSet.getFixStartLineNum();
int actionFixEnd = hunkActionSet.getFixEndLineNum();
if (bugStartLine == 0) {
bugStartLine = actionBugStart;
} else if (actionBugStart != 0 && actionBugStart < bugStartLine) {
bugStartLine = actionBugStart;
}
if (fixStartLine == 0) {
fixStartLine = actionFixStart;
} else if (actionFixStart != 0 && actionFixStart < fixStartLine) {
fixStartLine = actionFixStart;
}
if (bugEndLine < actionBugEnd) {
bugEndLine = actionBugEnd;
bugEndPosition = hunkActionSet.getBugEndPosition();
}
if (fixEndLine < actionFixEnd) {
fixEndLine = actionFixEnd;
fixEndPosition = hunkActionSet.getFixEndPosition();
}
}
if (fixStartLine == 0 && bugStartLine == 0) {
this.unfixedViolations += "#WRONG: " + revFile.getName() + ":" + violation.getStartLineNum() + ":" + violation.getEndLineNum() + ":" + violation.getViolationType() + "\n";
this.nullMappingGumTreeResult ++;
continue;
}
if (fixStartLine == 0 && bugStartLine != 0) {// pure delete actions.
// this.pureDeletions ++;
// this.unfixedViolations += "#PureDeletion:" + revFile.getName() + ":" + violation.getStartLineNum() + ":" + violation.getEndLineNum() + ":" + violation.getViolationType() + "\n";
// get the exact buggy code by violation's position. TODO later
}
// if (children.size() == 0) continue;
boolean isPureInsert = false;
if (bugStartLine == 0 && violation.getBugStartLineNum() > 0) {
bugStartLine = violation.getStartLineNum();
bugEndLine = violation.getEndLineNum();
isPureInsert = true;
// continue;
}
if ((bugEndLine - bugStartLine > Configuration.HUNK_SIZE && !isPureInsert) || fixEndLine - fixStartLine > Configuration.HUNK_SIZE) {
// this.largeHunk ++;
// this.unfixedViolations += "#LargeHunk:" + revFile.getName() + ":" + violation.getStartLineNum()
// + ":" + violation.getEndLineNum() + ":" + violation.getViolationType() + "\n";
// continue;
}
// Obtain Source Code of patches from DiffEntries.
String patchSourceCode = this.readPatchSourceCode(violation, bugStartLine, bugEndLine, fixStartLine, fixEndLine);
if ("".equals(patchSourceCode)) {
patchSourceCode = getPatchSourceCode(prevFile, revFile, bugStartLine, bugEndLine, fixStartLine, fixEndLine, isPureInsert);
if ("".equals(patchSourceCode)) {
this.nullSourceCode ++;
this.unfixedViolations += "#NullSourceCode:" + revFile.getName() + ":" + violation.getStartLineNum()
+ ":" + violation.getEndLineNum() + ":" + violation.getViolationType() + "\n";
continue;
}
}
/**
* Select edit scripts for deep learning.
* Edit scripts will be used to mine common fix patterns.
*/
// 1. First level: AST node type.
String astEditScripts = getASTEditScriptsDeepFirst(hunkActionSets, bugEndPosition, fixEndPosition);
if (astEditScripts.contains("\n") || astEditScripts.split(" ").length % 3 != 0) {
System.err.println("===+++===: " + revFile.getName() + ":" +violation.getStartLineNum() + ":" +
violation.getEndLineNum() + ":" + violation.getViolationType());
}
// 2. source code: raw tokens
// 3. abstract identifiers:
// 4. semi-source code:
String[] editScriptTokens = astEditScripts.split(" ");
int size = editScriptTokens.length;
if (size == 1) {
// if ("MOVMethodDeclaration".equals(astEditScripts)) {
// this.nullMappingGumTreeResult ++;
// this.unfixedViolations += "#NullMatchedGumTreeResult:" + revFile.getName() + ":" +violation.getStartLineNum() + ":" +
// violation.getEndLineNum() + ":" + violation.getViolationType() + "\n";
// continue;
public void parseFixPatterns(File prevFile, File revFile, File diffentryFile,String project) {
List<HierarchicalActionSet> actionSets = parseChangedSourceCodeWithGumTree2(prevFile, revFile);
if (actionSets.size() != 0) {
// boolean isUpdate =
// actionSets.stream().allMatch(p -> p.getAction() instanceof Update);
// boolean isInsert =
// actionSets.stream().allMatch(p -> p.getAction() instanceof Insert);
// boolean isDelete =
// actionSets.stream().allMatch(p -> p.getAction() instanceof Delete);
// boolean isMove =
// actionSets.stream().allMatch(p -> p.getAction() instanceof Move);
int hunkSet = 0;
// if (isUpdate || isInsert || isDelete || isMove) {
for (HierarchicalActionSet actionSet : actionSets) {
String folder = "/ALL/";
// if (isUpdate) {
// folder = "/UPD/";
// } else if (isDelete) {
// folder = "/DEL/";
// } else if (isInsert) {
// folder = "/INS/";
// } else if (isMove) {
// folder = "/MOV/";
// }
this.nullMappingGumTreeResult ++;
this.unfixedViolations += "#NullMatchedGumTreeResult1:" + revFile.getName() + ":" +violation.getStartLineNum() + ":" +
violation.getEndLineNum() + ":" + violation.getViolationType() + "\n";
continue;
}
String alarmType = violation.getViolationType();
String patchPosition = "\n" + revFile.getName() + "\nPosition: " + violation.getStartLineNum() + " --> " + violation.getEndLineNum() + "\n@@ -" + bugStartLine + ", " + bugEndLine + " +" + fixStartLine + ", " + fixEndLine + "@@\n";
// String patchPosition = "\n" + "Position: " + violation.getStartLineNum() + " --> " + violation.getEndLineNum() + "\n@@ -" + bugStartLine + ", " + bugEndLine + " +" + fixStartLine + ", " + fixEndLine + "@@\n";
String info = Configuration.PATCH_SIGNAL + "\nAlarm Type :" + alarmType + "\n" + patchPosition + patchSourceCode + "\nAST Diff###:\n" + getAstEditScripts(hunkActionSets, bugEndPosition, fixEndPosition) + "\n";
// String patchPosition = "\n" + "Position: " + violation.getStartLineNum() + " --> " + violation.getEndLineNum() + "\n@@ -" + bugStartLine + ", " + bugEndLine + " +" + fixStartLine + ", " + fixEndLine + "@@\n";
// String info = Configuration.PATCH_SIGNAL + "\nAlarm Type :" + alarmType + "\n" + patchSourceCode + "\n\n";// + patchPosition + revFile.getName() + "\n";
// if (noUpdate(editScriptTokens)) {
//
// if (!"SE_NO_SERIALVERSIONID".equals(alarmType)) {
// if (containsFiledDeclaration(hunkActionSets)) {
//// this.nullMappingGumTreeResult ++;
// this.testingInfo += info + revFile.getName() + "\n";
// this.testingInfo += "#TestingInfo: " + revFile.getName() + ":" + violation.getStartLineNum()
// + ":" + violation.getEndLineNum() + ":" + violation.getViolationType();
// this.unfixedViolations += "#TestingInfo:" + revFile.getName() + ":" + violation.getStartLineNum()
// + ":" + violation.getEndLineNum() + ":" + violation.getViolationType() + "\n";
// this.testInfos ++;
// continue;
// }
// }
//
// if ("UL_UNRELEASED_LOCK".equals(alarmType) ||
// "SF_SWITCH_NO_DEFAULT".equals(alarmType) ||
// "SF_SWITCH_FALLTHROUGH".equals(alarmType) ||
// "SE_NO_SERIALVERSIONID".equals(alarmType) ||
// "REC_CATCH_EXCEPTION".equals(alarmType) ||
// "OS_OPEN_STREAM".equals(alarmType) ||
// "NP_ALWAYS_NULL".equals(alarmType) ||
// "IS2_INCONSISTENT_SYNC".equals(alarmType) ||
// "EI_EXPOSE_REP".equals(alarmType) ) {
//
// } else if (containSpecialStmt(hunkActionSets, bugEndPosition, fixEndPosition)) {
//
// } else {
//// this.nullMappingGumTreeResult ++;
// this.testingInfo += info + revFile.getName() + "\n";
// this.testingInfo += "#TestingInfo: " + revFile.getName() + ":" + violation.getStartLineNum()
// + ":" + violation.getEndLineNum() + ":" + violation.getViolationType();
// this.unfixedViolations += "#TestingInfo:" + revFile.getName() + ":" + violation.getStartLineNum()
// + ":" + violation.getEndLineNum() + ":" + violation.getViolationType() + "\n";
// this.testInfos ++;
// continue;
// }
// }
this.patchesSourceCode += info;
this.sizes += size + "\n";
this.astEditScripts += astEditScripts + "\n";
this.violationTypes += alarmType + "\n";
// this.buggyTrees += Configuration.BUGGY_TREE_TOKEN + "\n" + simpleTree.toString() + "\n";
SimpleTree simpleTree = getBuggyCodeTree(violation, bugEndPosition, prevFile, bugStartLine, bugEndLine);
String tokens = Tokenizer.getTokensDeepFirst(simpleTree).trim();
if ("Block Block".equals(tokens)) {
tokens = getContextTokens(patchSourceCode);
}
this.tokensOfSourceCode += tokens + "\n";
// this.actionSets += Configuration.BUGGY_TREE_TOKEN + "\n" + readActionSet(actionSet, "") + "\n";
// this.originalTree += Configuration.BUGGY_TREE_TOKEN + "\n" + actionSet.getOriginalTree().toString() + "\n";
}
}
}
private SimpleTree getBuggyCodeTree(Violation violation, int bugEndPosition, File prevFile, int bugStartLine, int bugEndLine) {
SimpleTree simpleTree = null;
String type = violation.getViolationType();
if ("EQ_DOESNT_OVERRIDE_EQUALS".equals(type)|| "HE_EQUALS_USE_HASHCODE".equals(type) || "HE_INHERITS_EQUALS_USE_HASHCODE".equals(type)||
"SE_NO_SUITABLE_CONSTRUCTOR".equals(type) || "RI_REDUNDANT_INTERFACES".equals(type)
||"CN_IDIOM".equals(type)||"SE_NO_SERIALVERSIONID".equals(type) || "SE_COMPARATOR_SHOULD_BE_SERIALIZABLE".equals(type)) {
// Class name level, tokens
// classStartP <= vS <= vE <= classEndP
ViolationSourceCodeTree parser = new ViolationSourceCodeTree(prevFile, violation.getStartLineNum(), violation.getEndLineNum());
ITree classNameTree = parser.getClassNameTokens();
simpleTree = new SimplifyTree().canonicalizeSourceCodeTree(classNameTree, null);
} else {
simpleTree = new SimpleTree();
simpleTree.setLabel("Block");
simpleTree.setNodeType("Block");
List<SimpleTree> children = new ArrayList<>();
int vStartLine = violation.getStartLineNum();
int vEndLine = violation.getEndLineNum();
if (bugStartLine < vStartLine && vEndLine < bugEndLine) {
ViolationSourceCodeTree parser = new ViolationSourceCodeTree(prevFile, violation.getStartLineNum(), violation.getEndLineNum());
parser.extract();
List<ITree> matchedTrees = parser.getViolationSourceCodeTrees();
if (matchedTrees.size() > 0) {
for (ITree matchedTree : matchedTrees) {
SimpleTree simpleT = new SimplifyTree().canonicalizeSourceCodeTree(matchedTree, simpleTree);
children.add(simpleT);
}
}
}
if (children.size() == 0) {
List<HierarchicalActionSet> hunkActionSets = violation.getActionSets();
/*
* Convert the ITree of buggy code to a simple tree.
* It will be used to compute the similarity.
*/
for (HierarchicalActionSet hunkActionSet : hunkActionSets) {
// TODO simplify buggy tree with buggy code.
/**
* Select edit scripts for deep learning.
* Edit scripts will be used to mine common fix patterns.
*/
// // 1. First level: AST node type.
// // 2. source code: raw tokens
// // 3. abstract identifiers:
// // 4. semi-source code:
SimplifyTree abstractIdentifier = new SimplifyTree();
abstractIdentifier.abstractTree(hunkActionSet, bugEndPosition);
SimpleTree simpleT = hunkActionSet.getSimpleTree();
if (simpleT == null) { // Failed to get the simple tree for INS actions.
continue;
}
children.add(simpleT);
}
}
simpleTree.setChildren(children);
simpleTree.setParent(null);
}
return simpleTree;
}
FileOutputStream f = null;
private void removeOverlapperdUPD(List<HierarchicalActionSet> actionSets) {
if (actionSets.size() == 1) {
return;
}
List<HierarchicalActionSet> updates = new ArrayList<>();
for (HierarchicalActionSet actionSet : actionSets) {
if (actionSet.getActionString().startsWith("UPD")) {
updates.add(actionSet);
}
}
List<HierarchicalActionSet> overlappedUpdates = new ArrayList<>();
if (updates.size() > 1) {
for (HierarchicalActionSet update : updates) {
int startLine = update.getBugStartLineNum();
int endLine = update.getBugEndLineNum();
int endPosition = update.getBugEndPosition();
for (HierarchicalActionSet update2 : updates) {
if (update.equals(update2)) continue;
int startLine2 = update2.getBugStartLineNum();
int endLine2 = update2.getBugEndLineNum();
int endPosition2 = update2.getBugEndPosition();
if (startLine <= startLine2
&& endLine2 <= endLine && endPosition2 < endPosition) {
if (!overlappedUpdates.contains(update)) {
overlappedUpdates.add(update);
}
break;
}
}
}
}
actionSets.removeAll(overlappedUpdates);
}
private List<Violation> identifyFixRangeHeuristically(List<Violation> violations, List<DiffEntryHunk> diffentryHunks, File revFile) {
List<Violation> selectedViolations = new ArrayList<>();
for (Violation violation : violations) {
int violationStartLineNum = violation.getBugStartLineNum();
if (violationStartLineNum > 0) {
int violationEndLineNum = violation.getEndLineNum();
for (int index = 0, hunkListSize = diffentryHunks.size(); index < hunkListSize; index ++) {
DiffEntryHunk hunk = diffentryHunks.get(index);
int bugStartLine = hunk.getBugLineStartNum();
int bugRange = hunk.getBugRange();
if (violationStartLineNum > bugStartLine + bugRange - 1) continue;
if (violationEndLineNum < bugStartLine) break;
int fixStartLine = hunk.getFixLineStartNum();
String diffentry = hunk.getHunk();
BufferedReader reader = new BufferedReader(new StringReader(diffentry));
String line = null;
try {
int currentBugLine = bugStartLine - 1;
int currentFixLine = fixStartLine - 1;
int fixedLines = 0;
int bugS = 0;
int fixS = 0;
int fixE = 0;
int bugFixStartLine = 0; // the heuristic inferred fix start line of the corresponding bug
int bugFixEndLine = 0; // the heuristic inferred fix end line of the corresponding bug
boolean isFixRange= false;
while ((line = reader.readLine()) != null) {
if (line.startsWith("+")) {
currentFixLine ++;
fixedLines++;
if ((violationStartLineNum <= currentBugLine && currentBugLine <= violationEndLineNum) ||
(violationStartLineNum <= bugS && bugS <= violationEndLineNum)) {
if (fixS == 0) fixS = currentFixLine;
if (bugFixStartLine == -1) bugFixStartLine = currentFixLine;
if (isFixRange) bugFixEndLine = currentFixLine;
fixE = currentFixLine;
} else {
fixedLines = 0;
bugS = 0;
}
} else if (line.startsWith("-")) {
currentBugLine ++;
if (bugS == 0 || fixS != 0) bugS = currentBugLine;// currentBugLine may be larger than the violation end line.
// INS combined with DEL, UPD or MOV.
// Infer the fix range for a violation heuristically.
if (currentBugLine >= violation.getStartLineNum() && currentBugLine <= violation.getEndLineNum()) {
bugFixStartLine = -1;
}
if (currentBugLine <= violation.getEndLineNum() && bugFixStartLine != 0) {
isFixRange = true;
}
if (bugFixEndLine > 0) isFixRange = false;
if (bugFixStartLine <= 0) fixS = 0;
} else {
currentBugLine ++;
currentFixLine ++;
// pure INS
// Infer the fix range for a violation heuristically.
if (currentBugLine == violation.getStartLineNum()) {
if (bugFixStartLine <= 0) {
if (fixedLines != 0) {
if (fixedLines > THRESHOLD_LINE) {
fixedLines = THRESHOLD_LINE;// insert 3 lines.
}
bugFixStartLine = currentFixLine - fixedLines;
if (bugFixStartLine < violationStartLineNum) {
bugFixStartLine = violationStartLineNum;
}
}
else bugFixStartLine = currentFixLine;
}
}
if (currentBugLine == violation.getEndLineNum()) {
if (bugFixStartLine != 0 && bugFixEndLine == 0) {
bugFixEndLine = currentFixLine + THRESHOLD_LINE;
if (bugFixEndLine > violationEndLineNum) {
bugFixEndLine = violationEndLineNum;
}
}
}
if (bugFixEndLine > 0) isFixRange = false;
}
if ((violationStartLineNum <= currentBugLine && currentBugLine <= violationEndLineNum)) {
if (!violation.getHunks().contains(hunk))
violation.getHunks().add(hunk);
}
}
if (violation.getFixStartLineNum() == 0) {
violation.setFixStartLineNum(fixS);
}
violation.setFixEndLineNum(fixE);
if (violation.getBugFixStartLineNum() == 0 && bugFixStartLine > 0) {
violation.setBugFixStartLineNum(bugFixStartLine);
}
if (bugFixEndLine > 0) {
violation.setBugFixEndLineNum(bugFixEndLine);
}
// String pj = diffentryFile.getParent().split("Defects4J")[1];
String datasetName = project;
String[] split1 = diffentryFile.getParent().split(datasetName);
String root = split1[0];
String pj = split1[1].split("/")[1];
String hunkTreeFileName = root + "EnhancedASTDiff" + datasetName + "/" + pj + folder + diffentryFile.getName() + "_" + String.valueOf(hunkSet);
f = new FileOutputStream(new File(hunkTreeFileName));
ObjectOutputStream o = new ObjectOutputStream(f);
o.writeObject(actionSet);
o.close();
f.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
hunkSet++;
}
if (violation.getHunks().size() == 0) {
// This fixed violation cannot be matched with a DiffEntry, it is difficult to identify related source code change for it.
this.nullMatchedDiffEntry ++;
this.unfixedViolations += "#NullDiffEntry: " + revFile.getName() + " : " +violation.getStartLineNum() + " : "
+ violation.getEndLineNum() + " : " + violation.getViolationType() + "\n";
} else {
selectedViolations.add(violation);
}
} else {
selectedViolations.add(violation);
}
// }
}
return selectedViolations;
}
private String getContextTokens(String patchSourceCode) {
BufferedReader reader = new BufferedReader(new StringReader(patchSourceCode));
String line = null;
String context = "";
try {
while ((line = reader.readLine()) != null) {
if (line.startsWith("+") || line.startsWith("-")) {
continue;
}
context += line + "\n";
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if ("".equals(context)) {
return "Block Block";
} else {
ITree tree = new GumTreeGenerator().generateITreeForCodeBlock(context, GumTreeType.EXP_JDT);
if (tree == null) {
return "Block Block";
}
SimpleTree simpleTree = new SimplifyTree().canonicalizeSourceCodeTree(tree, null);
String tokens = Tokenizer.getTokensDeepFirst(simpleTree).trim();
return tokens;
}
}
}
src/main/java/edu/lu/uni/serval/FixPatternParser/violations/FixedViolationParser.java
+17 -191
View File
@@ -1,31 +1,24 @@
package edu.lu.uni.serval.FixPatternParser.violations;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.List;
import com.github.gumtreediff.actions.model.Action;
import edu.lu.uni.serval.FixPatternParser.Parser;
import edu.lu.uni.serval.diffentry.DiffEntryHunk;
import edu.lu.uni.serval.gumtree.GumTreeComparer;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalActionSet;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalRegrouper;
import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.utils.ListSorter;
import edu.lu.uni.serval.violation.code.parser.ViolationSourceCodeTree;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
/**
* Parse fix patterns with GumTree.
*
*
* @author kui.liu
*
*/
public class FixedViolationParser extends Parser {
/*
* ResultType:
* 0: normal GumTree results.
@@ -35,22 +28,16 @@ public class FixedViolationParser extends Parser {
* 4: useless violations
*/
public int resultType = 0;
private File positionFile = null;
protected String violationTypes = "";
// protected List<Violation> uselessViolations;
public void setPositionFile(File positionFile) {
this.positionFile = positionFile;
}
@Override
public void parseFixPatterns(File prevFile, File revFile, File diffentryFile) {
public void parseFixPatterns(File prevFile, File revFile, File diffentryFile,String project) {
}
/**
* Regroup GumTree results without remove the modification of variable names.
*
*
* @param prevFile
* @param revFile
* @return
@@ -68,16 +55,7 @@ public class FixedViolationParser extends Parser {
} else {
// Regroup GumTre results.
List<HierarchicalActionSet> allActionSets = new HierarchicalRegrouper().regroupGumTreeResults(gumTreeResults);
// for (HierarchicalActionSet actionSet : allActionSets) {
// String astNodeType = actionSet.getAstNodeType();
// if (astNodeType.endsWith("Statement") || "FieldDeclaration".equals(astNodeType)) {
// actionSets.add(actionSet);
// }
// }
// Filter out modified actions of changing method names, method parameters, variable names and field names in declaration part.
// variable effects range, sub-actions are these kinds of modification?
// actionSets.addAll(new ActionFilter().filterOutUselessActions(allActionSets));
ListSorter<HierarchicalActionSet> sorter = new ListSorter<>(allActionSets);
actionSets = sorter.sortAscending();
@@ -90,167 +68,15 @@ public class FixedViolationParser extends Parser {
}
}
protected List<Violation> readViolations(File prevFile, File revFile) {
String fileName = revFile.getName();
List<Violation> violations = new ArrayList<>();
String fileContent = FileHelper.readFile(positionFile);
BufferedReader reader = null;
reader = new BufferedReader(new StringReader(fileContent));
String line = null;
try {
while ((line = reader.readLine()) != null) {
String[] positionStr = line.split(":");
int startLine = Integer.parseInt(positionStr[1]);
int endLine = Integer.parseInt(positionStr[2]);
String violationType = positionStr[0];
Violation violation = new Violation(startLine, endLine, violationType);
violation.setFileName(fileName);
if (startLine == -1) {
violation.setBugStartLineNum(0);
continue;
}
/*
* Get the parent range of a violation.
* Read DiffEntries with this range to get the start line and end line of a violation.
*/
ViolationSourceCodeTree alarmTree = new ViolationSourceCodeTree(prevFile, startLine, endLine);
alarmTree.locateParentNode(violationType);
int violationStartLine = alarmTree.getViolationFinalStartLine();
violation.setBugStartLineNum(violationStartLine);
if (violationStartLine > 0) {// 0: no source code, -1: range is too large, which contains several inner classes, methods or fields.
violation.setBugEndLineNum(alarmTree.getViolationFinalEndLine());
// if (violationType.equals("SE_NO_SERIALVERSIONID")){
// FileHelper.outputToFile("OUTPUT/list1.txt", line + ":" + revFile.getName() + "\n", true);
// }
} else {
// if (!violationType.equals("SE_NO_SERIALVERSIONID")) {
// FileHelper.outputToFile("OUTPUT/list.txt", line + ":" + revFile.getName() + "\n", true);
// }
}
violations.add(violation);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return violations;
}
/**
* Read patch source code from buggy and fixed files.
* @param prevFile
* @param revFile
* @param bugStartLineNum
* @param bugEndLineNum
* @param fixStartLineNum
* @param fixEndLineNum
* @param isInsert
* @return
*/
protected String getPatchSourceCode(File prevFile, File revFile, int bugStartLineNum, int bugEndLineNum, int fixStartLineNum, int fixEndLineNum, boolean isInsert) {
String buggyStatements = "";
if (isInsert) {
buggyStatements = readSourceCode(prevFile, bugStartLineNum, bugEndLineNum, "");
} else {
buggyStatements = readSourceCode(prevFile, bugStartLineNum, bugEndLineNum, "-");
}
String fixedStatements = readSourceCode(revFile, fixStartLineNum, fixEndLineNum, "+");
return buggyStatements + fixedStatements;
}
private String readSourceCode(File file, int startLineNum, int endLineNum, String type) {
String sourceCode = "";
String fileContent = FileHelper.readFile(file);
BufferedReader reader = null;
try {
reader = new BufferedReader(new StringReader(fileContent));
String line = null;
int lineIndex = 0;
while ((line = reader.readLine()) != null) {
lineIndex ++;
if (lineIndex >= startLineNum && lineIndex <= endLineNum) {
sourceCode += type + line + "\n";
}
if (lineIndex == endLineNum) break;
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sourceCode;
}
/**
* Read patch source code from diffentries.
* @param violation
* @param bugStartLine
* @param bugEndLine
* @param fixStartLine
* @param fixEndLine
* @return
*/
protected String readPatchSourceCode(Violation violation, int bugStartLine, int bugEndLine, int fixStartLine, int fixEndLine) {
String patch = "";
List<DiffEntryHunk> diffentries = violation.getHunks();
for (DiffEntryHunk diffentry : diffentries) {
int currentBugLine = diffentry.getBugLineStartNum() - 1;
int currentFixLine = diffentry.getFixLineStartNum() - 1;
String sourceCode = diffentry.getHunk();
BufferedReader reader = new BufferedReader(new StringReader(sourceCode));
String line = null;
try {
while ((line = reader.readLine()) != null) {
if (line.startsWith("+")) {
currentFixLine ++;
if (fixStartLine <= currentFixLine && currentFixLine <= fixEndLine) {
patch += line + "\n";
}
} else if (line.startsWith("-")) {
currentBugLine ++;
if (bugStartLine <= currentBugLine && currentBugLine <= bugEndLine) {
patch += line + "\n";
}
} else {
currentFixLine ++;
currentBugLine ++;
if ((bugStartLine <= currentBugLine && currentBugLine <= bugEndLine) ||
(fixStartLine <= currentFixLine && currentFixLine <= fixEndLine)) {
patch += line + "\n";
}
}
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return patch;
}
public String getAlarmTypes() {
return violationTypes;
}
// @Override
// public void parseFixPatterns(File prevFile, File revFile, File diffEntryFile) {
//
// }
// public void setUselessViolations(List<Violation> uselessViolations) {
// this.uselessViolations = uselessViolations;
// }
src/main/java/edu/lu/uni/serval/FixPatternParser/violations/FixedViolationSingleStatementParser.java
-157
View File
@@ -1,157 +0,0 @@
package edu.lu.uni.serval.FixPatternParser.violations;
import java.io.File;
import java.util.List;
import org.eclipse.jdt.core.dom.CompilationUnit;
import com.github.gumtreediff.actions.model.Move;
import com.github.gumtreediff.actions.model.Update;
import com.github.gumtreediff.tree.ITree;
import edu.lu.uni.serval.FixPattern.utils.Checker;
import edu.lu.uni.serval.FixPatternParser.CUCreator;
import edu.lu.uni.serval.FixPatternParser.Tokenizer;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalActionSet;
import edu.lu.uni.serval.gumtree.regroup.SimpleTree;
import edu.lu.uni.serval.gumtree.regroup.SimplifyTree;
/**
* Parse fixed violations with GumTree in terms of single statement.
*
* @author kui.liu
*
*/
public class FixedViolationSingleStatementParser extends FixedViolationParser {
@Override
public void parseFixPatterns(File prevFile, File revFile, File diffentryFile) {
// GumTree results
List<HierarchicalActionSet> actionSets = parseChangedSourceCodeWithGumTree(prevFile, revFile);
if (actionSets.size() > 0) {
CUCreator cuCreator = new CUCreator();
CompilationUnit prevUnit = cuCreator.createCompilationUnit(prevFile);
CompilationUnit revUnit = cuCreator.createCompilationUnit(revFile);
if (prevUnit == null || revUnit == null) {
return;
}
// Read the positions of checked violations
List<Violation> violations = readViolations(prevFile, revFile);
for (HierarchicalActionSet actionSet : actionSets) {
// position of buggy statements
int startPosition = 0;
int endPosition = 0;
// position of fixed statements
int startPosition2 = 0;
int endPosition2 = 0;
String actionStr = actionSet.getActionString();
String astNodeType = actionSet.getAstNodeType();
if (actionStr.startsWith("INS")) {
startPosition2 = actionSet.getStartPosition();
endPosition2 = startPosition2 + actionSet.getLength();
List<Move> firstAndLastMov = getFirstAndLastMoveAction(actionSet);
if (firstAndLastMov != null) {
startPosition = firstAndLastMov.get(0).getNode().getPos();
ITree lastTree = firstAndLastMov.get(1).getNode();
endPosition = lastTree.getPos() + lastTree.getLength();
} else { // Ignore the pure insert actions without any move actions.
continue;
}
} else if (actionStr.startsWith("UPD")) {
startPosition = actionSet.getStartPosition();
endPosition = startPosition + actionSet.getLength();
Update update = (Update) actionSet.getAction();
ITree newNode = update.getNewNode();
startPosition2 = newNode.getPos();
endPosition2 = startPosition2 + newNode.getLength();
if (Checker.containsBodyBlock(astNodeType)) {
List<ITree> children = update.getNode().getChildren();
endPosition = getEndPosition(children);
List<ITree> newChildren = newNode.getChildren();
endPosition2 = getEndPosition(newChildren);
if (endPosition == 0) {
endPosition = startPosition + actionSet.getLength();
}
if (endPosition2 == 0) {
endPosition2 = startPosition2 + newNode.getLength();
}
}
} else {// DEL actions and MOV actions: we don't need these actions, as for now.
continue;
}
if (startPosition == 0 || startPosition2 == 0) {
continue;
}
// Get line numbers.
int startLine = prevUnit.getLineNumber(startPosition);
int endLine = prevUnit.getLineNumber(endPosition);
int startLine2 = revUnit.getLineNumber(startPosition2);
int endLine2 = revUnit.getLineNumber(endPosition2);
Violation violation = findViolation(startLine, endLine, violations);
if (violation == null) continue;
if (endLine - startLine >= Configuration.HUNK_SIZE - 2 || endLine2 - startLine2 >= Configuration.HUNK_SIZE - 2 ) continue;
/*
* Convert the ITree of buggy code to a simple tree.
* It will be used to compute the similarity.
*/
SimplifyTree abstractIdentifier = new SimplifyTree();
abstractIdentifier.abstractTree(actionSet);
SimpleTree simpleTree = actionSet.getSimpleTree();
if (simpleTree == null) { // Failed to get the simple tree for INS actions.
continue;
}
/**
* Select edit scripts for deep learning.
* Edit scripts will be used to mine common fix patterns.
*/
// 1. First level: AST node type.
String astEditScripts = getASTEditScriptsBreadthFirst(actionSet);
int size = astEditScripts.split(" ").length;
if (size == 1) {
continue;
}
// Source Code of patches.
String patchSourceCode = getPatchSourceCode(prevFile, revFile, startLine, endLine, startLine2, endLine2, false);
this.patchesSourceCode += Configuration.PATCH_SIGNAL + "\n" + revFile.getName() + "\n" + patchSourceCode + "\n";
this.sizes += size + "\n";
this.astEditScripts += astEditScripts + "\n";
this.violationTypes += violation.getViolationType() + "\n";
// 2. source code: raw tokens
// String rawTokenEditScripts = getRawTokenEditScripts(actionSet);
// // 3. abstract identifiers:
// String abstractIdentifiersEditScripts = getAbstractIdentifiersEditScripts(actionSet);
// // 4. semi-source code:
// String semiSourceCodeEditScripts = getSemiSourceCodeEditScripts(actionSet);
// this.buggyTrees += Configuration.BUGGY_TREE_TOKEN + "\n" + simpleTree.toString() + "\n";
this.tokensOfSourceCode += Tokenizer.getTokensDeepFirst(simpleTree).trim() + "\n";
// this.actionSets += Configuration.BUGGY_TREE_TOKEN + "\n" + readActionSet(actionSet, "") + "\n";
// this.originalTree += Configuration.BUGGY_TREE_TOKEN + "\n" + actionSet.getOriginalTree().toString() + "\n";
}
actionSets.clear();
} else {
System.out.println(1);
}
}
protected Violation findViolation(int startLine, int endLine, List<Violation> violations) {
for (Violation violation : violations) {
int vStartLine = violation.getStartLineNum();
int vEndLine = violation.getEndLineNum();
if (!(startLine > vEndLine && endLine< vStartLine)) return violation;
}
return null;
}
}
src/main/java/edu/lu/uni/serval/FixPatternParser/violations/TestHunkParser.java
-266
View File
@@ -1,266 +0,0 @@
package edu.lu.uni.serval.FixPatternParser.violations;
import static java.lang.System.err;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Scanner;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import edu.lu.uni.serval.FixPatternParser.RunnableParser;
import edu.lu.uni.serval.MultipleThreadsParser.MessageFile;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
public class TestHunkParser {
public static void main(String[] args) {
// input data
final List<MessageFile> msgFiles = getMessageFiles(Configuration.GUM_TREE_INPUT);
System.out.println(msgFiles.size());
// output path
final String editScriptsFilePath = Configuration.EDITSCRIPTS_FILE;
final String patchesSourceCodeFilePath = Configuration.PATCH_SOURCECODE_FILE;
final String buggyTokensFilePath = Configuration.BUGGY_CODE_TOKENS_FILE;
final String editScriptSizesFilePath = Configuration.EDITSCRIPT_SIZES_FILE;
final String alarmTypesFilePath = Configuration.ALARM_TYPES_FILE;
FileHelper.deleteDirectory(editScriptsFilePath);
FileHelper.deleteDirectory(patchesSourceCodeFilePath);
FileHelper.deleteDirectory(buggyTokensFilePath);
FileHelper.deleteDirectory(editScriptSizesFilePath);
FileHelper.deleteDirectory(alarmTypesFilePath);
StringBuilder astEditScripts = new StringBuilder();
StringBuilder tokens = new StringBuilder();
StringBuilder sizes = new StringBuilder();
StringBuilder patches = new StringBuilder();
StringBuilder alarmTypes = new StringBuilder();
int a = 0;
// int counter = 0;
for (MessageFile msgFile : msgFiles) {
FixedViolationHunkParser parser = new FixedViolationHunkParser();
parser.setPositionFile(msgFile.getPositionFile());
final ExecutorService executor = Executors.newSingleThreadExecutor();
// schedule the work
final Future<?> future = executor.submit(new RunnableParser(msgFile.getPrevFile(),
msgFile.getRevFile(), msgFile.getDiffEntryFile(), parser));
try {
// where we wait for task to complete
future.get(Configuration.SECONDS_TO_WAIT, TimeUnit.SECONDS);
String editScripts = parser.getAstEditScripts();
if (!editScripts.equals("")) {
astEditScripts.append(editScripts);
tokens.append(parser.getTokensOfSourceCode());
sizes.append(parser.getSizes());
patches.append(parser.getPatchesSourceCode());
alarmTypes.append(parser.getAlarmTypes());
a ++;
if (a % 100 == 0) {
FileHelper.outputToFile(editScriptsFilePath, astEditScripts, true);
FileHelper.outputToFile(buggyTokensFilePath, tokens, true);
FileHelper.outputToFile(editScriptSizesFilePath, sizes, true);
FileHelper.outputToFile(patchesSourceCodeFilePath, patches, true);
FileHelper.outputToFile(alarmTypesFilePath, alarmTypes, true);
astEditScripts.setLength(0);
tokens.setLength(0);
sizes.setLength(0);
patches.setLength(0);
alarmTypes.setLength(0);
System.out.println("Finish of parsing " + a + " files......");
}
}
} catch (TimeoutException e) {
err.println("task timed out");
future.cancel(true /* mayInterruptIfRunning */ );
} catch (InterruptedException e) {
err.println("task interrupted");
} catch (ExecutionException e) {
err.println("task aborted");
} finally {
executor.shutdownNow();
}
}
FileHelper.outputToFile(editScriptsFilePath, astEditScripts, true);
FileHelper.outputToFile(buggyTokensFilePath, tokens, true);
FileHelper.outputToFile(editScriptSizesFilePath, sizes, true);
FileHelper.outputToFile(patchesSourceCodeFilePath, patches, true);
FileHelper.outputToFile(alarmTypesFilePath, alarmTypes, true);
astEditScripts.setLength(0);
tokens.setLength(0);
sizes.setLength(0);
patches.setLength(0);
alarmTypes.setLength(0);
System.out.println(a);
// classifyByAlarmTypes();
}
private static List<MessageFile> getMessageFiles(String gumTreeInput) {
String inputPath = gumTreeInput; // prevFiles revFiles diffentryFile positionsFile
File revFilesPath = new File(inputPath + "revFiles/");
File[] revFiles = revFilesPath.listFiles(); // project folders
List<MessageFile> msgFiles = new ArrayList<>();
for (File revFile : revFiles) {
if (revFile.getName().endsWith(".java")) {
String fileName = revFile.getName();
File prevFile = new File(gumTreeInput + "prevFiles/prev_" + fileName);// previous file
fileName = fileName.replace(".java", ".txt");
File diffentryFile = new File(gumTreeInput + "diffentries/" + fileName); // DiffEntry file
File positionFile = new File(gumTreeInput + "positions/" + fileName); // position file
MessageFile msgFile = new MessageFile(revFile, prevFile, diffentryFile);
msgFile.setPositionFile(positionFile);
msgFiles.add(msgFile);
}
}
return msgFiles;
}
public static void classifyByAlarmTypes() {
final String alarmTypesFilePath = Configuration.ALARM_TYPES_FILE;
List<String> alarmTypes = readStringList(alarmTypesFilePath);
//edit scripts, sizes of edit scripts, buggy tokens, patches.
classifyByAlarmTypes(alarmTypes, Configuration.EDITSCRIPT_SIZES_FILE);
classifyByAlarmTypes(alarmTypes, Configuration.EDITSCRIPTS_FILE);
classifyByAlarmTypes(alarmTypes, Configuration.BUGGY_CODE_TOKENS_FILE);
classifyByAlarmTypes2(alarmTypes, Configuration.PATCH_SOURCECODE_FILE);
}
private static void classifyByAlarmTypes(List<String> alarmTypes, String file) {
Map<String, StringBuilder> buildersMap = new HashMap<>();
FileInputStream fis = null;
Scanner scanner = null;
try {
fis = new FileInputStream(file);
scanner = new Scanner(fis);
int counter = 0;
while (scanner.hasNextLine()) {
String alarmType = alarmTypes.get(counter);
StringBuilder builder = getBuilder(buildersMap, alarmType);
builder.append(scanner.nextLine() + "\n");
counter ++;
if (counter % 1000 == 0) {
outputBuilders(buildersMap, file);
}
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
try {
scanner.close();
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
outputBuilders(buildersMap, file);
}
private static void classifyByAlarmTypes2(List<String> alarmTypes, String patchSourcecodeFile) {
Map<String, StringBuilder> buildersMap = new HashMap<>();
FileInputStream fis = null;
Scanner scanner = null;
try {
fis = new FileInputStream(patchSourcecodeFile);
scanner = new Scanner(fis);
int counter = 0;
String singlePatch = "";
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
if (Configuration.PATCH_SIGNAL.equals(line)) {
if (!"".equals(singlePatch)) {
String alarmType = alarmTypes.get(counter);
StringBuilder builder = getBuilder(buildersMap, alarmType);
builder.append(scanner.nextLine() + "\n");
counter ++;
if (counter % 2000 == 0) {
outputBuilders(buildersMap, patchSourcecodeFile);
}
}
singlePatch = line + "\n";
}
singlePatch += line + "\n";
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
try {
scanner.close();
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
outputBuilders(buildersMap, patchSourcecodeFile);
}
private static void outputBuilders(Map<String, StringBuilder> map, String fileNameStr) {
File file = new File(fileNameStr);
String fileName = file.getName();
String parentPath = file.getParent();
for (Map.Entry<String, StringBuilder> entry : map.entrySet()) {
String alarmType = entry.getKey();
StringBuilder builder = entry.getValue();
FileHelper.outputToFile(parentPath + "/" + alarmType + "/" + fileName, builder, true);
builder.setLength(0);
entry.setValue(builder);
}
}
public static List<String> readStringList(String inputFile) {
List<String> list = new ArrayList<>();
FileInputStream fis = null;
Scanner scanner = null;
try {
fis = new FileInputStream(inputFile);
scanner = new Scanner(fis);
while(scanner.hasNextLine()) {
list.add(scanner.nextLine());
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
try {
scanner.close();
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return list;
}
private static StringBuilder getBuilder(Map<String, StringBuilder> buildersMap, String alarmType) {
if (buildersMap.containsKey(alarmType)) {
return buildersMap.get(alarmType);
} else {
StringBuilder builder = new StringBuilder();
buildersMap.put(alarmType, builder);
return builder;
}
}
}
src/main/java/edu/lu/uni/serval/FixPatternParser/violations/Violation.java
-133
View File
@@ -1,133 +0,0 @@
package edu.lu.uni.serval.FixPatternParser.violations;
import java.util.ArrayList;
import java.util.List;
import edu.lu.uni.serval.diffentry.DiffEntryHunk;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalActionSet;
public class Violation implements Comparable<Violation> {
private String fileName = "";
private Integer startLineNum;
private int endLineNum;
private int bugStartLineNum;
private int bugEndLineNum;
private int fixStartLineNum = 0;
private int fixEndLineNum;
private String violationType;
private List<DiffEntryHunk> hunks = new ArrayList<>();
private List<HierarchicalActionSet> actionSets;
private int bugFixStartLineNum = 0; // the heuristic matched fix start line of a violation
private int bugFixEndLineNum = 0; // the heuristic matched fix end line of a violation
public Violation(Integer startLineNum, int endLineNum, String violationType) {
super();
this.startLineNum = startLineNum;
this.endLineNum = endLineNum;
this.violationType = violationType;
this.actionSets = new ArrayList<>();
}
public String getFileName() {
return fileName;
}
public void setFileName(String fileName) {
this.fileName = fileName;
}
public Integer getStartLineNum() {
return startLineNum;
}
public int getEndLineNum() {
return endLineNum;
}
public int getBugStartLineNum() {
return bugStartLineNum;
}
public void setBugStartLineNum(int bugStartLineNum) {
this.bugStartLineNum = bugStartLineNum;
}
public int getBugEndLineNum() {
return bugEndLineNum;
}
public void setBugEndLineNum(int bugEndLineNum) {
this.bugEndLineNum = bugEndLineNum;
}
public int getFixStartLineNum() {
return fixStartLineNum;
}
public void setFixStartLineNum(int fixStartLineNum) {
this.fixStartLineNum = fixStartLineNum;
}
public int getFixEndLineNum() {
return fixEndLineNum;
}
public void setFixEndLineNum(int fixEndLineNum) {
this.fixEndLineNum = fixEndLineNum;
}
public List<DiffEntryHunk> getHunks() {
return hunks;
}
public void setHunks(List<DiffEntryHunk> hunks) {
this.hunks = hunks;
}
public String getViolationType() {
return violationType;
}
public List<HierarchicalActionSet> getActionSets() {
return actionSets;
}
public int getBugFixStartLineNum() {
return bugFixStartLineNum;
}
public void setBugFixStartLineNum(int bugFixStartLineNum) {
this.bugFixStartLineNum = bugFixStartLineNum;
}
public int getBugFixEndLineNum() {
return bugFixEndLineNum;
}
public void setBugFixEndLineNum(int bugFixEndLineNum) {
this.bugFixEndLineNum = bugFixEndLineNum;
}
@Override
public int compareTo(Violation v) {
return this.startLineNum.compareTo(v.startLineNum);
}
@Override
public String toString() {
return this.startLineNum + " : " + this.endLineNum + " : " + this.violationType;
}
@Override
public boolean equals(Object obj) {
if (obj instanceof Violation) {
Violation v = (Violation) obj;
if (this.fileName.equals(v.fileName) && this.violationType.equals(v.violationType) && this.startLineNum == v.startLineNum && this.endLineNum == v.endLineNum) {
return true;
}
}
return false;
}
}
src/main/java/edu/lu/uni/serval/MultipleThreadsParser/AkkaParser.java
-130
View File
@@ -1,130 +0,0 @@
package edu.lu.uni.serval.MultipleThreadsParser;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import akka.actor.ActorRef;
import akka.actor.ActorSystem;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
/**
* Multi-thread parser of parsing the difference between buggy code file and fixed code file.
*
* @author kui.liu
*
*/
public class AkkaParser {
private static Logger log = LoggerFactory.getLogger(AkkaParser.class);
@SuppressWarnings("deprecation")
public static void main(String[] args) {
// input data
log.info("Get the input data...");
// final List<MessageFile> msgFiles = getMessageFiles();
final List<MessageFile> msgFiles = getMessageFiles(Configuration.GUM_TREE_INPUT);
log.info("MessageFiles: " + msgFiles.size());
// output path
final String editScriptsFilePath = Configuration.EDITSCRIPTS_FILE_PATH;
final String patchesSourceCodeFilePath = Configuration.PATCH_SOURCECODE_FILE_PATH;
final String buggyTokensFilePath = Configuration.BUGGY_CODE_TOKEN_FILE_PATH;
final String editScriptSizesFilePath = Configuration.EDITSCRIPT_SIZES_FILE_PATH;
final String alarmTypesFilePath = Configuration.ALARM_TYPES_FILE_PATH;
FileHelper.deleteDirectory(editScriptsFilePath);
FileHelper.deleteDirectory(patchesSourceCodeFilePath);
FileHelper.deleteDirectory(buggyTokensFilePath);
FileHelper.deleteDirectory(editScriptSizesFilePath);
FileHelper.deleteDirectory(alarmTypesFilePath);
ActorSystem system = null;
ActorRef parsingActor = null;
int numberOfWorkers = 431;
final WorkMessage msg = new WorkMessage(0, msgFiles);
try {
log.info("Akka begins...");
system = ActorSystem.create("Mining-FixPattern-System");
parsingActor = system.actorOf(ParseFixPatternActor.props(numberOfWorkers, editScriptsFilePath,
patchesSourceCodeFilePath, buggyTokensFilePath, editScriptSizesFilePath, alarmTypesFilePath), "mine-fix-pattern-actor");
parsingActor.tell(msg, ActorRef.noSender());
} catch (Exception e) {
system.shutdown();
e.printStackTrace();
}
}
/**
* Get bug commit-related files.
*
* @return
*/
public static List<MessageFile> getMessageFiles() {
String inputPath = Configuration.GUM_TREE_INPUT; //DiffEntries prevFiles revFiles
File inputFileDirector = new File(inputPath);
File[] files = inputFileDirector.listFiles(); // project folders
log.info("Projects: " + files.length);
List<MessageFile> msgFiles = new ArrayList<>();
for (File file : files) {
if (!file.isDirectory()) continue;
// if (!(file.getName().startsWith("k") || file.getName().startsWith("l"))) continue;
if (file.getName().startsWith("a") || file.getName().startsWith("b")
|| file.getName().startsWith("c") || file.getName().startsWith("d")
|| file.getName().startsWith("e") || file.getName().startsWith("f")
|| file.getName().startsWith("g") || file.getName().startsWith("h")
||file.getName().startsWith("h") || file.getName().startsWith("i")
|| file.getName().startsWith("k") || file.getName().startsWith("l")
|| file.getName().startsWith("j") || file.getName().startsWith("t")) continue;
// if (!file.getName().startsWith("j")) continue;
log.info("Project name: " + file.getName());
String projectFolder = file.getPath();
File revFileFolder = new File(projectFolder + "/revFiles/");// revised file folder
File[] revFiles = revFileFolder.listFiles();
for (File revFile : revFiles) {
if (revFile.getName().endsWith(".java")) {
File prevFile = new File(projectFolder + "/prevFiles/prev_" + revFile.getName());// previous file
File diffentryFile = new File(projectFolder + "/DiffEntries/" + revFile.getName().replace(".java", ".txt")); // DiffEntry file
MessageFile msgFile = new MessageFile(revFile, prevFile, diffentryFile);
msgFiles.add(msgFile);
}
}
}
return msgFiles;
}
/**
* Get violation-related files.
*
* @param gumTreeInput
* @return
*/
public static List<MessageFile> getMessageFiles(String gumTreeInput) {
String inputPath = gumTreeInput; // prevFiles revFiles diffentryFile positionsFile
File revFilesPath = new File(inputPath + "revFiles/");
File[] revFiles = revFilesPath.listFiles(); // project folders
List<MessageFile> msgFiles = new ArrayList<>();
for (File revFile : revFiles) {
if (revFile.getName().endsWith(".java")) {
String fileName = revFile.getName();
File prevFile = new File(gumTreeInput + "prevFiles/prev_" + fileName);// previous file
fileName = fileName.replace(".java", ".txt");
File diffentryFile = new File(gumTreeInput + "diffentries/" + fileName); // DiffEntry file
File positionFile = new File(gumTreeInput + "positions/" + fileName); // position file
MessageFile msgFile = new MessageFile(revFile, prevFile, diffentryFile);
msgFile.setPositionFile(positionFile);
msgFiles.add(msgFile);
}
}
return msgFiles;
}
}
src/main/java/edu/lu/uni/serval/MultipleThreadsParser/ParseFixPatternActor.java
+17 -24
View File
@@ -1,16 +1,14 @@
package edu.lu.uni.serval.MultipleThreadsParser;
import java.util.ArrayList;
import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import akka.actor.ActorRef;
import akka.actor.Props;
import akka.actor.UntypedActor;
import akka.japi.Creator;
import akka.routing.RoundRobinPool;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.List;
public class ParseFixPatternActor extends UntypedActor {
@@ -20,16 +18,13 @@ public class ParseFixPatternActor extends UntypedActor {
private final int numberOfWorkers;
private int counter = 0;
public ParseFixPatternActor(int numberOfWorkers, String editScriptsFilePath, String patchesSourceCodeFilePath,
String buggyTokensFilePath, String editScriptSizesFilePath, String alarmTypesFilePath) {
public ParseFixPatternActor(int numberOfWorkers, String project) {
mineRouter = this.getContext().actorOf(new RoundRobinPool(numberOfWorkers)
.props(ParseFixPatternWorker.props(editScriptsFilePath, patchesSourceCodeFilePath,
buggyTokensFilePath, editScriptSizesFilePath, alarmTypesFilePath)), "mine-fix-pattern-router");
.props(ParseFixPatternWorker.props(project)), "mine-fix-pattern-router");
this.numberOfWorkers = numberOfWorkers;
}
public static Props props(final int numberOfWorkers, final String editScriptsFilePath, final String patchesSourceCodeFilePath,
final String buggyTokensFilePath, final String editScriptSizesFilePath, final String alarmTypesFilePath) {
public static Props props(final int numberOfWorkers, final String project) {
return Props.create(new Creator<ParseFixPatternActor>() {
@@ -37,8 +32,7 @@ public class ParseFixPatternActor extends UntypedActor {
@Override
public ParseFixPatternActor create() throws Exception {
return new ParseFixPatternActor(numberOfWorkers, editScriptsFilePath, patchesSourceCodeFilePath,
buggyTokensFilePath, editScriptSizesFilePath, alarmTypesFilePath);
return new ParseFixPatternActor(numberOfWorkers, project);
}
});
@@ -50,26 +44,25 @@ public class ParseFixPatternActor extends UntypedActor {
if (message instanceof WorkMessage) {
List<MessageFile> files = ((WorkMessage) message).getMsgFiles();
int size = files.size();
int average = (int) Math.round((double) size / numberOfWorkers);
int average = size / numberOfWorkers;
int reminder = size % numberOfWorkers;
int counter = 0;
for (int i = 0; i < numberOfWorkers; i ++) {
int fromIndex = i * average;
int toIndex = (i + 1) * average;
if (i == numberOfWorkers - 1) {
toIndex = size;
}
int fromIndex = i * average + counter;
if (counter < reminder) counter ++;
int toIndex = (i + 1) * average + counter;
List<MessageFile> filesOfWorkers = new ArrayList<>();
filesOfWorkers.addAll(files.subList(fromIndex, toIndex));
List<MessageFile> filesOfWorkers = files.subList(fromIndex, toIndex);
final WorkMessage workMsg = new WorkMessage(i + 1, filesOfWorkers);
mineRouter.tell(workMsg, getSelf());
logger.info("Assign a task to worker #" + (i + 1) + "...");
}
} else if ("STOP".equals(message.toString())) {
counter ++;
logger.info(counter + " workers finished their work...");
logger.info(counter + " workers finailized their work...");
if (counter >= numberOfWorkers) {
logger.info("All workers finished their work...");
logger.info("All workers finailized their work...");
this.getContext().stop(mineRouter);
this.getContext().stop(getSelf());
this.getContext().system().shutdown();
src/main/java/edu/lu/uni/serval/MultipleThreadsParser/ParseFixPatternWorker.java
+25 -157
View File
@@ -1,57 +1,36 @@
package edu.lu.uni.serval.MultipleThreadsParser;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.StringReader;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import akka.actor.Props;
import akka.actor.UntypedActor;
import akka.japi.Creator;
import edu.lu.uni.serval.FixPatternParser.RunnableParser;
import edu.lu.uni.serval.FixPatternParser.violations.FixedViolationHunkParser;
import edu.lu.uni.serval.FixPatternParser.violations.Violation;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.File;
import java.util.List;
import java.util.concurrent.*;
public class ParseFixPatternWorker extends UntypedActor {
private static Logger log = LoggerFactory.getLogger(ParseFixPatternActor.class);
private String editScriptsFilePath;
private String patchesSourceCodeFilePath;
private String editScriptSizesFilePath;
private String buggyTokensFilePath;
private String alarmTypesFilePath;
private String project;
public ParseFixPatternWorker(String editScriptsFilePath, String patchesSourceCodeFilePath,
String buggyTokensFilePath, String editScriptSizesFilePath, String alarmTypesFilePath) {
this.editScriptsFilePath = editScriptsFilePath;
this.patchesSourceCodeFilePath = patchesSourceCodeFilePath;
this.editScriptSizesFilePath = editScriptSizesFilePath;
this.buggyTokensFilePath = buggyTokensFilePath;
this.alarmTypesFilePath = alarmTypesFilePath;
public ParseFixPatternWorker(String project) {
this.project = project;
}
public static Props props(final String editScriptsFile, final String patchesSourceCodeFile, final String buggyTokensFilePath,
final String editScriptSizesFilePath, final String alarmTypesFilePath) {
public static Props props(final String project) {
return Props.create(new Creator<ParseFixPatternWorker>() {
private static final long serialVersionUID = -7615153844097275009L;
@Override
public ParseFixPatternWorker create() throws Exception {
return new ParseFixPatternWorker(editScriptsFile, patchesSourceCodeFile,
buggyTokensFilePath, editScriptSizesFilePath, alarmTypesFilePath);
return new ParseFixPatternWorker(project);
}
});
@@ -66,13 +45,11 @@ public class ParseFixPatternWorker extends UntypedActor {
StringBuilder patchesSourceCode = new StringBuilder();
StringBuilder sizes = new StringBuilder();
StringBuilder tokens = new StringBuilder();
StringBuilder alarmTypes = new StringBuilder();
StringBuilder testingInfo = new StringBuilder();
int id = msg.getId();
int counter = 0;
int testViolations = 0;
int nullGumTreeResults = 0;
int noSourceCodeChanges = 0;
int noStatementChanges = 0;
@@ -85,32 +62,17 @@ public class ParseFixPatternWorker extends UntypedActor {
int timeouts = 0;
StringBuilder builder = new StringBuilder();
// Read violations with Null_Violation_Hunk or Illegal_Line_Position
// List<Violation> uselessViolations = new ArrayList<>();// readUselessViolations("logs/FixedViolationCodeParseResults.log");
for (MessageFile msgFile : files) {
File revFile = msgFile.getRevFile();
File prevFile = msgFile.getPrevFile();
File diffentryFile = msgFile.getDiffEntryFile();
File positionFile = msgFile.getPositionFile();
if (revFile.getName().toLowerCase().contains("test#") || revFile.getName().toLowerCase().contains("tests#")) {
// testViolations += countAlarms(positionFile, "#TestViolation:", uselessViolations);
testViolations += countAlarms(positionFile, "#TestViolation:");
continue;
}
// Parser parser = null;
// if (containsAlarmTypes || positionFile != null) {
// parser = new FixedViolationHunkParser(positionFile);
// containsAlarmTypes = true;
// } else {
// parser = new CommitPatchSingleStatementParser();
// }
FixedViolationHunkParser parser = new FixedViolationHunkParser(positionFile);
// parser.setUselessViolations(uselessViolations);
FixedViolationHunkParser parser = new FixedViolationHunkParser();
final ExecutorService executor = Executors.newSingleThreadExecutor();
// schedule the work
final Future<?> future = executor.submit(new RunnableParser(prevFile, revFile, diffentryFile, parser));
final Future<?> future = executor.submit(new RunnableParser(prevFile, revFile, diffentryFile, parser,project));
try {
// wait for task to complete
future.get(Configuration.SECONDS_TO_WAIT, TimeUnit.SECONDS);
@@ -126,146 +88,52 @@ public class ParseFixPatternWorker extends UntypedActor {
String editScript = parser.getAstEditScripts();
if ("".equals(editScript)) {
if (parser.resultType == 1) {
nullGumTreeResults += countAlarms(positionFile, "");
// System.err.println("#NullGumTreeResult:" + revFile.getName());
} else if (parser.resultType == 2) {
noSourceCodeChanges += countAlarms(positionFile, "");
} else if (parser.resultType == 3) {
noStatementChanges += countAlarms(positionFile, "");
// } else if (parser.resultType == 4) {
// illegalV += countAlarms(positionFile, "", uselessViolations);
}
} else {
editScripts.append(editScript);
patchesSourceCode.append(parser.getPatchesSourceCode());
sizes.append(parser.getSizes());
// tokens.append(parser.getTokensOfSourceCode());
alarmTypes.append(parser.getAlarmTypes());
tokens.append(parser.getTokensOfSourceCode());
counter ++;
if (counter % 100 == 0) {
FileHelper.outputToFile(editScriptsFilePath + "edistScripts_" + id + ".list", editScripts, true);
FileHelper.outputToFile(patchesSourceCodeFilePath + "patches_" + id + ".list", patchesSourceCode, true);
FileHelper.outputToFile(editScriptSizesFilePath + "sizes_" + id + ".list", sizes, true);
FileHelper.outputToFile(buggyTokensFilePath + "tokens_" + id + ".list", tokens, true);
editScripts.setLength(0);
patchesSourceCode.setLength(0);
sizes.setLength(0);
tokens.setLength(0);
FileHelper.outputToFile(alarmTypesFilePath + "alarmTypes_" + id + ".list", alarmTypes, true);
alarmTypes.setLength(0);
log.info("Worker #" + id +"Finish of parsing " + counter + " files...");
FileHelper.outputToFile("OUTPUT/testingInfo_" + id + ".list", testingInfo, true);
log.info("Worker #" + id +" finialized parsing " + counter + " files...");
testingInfo.setLength(0);
}
}
} catch (TimeoutException e) {
// err.println("task timed out");
future.cancel(true);
timeouts += countAlarms(positionFile, "#Timeout:");
// System.err.println("#Timeout: " + revFile.getName());
System.err.println("#Timeout: " + revFile.getName());
} catch (InterruptedException e) {
timeouts += countAlarms(positionFile, "#TimeInterrupted:");
// err.println("task interrupted");
// System.err.println("#TimeInterrupted: " + revFile.getName());
System.err.println("#TimeInterrupted: " + revFile.getName());
e.printStackTrace();
} catch (ExecutionException e) {
timeouts += countAlarms(positionFile, "#TimeAborted:");
// err.println("task aborted");
// System.err.println("#TimeAborted: " + revFile.getName());
System.err.println("#TimeAborted: " + revFile.getName());
e.printStackTrace();
} finally {
executor.shutdownNow();
}
}
if (sizes.length() > 0) {
FileHelper.outputToFile(editScriptsFilePath + "edistScripts_" + id + ".list", editScripts, true);
FileHelper.outputToFile(patchesSourceCodeFilePath + "patches_" + id + ".list", patchesSourceCode, true);
FileHelper.outputToFile(editScriptSizesFilePath + "sizes_" + id + ".list", sizes, true);
FileHelper.outputToFile(buggyTokensFilePath + "tokens_" + id + ".list", tokens, true);
editScripts.setLength(0);
patchesSourceCode.setLength(0);
sizes.setLength(0);
tokens.setLength(0);
FileHelper.outputToFile(alarmTypesFilePath + "alarmTypes_" + id + ".list", alarmTypes, true);
alarmTypes.setLength(0);
FileHelper.outputToFile("OUTPUT/testingInfo_" + id + ".list", testingInfo, true);
testingInfo.setLength(0);
}
String statistic = "TestViolations: " + testViolations + "\nNullGumTreeResults: " + nullGumTreeResults + "\nNoSourceCodeChanges: " + noSourceCodeChanges +
"\nNoStatementChanges: " + noStatementChanges + "\nNullDiffEntry: " + nullDiffEntry + "\nNullMatchedGumTreeResults: " + nullMappingGumTreeResults +
"\nPureDeletion: " + pureDeletion + "\nLargeHunk: " + largeHunk + "\nNullSourceCode: " + nullSourceCode +
"\nTestingInfo: " + testInfos + "\nTimeout: " + timeouts;
FileHelper.outputToFile("OUTPUT/statistic_" + id + ".list", statistic, false);
FileHelper.outputToFile("OUTPUT/UnfixedV_" + id + ".list", builder, false);
log.info("Worker #" + id +"Finish of parsing " + counter + " files...");
log.info("Worker #" + id + " finished the work...");
log.info("Worker #" + id +"finialized parsing " + counter + " files...");
log.info("Worker #" + id + " finialized the work...");
this.getSender().tell("STOP", getSelf());
} else {
unhandled(message);
}
}
// private List<Violation> readUselessViolations(String filePath) {
// List<Violation> uselessViolations = new ArrayList<>();
//
// String content = FileHelper.readFile(filePath);
// BufferedReader reader = new BufferedReader(new StringReader(content));
// String line = null;
// try {
// while ((line = reader.readLine()) != null) {
// if (line.startsWith("#")) {
// String[] elements = line.split(":");
// String fileName = elements[1];
// int startLine = Integer.parseInt(elements[2]);
// int endLine = Integer.parseInt(elements[3]);
// String violationType = elements[4];
//
// Violation violation = new Violation(startLine, endLine, violationType);
// violation.setFileName(fileName);
// uselessViolations.add(violation);
// }
// }
// } catch (IOException e) {
// e.printStackTrace();
// } finally {
// try {
// reader.close();
// } catch (IOException e) {
// e.printStackTrace();
// }
// }
// return uselessViolations;
// }
private int countAlarms(File positionFile, String type) {//, List<Violation> uselessViolations) {
int counter = 0;
String content = FileHelper.readFile(positionFile);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
try {
while ((line = reader.readLine()) != null) {
String[] elements = line.split(":");
Violation v = new Violation(Integer.parseInt(elements[1]), Integer.parseInt(elements[2]), elements[0]);
String fileName = positionFile.getName().replace(".txt", ".java");
v.setFileName(fileName);
counter ++;
if (!"".equals(type)) {
System.err.println(type + fileName + ":" + elements[1] + ":" + elements[2] + ":" + elements[0]);
}
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return counter;
}
}
src/main/java/edu/lu/uni/serval/MultipleThreadsParser/SingleThreadParser.java
-190
View File
@@ -1,190 +0,0 @@
package edu.lu.uni.serval.MultipleThreadsParser;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.StringReader;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import edu.lu.uni.serval.FixPatternParser.RunnableParser;
import edu.lu.uni.serval.FixPatternParser.violations.FixedViolationHunkParser;
import edu.lu.uni.serval.FixPatternParser.violations.Violation;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
public class SingleThreadParser {
public static void main(String[] args) {
// output path
final String editScriptsFilePath = Configuration.EDITSCRIPTS_FILE_PATH;
final String patchesSourceCodeFilePath = Configuration.PATCH_SOURCECODE_FILE_PATH;
final String buggyTokensFilePath = Configuration.BUGGY_CODE_TOKEN_FILE_PATH;
final String editScriptSizesFilePath = Configuration.EDITSCRIPT_SIZES_FILE_PATH;
final String alarmTypesFilePath = Configuration.ALARM_TYPES_FILE_PATH;
FileHelper.deleteDirectory(editScriptsFilePath);
FileHelper.deleteDirectory(patchesSourceCodeFilePath);
FileHelper.deleteDirectory(buggyTokensFilePath);
FileHelper.deleteDirectory(editScriptSizesFilePath);
FileHelper.deleteDirectory(alarmTypesFilePath);
final List<MessageFile> files = AkkaParser.getMessageFiles(Configuration.GUM_TREE_INPUT);
StringBuilder editScripts = new StringBuilder();
StringBuilder patchesSourceCode = new StringBuilder();
StringBuilder sizes = new StringBuilder();
StringBuilder tokens = new StringBuilder();
StringBuilder alarmTypes = new StringBuilder();
StringBuilder testingInfo = new StringBuilder();
int counter = 0;
int testViolations = 0;
int nullGumTreeResults = 0;
int noSourceCodeChanges = 0;
int noStatementChanges = 0;
int nullDiffEntry = 0;
int nullMappingGumTreeResults = 0;
int pureDeletion = 0;
int largeHunk = 0;
int nullSourceCode = 0;
int testInfos = 0;
int timeouts = 0;
StringBuilder builder = new StringBuilder();
for (MessageFile msgFile : files) {
File revFile = msgFile.getRevFile();
File prevFile = msgFile.getPrevFile();
File diffentryFile = msgFile.getDiffEntryFile();
File positionFile = msgFile.getPositionFile();
if (revFile.getName().toLowerCase().contains("test#") || revFile.getName().toLowerCase().contains("tests#")) {
testViolations += countAlarms(positionFile, "#TestViolation:");
continue;
}
FixedViolationHunkParser parser = new FixedViolationHunkParser(positionFile);
// parser.setUselessViolations(uselessViolations);
final ExecutorService executor = Executors.newSingleThreadExecutor();
// schedule the work
final Future<?> future = executor.submit(new RunnableParser(prevFile, revFile, diffentryFile, parser));
try {
// wait for task to complete
future.get(Configuration.SECONDS_TO_WAIT, TimeUnit.SECONDS);
nullDiffEntry += parser.nullMatchedDiffEntry;
nullMappingGumTreeResults += parser.nullMappingGumTreeResult;
pureDeletion += parser.pureDeletions;
largeHunk += parser.largeHunk;
nullSourceCode += parser.nullSourceCode;
testInfos += parser.testInfos;
testingInfo.append(parser.testingInfo);
builder.append(parser.unfixedViolations);
String editScript = parser.getAstEditScripts();
if ("".equals(editScript)) {
if (parser.resultType == 1) {
nullGumTreeResults += countAlarms(positionFile, "");
} else if (parser.resultType == 2) {
noSourceCodeChanges += countAlarms(positionFile, "");
} else if (parser.resultType == 3) {
noStatementChanges += countAlarms(positionFile, "");
// } else if (parser.resultType == 4) {
}
} else {
editScripts.append(editScript);
patchesSourceCode.append(parser.getPatchesSourceCode());
sizes.append(parser.getSizes());
alarmTypes.append(parser.getAlarmTypes());
tokens.append(parser.getTokensOfSourceCode());
counter ++;
if (counter % 5000 == 0) {
FileHelper.outputToFile(editScriptsFilePath + "edistScripts.list", editScripts, true);
FileHelper.outputToFile(patchesSourceCodeFilePath + "patches.list", patchesSourceCode, true);
FileHelper.outputToFile(editScriptSizesFilePath + "sizes.list", sizes, true);
FileHelper.outputToFile(buggyTokensFilePath + "tokens.list", tokens, true);
editScripts.setLength(0);
patchesSourceCode.setLength(0);
sizes.setLength(0);
tokens.setLength(0);
FileHelper.outputToFile(alarmTypesFilePath + "alarmTypes.list", alarmTypes, true);
alarmTypes.setLength(0);
FileHelper.outputToFile("OUTPUT/testingInfo.list", testingInfo, true);
testingInfo.setLength(0);
}
}
} catch (TimeoutException e) {
// err.println("task timed out");
future.cancel(true);
timeouts += countAlarms(positionFile, "#Timeout:");
// System.err.println("#Timeout: " + revFile.getName());
} catch (InterruptedException e) {
timeouts += countAlarms(positionFile, "#TimeInterrupted:");
// err.println("task interrupted");
// System.err.println("#TimeInterrupted: " + revFile.getName());
} catch (ExecutionException e) {
timeouts += countAlarms(positionFile, "#TimeAborted:");
// err.println("task aborted");
// System.err.println("#TimeAborted: " + revFile.getName());
} finally {
executor.shutdownNow();
}
}
if (sizes.length() > 0) {
FileHelper.outputToFile(editScriptsFilePath + "edistScripts.list", editScripts, true);
FileHelper.outputToFile(patchesSourceCodeFilePath + "patches.list", patchesSourceCode, true);
FileHelper.outputToFile(editScriptSizesFilePath + "sizes.list", sizes, true);
FileHelper.outputToFile(buggyTokensFilePath + "tokens.list", tokens, true);
editScripts.setLength(0);
patchesSourceCode.setLength(0);
sizes.setLength(0);
tokens.setLength(0);
FileHelper.outputToFile(alarmTypesFilePath + "alarmTypes.list", alarmTypes, true);
alarmTypes.setLength(0);
FileHelper.outputToFile("OUTPUT/testingInfo.list", testingInfo, true);
testingInfo.setLength(0);
}
String statistic = "TestViolations: " + testViolations + "\nNullGumTreeResults: " + nullGumTreeResults + "\nNoSourceCodeChanges: " + noSourceCodeChanges +
"\nNoStatementChanges: " + noStatementChanges + "\nNullDiffEntry: " + nullDiffEntry + "\nNullMatchedGumTreeResults: " + nullMappingGumTreeResults +
"\nPureDeletion: " + pureDeletion + "\nLargeHunk: " + largeHunk + "\nNullSourceCode: " + nullSourceCode +
"\nTestingInfo: " + testInfos + "\nTimeout: " + timeouts;
FileHelper.outputToFile("OUTPUT/statistic.list", statistic, false);
FileHelper.outputToFile("OUTPUT/UnfixedV.list", builder, false);
}
private static int countAlarms(File positionFile, String type) {//, List<Violation> uselessViolations) {
int counter = 0;
String content = FileHelper.readFile(positionFile);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
try {
while ((line = reader.readLine()) != null) {
String[] elements = line.split(":");
Violation v = new Violation(Integer.parseInt(elements[1]), Integer.parseInt(elements[2]), elements[0]);
String fileName = positionFile.getName().replace(".txt", ".java");
v.setFileName(fileName);
counter ++;
if (!"".equals(type)) {
System.err.println(type + fileName + ":" + elements[1] + ":" + elements[2] + ":" + elements[0]);
}
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return counter;
}
}
src/main/java/edu/lu/uni/serval/MultipleThreadsParser2/AkkaMatcher.java
-214
View File
@@ -1,214 +0,0 @@
package edu.lu.uni.serval.MultipleThreadsParser2;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Scanner;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import akka.actor.ActorRef;
import akka.actor.ActorSystem;
import edu.lu.uni.serval.MultipleThreadsParser.MessageFile;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.utils.MapSorter;
/**
* Multi-thread parser of parsing the difference between buggy code file and fixed code file.
*
* @author kui.liu
*
*/
public class AkkaMatcher {
private static Logger log = LoggerFactory.getLogger(AkkaMatcher.class);
public static void main(String[] args) {
List<Double[]> extractedFeatures = readStringList(Configuration.ROOT_PATH + "TestData/2_CNNinput.csv");
int size = extractedFeatures.size();
List<Double[]> trainingFeatures = extractedFeatures.subList(0, size - 178);
List<Double[]> bugFeatures = extractedFeatures.subList(size - 178, size);
for (int index = 0; index < 178; index ++) {
// AkkaMatcher computor = new AkkaMatcher();
// computor.matchFixPatterns(bugFeatures.get(index), index, trainingFeatures);
Map<Integer, Double> similarities = new HashMap<>();
for (int i = 0; i < size - 178; i ++) {
Double similarity = Math.abs(computeSimilarity(bugFeatures.get(index), trainingFeatures.get(i)));
similarities.put(i + 1, similarity);
}
MapSorter<Integer, Double> mapSorter = new MapSorter<Integer, Double>();
Map<Integer, Double> sortedSimilarities = mapSorter.sortByValueDescending(similarities);
List<Integer> similarityList = new ArrayList<>();
double similarity = 0;
int num = 0;
for (Map.Entry<Integer, Double> entry : sortedSimilarities.entrySet()) {
if (entry.getValue().equals(Double.NaN)) {
continue;
}
if (entry.getValue() == similarity) {
continue;
}
similarityList.add(entry.getKey());
similarity = entry.getValue();
if (++ num % 100 == 0) {
break;
}
}
outputMatchedPatterns(similarityList, index + 1);
}
}
private static Double computeSimilarity(Double[] feature, Double[] trainingFeature) {
Double similarity = DistanceCalculator.cosineSimilarityDistance(trainingFeature, feature);
return similarity;
}
private static void outputMatchedPatterns(List<Integer> similarityList, int bugId) {
String outputFile = Configuration.ROOT_PATH + "TestData/MatchedFixPatterns/Bug_" + bugId + ".list";
StringBuilder builder = new StringBuilder("BugId: " + bugId + "\n\n\n");
for (int i = 0, size = similarityList.size(); i < size; i ++) {
int patternPosition = similarityList.get(i);
builder.append(readPattern(patternPosition));
}
FileHelper.outputToFile(outputFile, builder, false);
}
private static String readPattern(int patternPosition) {
String patternInfo = "";
FileInputStream fis = null;
Scanner scanner = null;
try {
fis = new FileInputStream(Configuration.SELECTED_PATCHES_SOURE_CODE_FILE);
scanner = new Scanner(fis);
int index = 0;
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
if (line.equals("PATCH###")) {
if (patternPosition == index) {
break;
}
index ++;
patternInfo = "";
}
patternInfo += line + "\n";
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
try {
scanner.close();
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return patternInfo;
}
@SuppressWarnings("deprecation")
public void matchFixPatterns(Double[] bugFeature, int bugIndex, List<Double[]> trainingFeatures) {
ActorSystem system = null;
ActorRef parsingActor = null;
int numberOfWorkers = 100;
final WorkMessage msg = new WorkMessage(bugIndex, bugFeature, trainingFeatures);
try {
log.info("Akka begins...");
system = ActorSystem.create("Matching-FixPattern-System");
parsingActor = system.actorOf(MatchFixPatternActor.props(numberOfWorkers), "match-fix-pattern-actor");
parsingActor.tell(msg, ActorRef.noSender());
} catch (Exception e) {
system.shutdown();
e.printStackTrace();
}
}
/**
* Get bug commit-related files.
*
* @return
*/
public static List<MessageFile> getMessageFiles() {
String inputPath = Configuration.GUM_TREE_INPUT; //DiffEntries prevFiles revFiles
File inputFileDirector = new File(inputPath);
File[] files = inputFileDirector.listFiles(); // project folders
log.info("Projects: " + files.length);
List<MessageFile> msgFiles = new ArrayList<>();
for (File file : files) {
if (!file.isDirectory()) continue;
// if (!(file.getName().startsWith("k") || file.getName().startsWith("l"))) continue;
if (file.getName().startsWith("a") || file.getName().startsWith("b")
|| file.getName().startsWith("c") || file.getName().startsWith("d")
|| file.getName().startsWith("e") || file.getName().startsWith("f")
|| file.getName().startsWith("g") || file.getName().startsWith("h")
||file.getName().startsWith("h") || file.getName().startsWith("i")
|| file.getName().startsWith("k") || file.getName().startsWith("l")
|| file.getName().startsWith("j") || file.getName().startsWith("t")) continue;
// if (!file.getName().startsWith("j")) continue;
log.info("Project name: " + file.getName());
String projectFolder = file.getPath();
File revFileFolder = new File(projectFolder + "/revFiles/");// revised file folder
File[] revFiles = revFileFolder.listFiles();
for (File revFile : revFiles) {
if (revFile.getName().endsWith(".java")) {
File prevFile = new File(projectFolder + "/prevFiles/prev_" + revFile.getName());// previous file
File diffentryFile = new File(projectFolder + "/DiffEntries/" + revFile.getName().replace(".java", ".txt")); // DiffEntry file
MessageFile msgFile = new MessageFile(revFile, prevFile, diffentryFile);
msgFiles.add(msgFile);
}
}
}
return msgFiles;
}
public static List<Double[]> readStringList(String inputFile) {
List<Double[]> list = new ArrayList<>();
FileInputStream fis = null;
Scanner scanner = null;
try {
fis = new FileInputStream(inputFile);
scanner = new Scanner(fis);
while(scanner.hasNextLine()) {
String line = scanner.nextLine();
Double[] features = doubleParseFeature(line);
list.add(features);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
try {
scanner.close();
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return list;
}
private static Double[] doubleParseFeature(String feature) {
String[] features = feature.split(", ");
int length = features.length;
Double[] doubleFeatures = new Double[length];
for (int i = 0; i < length; i ++) {
doubleFeatures[i] = Double.parseDouble(features[i]);
}
return doubleFeatures;
}
}
src/main/java/edu/lu/uni/serval/MultipleThreadsParser2/DistanceCalculator.java
-68
View File
@@ -1,68 +0,0 @@
package edu.lu.uni.serval.MultipleThreadsParser2;
public class DistanceCalculator {
public static double calculateDistance(DistanceFunction distanceType, Double[] targetPoint, Double[] selfPoint) {
double distance = 0;
switch (distanceType) {
case EUCLIDEAN:
distance = DistanceCalculator.euclideanDistance(targetPoint, selfPoint);
break;
case COSINESIMILARITY:
distance = DistanceCalculator.cosineSimilarityDistance(targetPoint, selfPoint);
break;
case MANHATTAN:
distance = DistanceCalculator.manhattanDistance(targetPoint, selfPoint);
break;
default:
distance = DistanceCalculator.euclideanDistance(targetPoint, selfPoint);
break;
}
return distance;
}
public static double euclideanDistance(Double[] targetPoint, Double[] selfPoint) {
double sum = 0.0;
for (int i = 0, length = targetPoint.length; i < length; i++) {
double diff = targetPoint[i] - selfPoint[i];
sum += diff * diff;
}
return Math.sqrt(sum);
}
public static Double cosineSimilarityDistance(Double[] targetPoint, Double[] selfPoint) {
Double sim = 0.0d;
int length = targetPoint.length;
double dot = 0.0d;
double mag1 = 0.0d;
double mag2 = 0.0d;
for (int i = 0; i < length; i ++) {
dot += targetPoint[i] * selfPoint[i];
mag1 += Math.pow(targetPoint[i], 2);
mag2 += Math.pow(selfPoint[i], 2);
}
sim = dot / (Math.sqrt(mag1) * Math.sqrt(mag2));
return sim;
}
public static Double manhattanDistance(Double[] targetPoint, Double[] selfPoint) {
double result = 0.0;
for (int i = 0; i < targetPoint.length; i++) {
result += Math.abs(selfPoint[i] - targetPoint[i]);
}
return result;
}
public static Double minkowskiDistance(Double[] targetPoint, Double[] selfPoint) {
return null;
}
public static Double jaccardSimilarity(Double[] targetPoint, Double[] selfPoint) {
return null;
}
}
src/main/java/edu/lu/uni/serval/MultipleThreadsParser2/DistanceFunction.java
-5
View File
@@ -1,5 +0,0 @@
package edu.lu.uni.serval.MultipleThreadsParser2;
public enum DistanceFunction {
EUCLIDEAN, COSINESIMILARITY, MANHATTAN
}
src/main/java/edu/lu/uni/serval/MultipleThreadsParser2/MatchFixPatternActor.java
-172
View File
@@ -1,172 +0,0 @@
package edu.lu.uni.serval.MultipleThreadsParser2;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Scanner;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import akka.actor.ActorRef;
import akka.actor.Props;
import akka.actor.UntypedActor;
import akka.japi.Creator;
import akka.routing.RoundRobinPool;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.utils.ListSorter;
import edu.lu.uni.serval.utils.MapSorter;
public class MatchFixPatternActor extends UntypedActor {
private static Logger logger = LoggerFactory.getLogger(MatchFixPatternActor.class);
private ActorRef mineRouter;
private final int numberOfWorkers;
private int counter = 0;
public MatchFixPatternActor(int numberOfWorkers) {
mineRouter = this.getContext().actorOf(new RoundRobinPool(numberOfWorkers)
.props(MatchFixPatternWorker.props()), "match-fix-pattern-router");
this.numberOfWorkers = numberOfWorkers;
}
public static Props props(final int numberOfWorkers) {
return Props.create(new Creator<MatchFixPatternActor>() {
private static final long serialVersionUID = 9207427376110704705L;
@Override
public MatchFixPatternActor create() throws Exception {
return new MatchFixPatternActor(numberOfWorkers);
}
});
}
@SuppressWarnings("deprecation")
@Override
public void onReceive(Object message) throws Exception {
if (message instanceof WorkMessage) {
WorkMessage msg = (WorkMessage) message;
List<Double[]> trainingFeatures = msg.getTrainingFeatures();
int size = trainingFeatures.size();
int average = size / numberOfWorkers;
this.bugId = msg.getId();
Double[] bugFeature = msg.getBugFeature();
for (int i = 0; i < numberOfWorkers; i ++) {
int fromIndex = i * average;
int toIndex = (i + 1) * average;
if (i == numberOfWorkers - 1) {
toIndex = size;
}
List<Double[]> subTrainingFeatures = new ArrayList<>();
subTrainingFeatures.addAll(trainingFeatures.subList(fromIndex, toIndex));
final WorkMessage workMsg = new WorkMessage(bugId, bugFeature, subTrainingFeatures);
workMsg.setNum(i + 1);
mineRouter.tell(workMsg, getSelf());
logger.info("Assign a task to worker #" + (i + 1) + "...");
}
} else if (message instanceof ReturnMessage) {
counter ++;
logger.info(counter + " workers finished their work...");
ReturnMessage rMsg = (ReturnMessage) message;
returnMessages.add(rMsg);
if (counter >= numberOfWorkers) {
ListSorter<ReturnMessage> sorter = new ListSorter<ReturnMessage>(returnMessages);
returnMessages = sorter.sortAscending();
Map<Integer, Double> similarities = new HashMap<>();
int index = 0;
for (int i = 0; i < numberOfWorkers; i ++) {
ReturnMessage returnMessage = returnMessages.get(i);
List<Double> similarity = returnMessage.getSimilarities();
for (int j = 0, size = similarity.size(); j < size; j ++) {
index ++;
similarities.put(index, similarity.get(j));
}
}
MapSorter<Integer, Double> mapSorter = new MapSorter<Integer, Double>();
Map<Integer, Double> sortedSimilarities = mapSorter.sortByValueDescending(similarities);
List<Integer> similarityList = new ArrayList<>();
double similarity = 0;
int num = 0;
for (Map.Entry<Integer, Double> entry : sortedSimilarities.entrySet()) {
if (entry.getValue() == similarity) {
continue;
}
similarityList.add(entry.getKey());
similarity = entry.getValue();
if (++ num % 100 == 0) {
break;
}
}
outputMatchedPatterns(similarityList);
logger.info("All workers finished their work...");
this.getContext().stop(mineRouter);
this.getContext().stop(getSelf());
this.getContext().system().shutdown();
}
} else {
unhandled(message);
}
}
private void outputMatchedPatterns(List<Integer> similarityList) {
String outputFile = Configuration.ROOT_PATH + "TestData/MatchedFixPatterns/Bug_" + bugId + ".list";
StringBuilder builder = new StringBuilder("BugId: " + bugId + "\n\n\n");
for (int i = 0, size = similarityList.size(); i < size; i ++) {
int patternPosition = similarityList.get(i);
builder.append(readPattern(patternPosition));
}
FileHelper.outputToFile(outputFile, builder, false);
}
private String readPattern(int patternPosition) {
String patternInfo = "";
FileInputStream fis = null;
Scanner scanner = null;
try {
fis = new FileInputStream(Configuration.SELECTED_PATCHES_SOURE_CODE_FILE);
scanner = new Scanner(fis);
int index = 0;
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
if (line.equals("PATCH###")) {
if (patternPosition == index) {
break;
}
index ++;
patternInfo = "";
}
patternInfo += line + "\n";
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
try {
scanner.close();
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return patternInfo;
}
private int bugId;
private List<ReturnMessage> returnMessages = new ArrayList<>();
}
src/main/java/edu/lu/uni/serval/MultipleThreadsParser2/MatchFixPatternWorker.java
-60
View File
@@ -1,60 +0,0 @@
package edu.lu.uni.serval.MultipleThreadsParser2;
import java.util.ArrayList;
import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import akka.actor.Props;
import akka.actor.UntypedActor;
import akka.japi.Creator;
public class MatchFixPatternWorker extends UntypedActor {
private static Logger log = LoggerFactory.getLogger(MatchFixPatternActor.class);
public MatchFixPatternWorker() {
}
public static Props props() {
return Props.create(new Creator<MatchFixPatternWorker>() {
private static final long serialVersionUID = -7615153844097275009L;
@Override
public MatchFixPatternWorker create() throws Exception {
return new MatchFixPatternWorker();
}
});
}
@Override
public void onReceive(Object message) throws Exception {
if (message instanceof WorkMessage) {
WorkMessage msg = (WorkMessage) message;
List<Double[]> trainingFeatures = msg.getTrainingFeatures();
Double[] bugFeature = msg.getBugFeature();
int bugID = msg.getId();
int workNum = msg.getNum();
List<Double> similarities = new ArrayList<>();
for (int i = 0, size = trainingFeatures.size(); i < size; i ++) {
Double similarity = Math.abs(computeSimilarity(bugFeature, trainingFeatures.get(i)));
similarities.add(similarity);
}
final ReturnMessage rMsg = new ReturnMessage(bugID, workNum, similarities);
log.info("Worker #" + workNum + " finished the work...");
this.getSender().tell(rMsg, getSelf());
} else {
unhandled(message);
}
}
private Double computeSimilarity(Double[] feature, Double[] trainingFeature) {
Double similarity = DistanceCalculator.cosineSimilarityDistance(trainingFeature, feature);
return similarity;
}
}
src/main/java/edu/lu/uni/serval/MultipleThreadsParser2/ReturnMessage.java
-35
View File
@@ -1,35 +0,0 @@
package edu.lu.uni.serval.MultipleThreadsParser2;
import java.util.List;
public class ReturnMessage implements Comparable<ReturnMessage>{
private int bugId;
private Integer workerId;
private List<Double> similarities;
public ReturnMessage(int bugId, int workerId, List<Double> similarities) {
super();
this.bugId = bugId;
this.workerId = workerId;
this.similarities = similarities;
}
public int getBugId() {
return bugId;
}
public int getWorkerId() {
return workerId;
}
public List<Double> getSimilarities() {
return similarities;
}
@Override
public int compareTo(ReturnMessage o) {
return this.workerId.compareTo(o.workerId);
}
}
src/main/java/edu/lu/uni/serval/MultipleThreadsParser2/WorkMessage.java
-39
View File
@@ -1,39 +0,0 @@
package edu.lu.uni.serval.MultipleThreadsParser2;
import java.util.List;
public class WorkMessage {
private int id;
private Double[] bugFeature;
private List<Double[]> trainingFeatures;
private int num;
public WorkMessage(int id, Double[] bugFeature, List<Double[]> trainingFeatures) {
super();
this.id = id;
this.bugFeature = bugFeature;
this.trainingFeatures = trainingFeatures;
}
public int getId() {
return id;
}
public Double[] getBugFeature() {
return bugFeature;
}
public List<Double[]> getTrainingFeatures() {
return trainingFeatures;
}
public int getNum() {
return num;
}
public void setNum(int num) {
this.num = num;
}
}
src/main/java/edu/lu/uni/serval/MultipleThreadsParser3/AkkaParser.java
-90
View File
@@ -1,90 +0,0 @@
package edu.lu.uni.serval.MultipleThreadsParser3;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import akka.actor.ActorRef;
import akka.actor.ActorSystem;
import edu.lu.uni.serval.MultipleThreadsParser.MessageFile;
import edu.lu.uni.serval.MultipleThreadsParser.WorkMessage;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
/**
* Multi-thread parser of parsing source code of unfixed violations.
*
* @author kui.liu
*
*/
public class AkkaParser {
private static Logger log = LoggerFactory.getLogger(AkkaParser.class);
@SuppressWarnings("deprecation")
public static void main(String[] args) {
String violationType = Configuration.GUM_TREE_INPUT + "UnfixedViolations/";
File file = new File(violationType);
File[] violationTypes = file.listFiles();
for (File violationT : violationTypes) {
if (violationT.isDirectory()) {
violationType = violationT.getName();
if (//violationType.equals("SF_SWITCH_NO_DEFAULT") ||
// violationType.equals("SE_NO_SERIALVERSIONID") ||
violationType.equals("DM_DEFAULT_ENCODING")) {
// input data
log.info("Get the input data...");
final List<MessageFile> msgFiles = getMessageFiles(Configuration.GUM_TREE_INPUT, violationType);
log.info("MessageFiles: " + msgFiles.size());
// output path
final String sourceCodeFilesPath = Configuration.ROOT_PATH + "UnfixedViolations_RQ3/" + violationType + "/";
FileHelper.deleteDirectory(sourceCodeFilesPath);
ActorSystem system = null;
ActorRef parsingActor = null;
int numberOfWorkers = 200;
final WorkMessage msg = new WorkMessage(0, msgFiles);
try {
log.info("Akka begins...");
system = ActorSystem.create("Mining-Pattern-System-" + violationType);
parsingActor = system.actorOf(ParseFixPatternActor.props(numberOfWorkers, sourceCodeFilesPath, violationType), "mine-pattern-actor-" + violationType);
parsingActor.tell(msg, ActorRef.noSender());
} catch (Exception e) {
system.shutdown();
e.printStackTrace();
}
}
}
}
}
/**
* Get violation-related files.
*
* @param gumTreeInput
* @return
*/
public static List<MessageFile> getMessageFiles(String inputPath, String violationType) {
File sourceCodeFilesPath = new File(inputPath + "UnfixedViolations/" + violationType + "/");
File[] sourceCodeFiles = sourceCodeFilesPath.listFiles(); // project folders
List<MessageFile> msgFiles = new ArrayList<>();
for (File sourceCodeFile : sourceCodeFiles) {
if (sourceCodeFile.getName().endsWith(".java")) {
String fileName = sourceCodeFile.getName();
fileName = fileName.substring(8).replace(".java", ".txt");
File positionFile = new File(inputPath + "UnFV_positions/" + violationType + "/" + fileName); // position file
MessageFile msgFile = new MessageFile(null, sourceCodeFile, null);
msgFile.setPositionFile(positionFile);
msgFiles.add(msgFile);
}
}
return msgFiles;
}
}
src/main/java/edu/lu/uni/serval/MultipleThreadsParser3/AkkaParser2.java
-78
View File
@@ -1,78 +0,0 @@
package edu.lu.uni.serval.MultipleThreadsParser3;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import akka.actor.ActorRef;
import akka.actor.ActorSystem;
import edu.lu.uni.serval.MultipleThreadsParser.MessageFile;
import edu.lu.uni.serval.MultipleThreadsParser.WorkMessage;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
/**
* Multi-thread parser of parsing source code of fixed violations.
*
* @author kui.liu
*
*/
public class AkkaParser2 {
private static Logger log = LoggerFactory.getLogger(AkkaParser2.class);
@SuppressWarnings("deprecation")
public static void main(String[] args) {
// input data
log.info("Get the input data...");
final List<MessageFile> msgFiles = getMessageFiles(Configuration.GUM_TREE_INPUT);
log.info("MessageFiles: " + msgFiles.size());
// output path
final String sourceCodeFilesPath = Configuration.ROOT_PATH + "fixedViolations/";
FileHelper.deleteDirectory(sourceCodeFilesPath);
ActorSystem system = null;
ActorRef parsingActor = null;
int numberOfWorkers = 431;
final WorkMessage msg = new WorkMessage(0, msgFiles);
try {
log.info("Akka begins...");
system = ActorSystem.create("Mining-Pattern-System");
parsingActor = system.actorOf(ParseFixPatternActor.props(numberOfWorkers, sourceCodeFilesPath, "Type"), "mine-pattern-actor");
parsingActor.tell(msg, ActorRef.noSender());
} catch (Exception e) {
system.shutdown();
e.printStackTrace();
}
}
/**
* Get violation-related files.
*
* @param gumTreeInput
* @return
*/
public static List<MessageFile> getMessageFiles(String inputPath) {
File sourceCodeFilesPath = new File(inputPath + "prevFiles/");
File[] sourceCodeFiles = sourceCodeFilesPath.listFiles();
List<MessageFile> msgFiles = new ArrayList<>();
for (File sourceCodeFile : sourceCodeFiles) {
if (sourceCodeFile.getName().endsWith(".java")) {
String fileName = sourceCodeFile.getName();
fileName = fileName.substring(5).replace(".java", ".txt");
File positionFile = new File(inputPath + "positions/" + fileName); // position file
MessageFile msgFile = new MessageFile(null, sourceCodeFile, null);
msgFile.setPositionFile(positionFile);
msgFiles.add(msgFile);
}
}
return msgFiles;
}
}
src/main/java/edu/lu/uni/serval/MultipleThreadsParser3/ParseFixPatternActor.java
-80
View File
@@ -1,80 +0,0 @@
package edu.lu.uni.serval.MultipleThreadsParser3;
import java.util.ArrayList;
import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import akka.actor.ActorRef;
import akka.actor.Props;
import akka.actor.UntypedActor;
import akka.japi.Creator;
import akka.routing.RoundRobinPool;
import edu.lu.uni.serval.MultipleThreadsParser.MessageFile;
import edu.lu.uni.serval.MultipleThreadsParser.WorkMessage;
public class ParseFixPatternActor extends UntypedActor {
private static Logger logger = LoggerFactory.getLogger(ParseFixPatternActor.class);
private ActorRef mineRouter;
private final int numberOfWorkers;
private int counter = 0;
public ParseFixPatternActor(int numberOfWorkers, String sourceCodeFilesPath, String vType) {
mineRouter = this.getContext().actorOf(new RoundRobinPool(numberOfWorkers)
.props(ParseFixPatternWorker.props(sourceCodeFilesPath)), "mine-pattern-router-" + vType);
this.numberOfWorkers = numberOfWorkers;
}
public static Props props(final int numberOfWorkers, final String sourceCodeFilesPath, final String vType) {
return Props.create(new Creator<ParseFixPatternActor>() {
private static final long serialVersionUID = 9207427376110704705L;
@Override
public ParseFixPatternActor create() throws Exception {
return new ParseFixPatternActor(numberOfWorkers, sourceCodeFilesPath, vType);
}
});
}
@SuppressWarnings("deprecation")
@Override
public void onReceive(Object message) throws Exception {
if (message instanceof WorkMessage) {
List<MessageFile> files = ((WorkMessage) message).getMsgFiles();
int size = files.size();
int average = size / numberOfWorkers;
for (int i = 0; i < numberOfWorkers; i ++) {
int fromIndex = i * average;
int toIndex = (i + 1) * average;
if (i == numberOfWorkers - 1) {
toIndex = size;
}
List<MessageFile> filesOfWorkers = new ArrayList<>();
filesOfWorkers.addAll(files.subList(fromIndex, toIndex));
final WorkMessage workMsg = new WorkMessage(i + 1, filesOfWorkers);
mineRouter.tell(workMsg, getSelf());
logger.info("Assign a task to worker #" + (i + 1) + "...");
}
} else if ("STOP".equals(message.toString())) {
counter ++;
logger.info(counter + " workers finished their work...");
if (counter >= numberOfWorkers) {
logger.info("All workers finished their work...");
this.getContext().stop(mineRouter);
this.getContext().stop(getSelf());
this.getContext().system().shutdown();
}
} else {
unhandled(message);
}
}
}
src/main/java/edu/lu/uni/serval/MultipleThreadsParser3/ParseFixPatternWorker.java
-122
View File
@@ -1,122 +0,0 @@
package edu.lu.uni.serval.MultipleThreadsParser3;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import akka.actor.Props;
import akka.actor.UntypedActor;
import akka.japi.Creator;
import edu.lu.uni.serval.MultipleThreadsParser.MessageFile;
import edu.lu.uni.serval.MultipleThreadsParser.WorkMessage;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.violation.code.parser.ViolationCodeParser;
public class ParseFixPatternWorker extends UntypedActor {
private static Logger log = LoggerFactory.getLogger(ParseFixPatternActor.class);
private String sourceCodeFilesPath;
public ParseFixPatternWorker(String sourceCodeFilesPath) {
this.sourceCodeFilesPath = sourceCodeFilesPath;
}
public static Props props(final String sourceCodeFilesPath) {
return Props.create(new Creator<ParseFixPatternWorker>() {
private static final long serialVersionUID = -7615153844097275009L;
@Override
public ParseFixPatternWorker create() throws Exception {
return new ParseFixPatternWorker(sourceCodeFilesPath);
}
});
}
@Override
public void onReceive(Object message) throws Exception {
if (message instanceof WorkMessage) {
WorkMessage msg = (WorkMessage) message;
List<MessageFile> files = msg.getMsgFiles();
StringBuilder sourceCode = new StringBuilder();
StringBuilder sizes = new StringBuilder();
StringBuilder tokens = new StringBuilder();
List<String> violationTypes = readTypes();
int id = msg.getId();
int counter = 0;
for (MessageFile msgFile : files) {
File prevFile = msgFile.getPrevFile();
File positionFile = msgFile.getPositionFile();
if (prevFile.getName().toLowerCase().contains("test")) {
continue;
}
ViolationCodeParser parser = new ViolationCodeParser();
parser.parse(prevFile, positionFile);
parser.setTypes(violationTypes);
String sourceCodeStr = parser.sourceCode;
if ("".equals(sourceCodeStr)) {
} else {
sourceCode.append(sourceCodeStr);
sizes.append(parser.sizes);
tokens.append(parser.tokens);
counter ++;
if (counter % 100 == 0) {
FileHelper.outputToFile(sourceCodeFilesPath + "SourceCode/worker_" + id + ".list", sourceCode, true);
FileHelper.outputToFile(sourceCodeFilesPath + "Sizes/worker_" + id + ".list", sizes, true);
FileHelper.outputToFile(sourceCodeFilesPath + "Tokens/worker_" + id + ".list", tokens, true);
sourceCode.setLength(0);
sizes.setLength(0);
tokens.setLength(0);
}
}
}
if (sizes.length() > 0) {
FileHelper.outputToFile(sourceCodeFilesPath + "SourceCode/worker_" + id + ".list", sourceCode, true);
FileHelper.outputToFile(sourceCodeFilesPath + "Sizes/worker_" + id + ".list", sizes, true);
FileHelper.outputToFile(sourceCodeFilesPath + "Tokens/worker_" + id + ".list", tokens, true);
sourceCode.setLength(0);
sizes.setLength(0);
tokens.setLength(0);
}
log.info("Worker #" + id +"Finish of parsing " + counter + " files...");
log.info("Worker #" + id + " finished the work...");
this.getSender().tell("STOP", getSelf());
} else {
unhandled(message);
}
}
private List<String> readTypes() {
String fileName = Configuration.ROOT_PATH + "fixedViolations/types.list";
String content = FileHelper.readFile(fileName);
List<String> types = new ArrayList<>();
BufferedReader reader = new BufferedReader(new StringReader(content));
try {
String line = null;
while ((line = reader.readLine()) != null) {
types.add(line);
}
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
// System.err.println(types.size());
return types;
}
}
src/main/java/edu/lu/uni/serval/config/Configuration.java
+1 -88
View File
@@ -2,93 +2,6 @@ package edu.lu.uni.serval.config;
public class Configuration {
public static final long SECONDS_TO_WAIT = 300000L;
public static final String ROOT_PATH = "../FPM_Violations/"; // The root path of all output data.
public static final int HUNK_SIZE = 10; // The limitation of source code lines of each DiffEntry, which will be selected as training data.
public static final String BUGGY_TREE_SIGNAL = "BUGGY_TREE###"; // The starting signal of the tree of buggy source code .
public static final String PATCH_SIGNAL = "PATCH###"; // The starting signal of each patch.
// input path of GumTree. (i.e., Fix patterns parser)
public static final String GUM_TREE_INPUT = ROOT_PATH + "GumTreeInput/";// Buggy version file VS. Fixing version file, (DiffEntry File)
// the output path of GumTree results.
private static final String GUM_TREE_OUTPUT = ROOT_PATH + "GumTreeResults/";
public static final String EDITSCRIPTS_FILE_PATH = GUM_TREE_OUTPUT + "editScripts/";
public static final String PATCH_SOURCECODE_FILE_PATH = GUM_TREE_OUTPUT + "sourceCode/";
public static final String BUGGYTREE_FILE_PATH = GUM_TREE_OUTPUT + "buggyTrees/";
public static final String BUGGY_CODE_TOKEN_FILE_PATH = GUM_TREE_OUTPUT + "tokens/";
public static final String EDITSCRIPT_SIZES_FILE_PATH = GUM_TREE_OUTPUT + "editScriptSizes/";
public static final String ALARM_TYPES_FILE_PATH = GUM_TREE_OUTPUT + "alarmTypes/";
public static final String EDITSCRIPTS_FILE = GUM_TREE_OUTPUT + "editScripts.list";
public static final String PATCH_SOURCECODE_FILE = GUM_TREE_OUTPUT + "patchSourceCode.list";
public static final String BUGGYTREES_FILE = GUM_TREE_OUTPUT + "buggyTrees.list";
public static final String BUGGY_CODE_TOKENS_FILE = GUM_TREE_OUTPUT + "tokens.list";
public static final String EDITSCRIPT_SIZES_FILE = GUM_TREE_OUTPUT + "editScriptSizes.csv";
public static final String ALARM_TYPES_FILE = GUM_TREE_OUTPUT + "alarmTypes.list";
public static final int VECTOR_SIZE_OF_EMBEDED_TOKEN1 = 100; // tokens of edit scripts.
public static final int VECTOR_SIZE_OF_EMBEDED_TOKEN2 = 200; // tokens of source code
// the input path of fix patterns mining.
private static final String MINING_INPUT = ROOT_PATH + "MiningInput/";
public static final String MAX_TOKEN_VECTORS_SIZE_OF_EDIT_SCRIPTS = MINING_INPUT + "/MaxTokenVectorSizeOfEditScripts.list"; // The max size of edit scripts: upper limitation of max size.
public static final String MAX_TOKEN_VECTORS_SIZE_OF_SOURCE_CODE = MINING_INPUT + "/MaxTokenVectorSizeOfBuggySourceCode.list"; // The max size of all buggy source code token vectors.
// the input path of token embedding.
public static final String EMBEDDING_INPUT = MINING_INPUT + "Embedding/";
public static final String SELECTED_PATCHES_SOURE_CODE_FILE = EMBEDDING_INPUT + "patchSourceCode.list";// Selected patches.
public static final String SELECTED_BUGGY_TREE_FILE = EMBEDDING_INPUT + "buggyTrees.list";
public static final String SELECTED_BUGGY_TOKEN_FILE = EMBEDDING_INPUT + "tokens.list"; // Selected token vectors of buggy source code.
public static final String SELECTED_EDITSCRIPTES_FILE = EMBEDDING_INPUT + "editScripts.list"; // Selected edit script vectors.
public static final String SELECTED_ALARM_TYPES_FILE = EMBEDDING_INPUT + "alarmTypes.list"; // Selected edit script vectors.
// the input path of feature learning.
public static final String FEATURE_LEARNING_INPUT = MINING_INPUT + "FeatureLearning/";
public static final String EMBEDDED_EDIT_SCRIPT_TOKENS = FEATURE_LEARNING_INPUT + "embeddedEditScriptTokens.list"; // All embedded tokens of selected edit scripts.
public static final String VECTORIED_EDIT_SCRIPTS = FEATURE_LEARNING_INPUT + "vectorizedEditScripts.csv"; // Embedded and vectorized edit script vectors.
// the input path of clustering.
public static final String EXTRACTED_FEATURES = MINING_INPUT + "ExtractedFeatures/"; // Extracted features of all edit scripts.
public static final String CLUSTER_INPUT = MINING_INPUT + "ClusteringInput/input.arff";
// the output path of fix patterns mining.
private static final String MINING_OUTPUT = ROOT_PATH + "MiningOutput/";
public static final String CLUSTER_OUTPUT = MINING_OUTPUT + "ClusteringOutput/clusterResults.list";
public static final String CLUSTERED_PATCHES_FILE = MINING_OUTPUT + "ClusteredPatches/";
public static final String CLUSTERED_TOKENSS_FILE = MINING_OUTPUT + "ClusteredTokens/"; // Token vectors of buggy source code.
public static final String COMMON_CLUSTERS_SIZES = MINING_OUTPUT + "ClusteringOutput/CommonClusterSizes.list";
// evaluation data
public static final String TEST_INPUT = ROOT_PATH + "TestProjects/";
public static final String TEST_POSITION_FILE = ROOT_PATH + "TestData/Positions/"; // Positions of all test statements.
public static final String TEST_DATA_FILE = ROOT_PATH + "TestData/TestStatements/"; // Token vectors of all test statements.
public static final String NUMBER_OF_TRAINING_DATA = ROOT_PATH + "TestData/NumberOfTrainingData.list";;
// data of unsupervised learning
public static final String EMBEDDING_DATA_TOKENS1 = ROOT_PATH + "TestData/AllTokenVectorsForEvaluation.list";
public static final String EMBEDDED_ALL_TOKENS1 = ROOT_PATH + "TestData/AllEmbeddedTokens.list";
public static final String VECTORIED_ALL_SOURCE_CODE1 = ROOT_PATH + "TestData/AllVectorizedSourceCode/";
public static final String EXTRACTED_FEATURES_EVALUATION = ROOT_PATH + "TestDataExtractedFeatures/"; // extracted features of all source code (training data and testing data)
// Data of supervised learning
public static final String CLUSTERNUMBER_LABEL_MAP = ROOT_PATH + "TestData/clusterMappingLabel.list";
public static final String EMBEDDING_DATA_TOKENS2 = ROOT_PATH + "TestData/AllTokenVectorsForSupervisedEvaluation.list";
public static final String EMBEDDED_ALL_TOKENS2 = ROOT_PATH + "TestData/AllEmbeddedTokensForSuperVisedEvaluation.list";
public static final String TRAINING_DATA = ROOT_PATH + "TestData/TrainingData.csv"; // Training data of supervised learning
public static final String TESTING_DATA = ROOT_PATH + "TestData/SupervisedLearning/"; // testing data of supervised learning
public static final String FEATURES_OF_TRAINING_DATA = ROOT_PATH + "TestingOutput/TraingFeatures/";
public static final String FEATURES_OF_TESTING_DATA = ROOT_PATH + "TestingOutput/TestingFeatures/";
public static final String POSSIBILITIES_OF_TESTING_DATA = ROOT_PATH + "TestingOutput/Posibilities/";
public static final String PREDICTED_RESULTS_OF_TESTING_DATA = ROOT_PATH + "TestingOutput/Prediction/";
public static final String SUPERVISED_LEARNING_MODEL = ROOT_PATH + "TestingOutput/SupervisedLearningModel.zip";
public static final String FEATURES_OF_COMMON_CLUSTERS = ROOT_PATH + "FeaturesOfCommonClusters/";
public static final String TESTING_DATA_BUGS90 = ROOT_PATH + "Bugs/bugs90.list";
public static final String TESTING_DATA_BUGS80 = ROOT_PATH + "Bugs/bugs80.list";
public static final String TESTING_DATA_BUGS70 = ROOT_PATH + "Bugs/bugs70.list";
public static final String TESTING_DATA_BUGS60 = ROOT_PATH + "Bugs/bugs60.list";
public static final long SECONDS_TO_WAIT = 900L;
}
src/main/java/edu/lu/uni/serval/defects4j/Bug.java
-43
View File
@@ -1,43 +0,0 @@
package edu.lu.uni.serval.defects4j;
public class Bug {
private String project;
private String fileName;
private int startLine;
private int endLine;
private String type;
public Bug(String project, String fileName, int startLine, int endLine, String type) {
super();
this.project = project;
this.fileName = fileName;
this.startLine = startLine;
this.endLine = endLine;
this.type = type;
}
public String getProject() {
return project;
}
public String getFileName() {
return fileName;
}
public int getStartLine() {
return startLine;
}
public int getEndLine() {
return endLine;
}
public String getType() {
return type;
}
}
src/main/java/edu/lu/uni/serval/defects4j/BugParser.java
-206
View File
@@ -1,206 +0,0 @@
package edu.lu.uni.serval.defects4j;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;
import com.github.gumtreediff.tree.ITree;
import edu.lu.uni.serval.FixPatternParser.Tokenizer;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.gumtree.regroup.SimpleTree;
import edu.lu.uni.serval.gumtree.regroup.SimplifyTree;
import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.violation.code.parser.ViolationSourceCodeTree;
public class BugParser {
public static void main(String[] args) throws IOException {
List<Bug> bugs = new ArrayList<>();
String bugsFile = "Dataset/Defects4j/Bugs.txt";
String content = FileHelper.readFile(bugsFile);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
StringBuilder tokensBuilder = new StringBuilder();
StringBuilder buggyCodeBuilder = new StringBuilder();
while ((line = reader.readLine()) != null) {
String[] elements = line.split(" : ");
String type = elements[0];
String project = elements[1];
project = project.substring(0, project.lastIndexOf("."));
String fileName = elements[2];
int startLine = Integer.parseInt(elements[3]);
int endLine = Integer.parseInt(elements[4]);
Bug bug = new Bug(project, fileName, startLine, endLine, type);
bugs.add(bug);
}
int i = 0;
for (Bug bug : bugs) {
String type = bug.getType();
String project = bug.getProject();
String fileName = bug.getFileName();
int startLine = bug.getStartLine();
int endLine = bug.getEndLine();
File buggyFile = null;
String[] elements = project.split("_");
if (elements.length < 2) System.out.println(type + "-" + project + "-" + fileName + "-" + startLine + "-" + endLine);
List<File> javaFiles = FileHelper.getAllFiles(Configuration.ROOT_PATH + "Testing/projects1/" + elements[0] + "/" + elements[1] + "-b", ".java");
for (File javaFile : javaFiles) {
if (javaFile.getPath().endsWith(fileName)) {
buggyFile = javaFile;
break;
}
}
if (i == 31) {
i = i + 1 - 1;
}
SimpleTree simpleTree = null;
if ("EQ_DOESNT_OVERRIDE_EQUALS".equals(type)|| "HE_EQUALS_USE_HASHCODE".equals(type) || "HE_INHERITS_EQUALS_USE_HASHCODE".equals(type)||
"SE_NO_SUITABLE_CONSTRUCTOR".equals(type) || "RI_REDUNDANT_INTERFACES".equals(type)
// ||"CN_IDIOM_NO_SUPER_CALL".equals(type)
||"SE_NO_SERIALVERSIONID".equals(type) || "SE_COMPARATOR_SHOULD_BE_SERIALIZABLE".equals(type)) {
ViolationSourceCodeTree parser = new ViolationSourceCodeTree(buggyFile, startLine, endLine);
ITree classNameTree = parser.getClassNameTokens();
simpleTree = new SimplifyTree().canonicalizeSourceCodeTree(classNameTree, null);
startLine = parser.getViolationFinalStartLine();
endLine = parser.getViolationFinalEndLine();
} else {
ViolationSourceCodeTree alarmTree = new ViolationSourceCodeTree(buggyFile, startLine, endLine);
alarmTree.extract();
List<ITree> matchedTrees = alarmTree.getViolationSourceCodeTrees();
if (matchedTrees.size() == 0) {
System.err.println("#Null_Violation_Hunk: " + buggyFile.getName() + ":" + startLine + ":" + endLine);
continue;
}
simpleTree = new SimpleTree();
simpleTree.setLabel("Block");
simpleTree.setNodeType("Block");
List<SimpleTree> children = new ArrayList<>();
for (ITree matchedTree : matchedTrees) {
SimpleTree simpleT = new SimplifyTree().canonicalizeSourceCodeTree(matchedTree, null);
children.add(simpleT);
}
simpleTree.setChildren(children);
startLine = alarmTree.getViolationFinalStartLine();
endLine = alarmTree.getViolationFinalEndLine();
}
String tokens = Tokenizer.getTokensDeepFirst(simpleTree);
String[] tokensArray = tokens.split(" ");
int length = tokensArray.length;
System.out.println((++ i) + "==" + length);
// sizes += length + "\n";
// this.tokens += tokens + "\n";
String sourceCode = readSourceCode(buggyFile, startLine, endLine, type);
// this.sourceCode += sourceCode + "\n";
// tokensBuilder.append(type).append(":").append(tokens).append("\n");
tokensBuilder.append(tokens).append("\n");
buggyCodeBuilder.append(sourceCode).append("\n");
}
FileHelper.outputToFile("Dataset/Defects4j/buggyTokens.list", tokensBuilder, false);
FileHelper.outputToFile("Dataset/Defects4j/buggySourceCode.list", buggyCodeBuilder, false);
}
private static String readSourceCode(File javaFile, int startLine, int endLine, String violationType) {
StringBuilder sourceCode = new StringBuilder("##Source_Code:\n");
sourceCode.append(violationType).append("\n");
sourceCode.append(javaFile.getName().replaceAll("#", "/")).append("\nPosition: ").append(startLine).append(" : ").append(endLine).append("\n");
FileInputStream fis = null;
Scanner scanner = null;
try {
fis = new FileInputStream(javaFile);
scanner = new Scanner(fis);
int counter = 0;
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
counter ++;
if (startLine <= counter && counter <= endLine) {
sourceCode.append(line + "\n");
}
if (counter == endLine) break;
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
try {
scanner.close();
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sourceCode.toString();
}
}
/*
20
32
20
8
8
16
22
10
10
10
12
16
18
10
12
8
14
44
8
26
20
10
12
12
10
4
10
8
8
12
16
1776
8
10
10
12
8
50
18
8
12
38
4
4
14
16
20
26
8
14
12
14
*/
src/main/java/edu/lu/uni/serval/diffentry/DiffEntryHunk.java
-75
View File
@@ -1,75 +0,0 @@
package edu.lu.uni.serval.diffentry;
public class DiffEntryHunk {
private int bugLineStartNum;
private int fixLineStartNum;
private int bugRange;
private int fixRange;
private String hunk;
private int buggyHunkSize;
private int fixedHunkSize;
private String file;
public DiffEntryHunk(int bugLineStartNum, int fixLineStartNum, int bugRange, int fixRange) {
super();
this.bugLineStartNum = bugLineStartNum;
this.fixLineStartNum = fixLineStartNum;
this.bugRange = bugRange;
this.fixRange = fixRange;
}
public int getBugLineStartNum() {
return bugLineStartNum;
}
public int getFixLineStartNum() {
return fixLineStartNum;
}
public int getBugRange() {
return bugRange;
}
public int getFixRange() {
return fixRange;
}
public String getHunk() {
return hunk;
}
public void setHunk(String hunk) {
this.hunk = hunk;
}
public int getBuggyHunkSize() {
return buggyHunkSize;
}
public void setBuggyHunkSize(int buggyHunkSize) {
this.buggyHunkSize = buggyHunkSize;
}
public int getFixedHunkSize() {
return fixedHunkSize;
}
public void setFixedHunkSize(int fixedHunkSize) {
this.fixedHunkSize = fixedHunkSize;
}
public String getFile() {
return file;
}
public void setFile(String file) {
this.file = file;
}
@Override
public String toString() {
return "@@ -" + this.bugLineStartNum + ", " + this.bugRange + " +" + this.fixLineStartNum + ", " + this.fixRange + "\n" + this.hunk;
}
}
src/main/java/edu/lu/uni/serval/diffentry/DiffEntryReader.java
-248
View File
@@ -1,248 +0,0 @@
package edu.lu.uni.serval.diffentry;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.List;
import edu.lu.uni.serval.utils.FileHelper;
public class DiffEntryReader {
public List<DiffEntryHunk> readHunks(File diffentryFile) {
List<DiffEntryHunk> diffentryHunks = new ArrayList<>();
String content = FileHelper.readFile(diffentryFile);
BufferedReader reader = null;
try {
reader = new BufferedReader(new StringReader(content));
String line = null;
int startLine = 0;
int range = 0;
int startLine2 = 0;
int range2 = 0;
StringBuilder hunk = new StringBuilder();
boolean sourceCode = false;
while ((line = reader.readLine()) != null) {
if (RegExp.filterSignal(line.trim())) {
sourceCode = true;
if (hunk.length() > 0) {
if ((range < 7 && range2 < 7) || range == 0 || range2 == 0) { // filter out big hunks
DiffEntryHunk diffEntryHunk = new DiffEntryHunk(startLine, startLine2, range, range2);
diffEntryHunk.setHunk(hunk.toString());
diffentryHunks.add(diffEntryHunk);
}
hunk.setLength(0);
}
int plusIndex = line.indexOf("+");
String lineNum = line.substring(4, plusIndex);
String[] nums = lineNum.split(",");
startLine = Integer.parseInt(nums[0].trim());
if (nums.length == 2) {
range = Integer.parseInt(nums[1].trim());
}
String lineNum2 = line.substring(plusIndex) .trim();
lineNum2 = lineNum2.substring(1, lineNum2.length() - 2);
String[] nums2 = lineNum2.split(",");
startLine2 = Integer.parseInt(nums2[0].trim());
if (nums2.length == 2) {
range2 = Integer.parseInt(nums2[1].trim());
}
continue;
}
if (sourceCode) hunk.append(line + "\n");
}
if ((range < 7 && range2 < 7) || range == 0 || range2 == 0) { // filter out big hunks
DiffEntryHunk diffEntryHunk = new DiffEntryHunk(startLine, startLine2, range, range2);
diffEntryHunk.setHunk(hunk.toString());
diffentryHunks.add(diffEntryHunk);
}
hunk.setLength(0);
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (reader != null) {
reader.close();
reader = null;
}
} catch (IOException e) {
e.printStackTrace();
}
}
return diffentryHunks;
}
/**
* Read all hunks without considering their sizes.
*
* @param diffentryFile
* @return
*/
public List<DiffEntryHunk> readHunks2(File diffentryFile) {
List<DiffEntryHunk> diffentryHunks = new ArrayList<>();
String content = FileHelper.readFile(diffentryFile);
BufferedReader reader = null;
try {
reader = new BufferedReader(new StringReader(content));
String line = null;
int startLine = 0;
int range = 0;
int startLine2 = 0;
int range2 = 0;
StringBuilder hunk = new StringBuilder();
boolean sourceCode = false;
while ((line = reader.readLine()) != null) {
if (RegExp.filterSignal(line.trim())) {
sourceCode = true;
if (hunk.length() > 0) {
if (startLine > 0) {
DiffEntryHunk diffEntryHunk = new DiffEntryHunk(startLine, startLine2, range, range2);
diffEntryHunk.setHunk(hunk.toString());
diffentryHunks.add(diffEntryHunk);
}
hunk.setLength(0);
}
int plusIndex = line.indexOf("+");
String lineNum = line.substring(4, plusIndex);
String[] nums = lineNum.split(",");
startLine = Integer.parseInt(nums[0].trim());
if (nums.length == 2) {
range = Integer.parseInt(nums[1].trim());
}
String lineNum2 = line.substring(plusIndex) .trim();
lineNum2 = lineNum2.substring(1, lineNum2.length() - 2);
String[] nums2 = lineNum2.split(",");
startLine2 = Integer.parseInt(nums2[0].trim());
if (nums2.length == 2) {
range2 = Integer.parseInt(nums2[1].trim());
}
continue;
}
if (sourceCode) hunk.append(line + "\n");
}
if (startLine > 0) {
DiffEntryHunk diffEntryHunk = new DiffEntryHunk(startLine, startLine2, range, range2);
diffEntryHunk.setHunk(hunk.toString());
diffentryHunks.add(diffEntryHunk);
}
hunk.setLength(0);
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (reader != null) {
reader.close();
reader = null;
}
} catch (IOException e) {
e.printStackTrace();
}
}
return diffentryHunks;
}
/**
* Read all hunks with pure deleted lines and added lines.
*
* @param diffentryFile
* @return
*/
public List<DiffEntryHunk> readHunks3(File diffentryFile) {
List<DiffEntryHunk> diffentryHunks = new ArrayList<>();
String content = FileHelper.readFile(diffentryFile);
BufferedReader reader = null;
try {
reader = new BufferedReader(new StringReader(content));
String line = null;
int buggyStartLine = 0;
int buggyRange = 0;
int fixedStartLine = 0;
int fixedRange = 0;
int buggyHunkSize = 0;
int fixedHunkSize = 0;
StringBuilder hunk = new StringBuilder();
boolean sourceCode = false;
while ((line = reader.readLine()) != null) {
if (RegExp.filterSignal(line.trim())) {
sourceCode = true;
if (hunk.length() > 0) {
if (buggyStartLine > 0) {
DiffEntryHunk diffEntryHunk = new DiffEntryHunk(buggyStartLine, fixedStartLine, buggyRange, fixedRange);
diffEntryHunk.setHunk(hunk.toString());
diffEntryHunk.setBuggyHunkSize(buggyHunkSize);
diffEntryHunk.setFixedHunkSize(fixedHunkSize);
diffEntryHunk.setFile(diffentryFile.getPath());
diffentryHunks.add(diffEntryHunk);
}
hunk.setLength(0);
buggyStartLine = 0;
buggyRange = 0;
fixedStartLine = 0;
fixedRange = 0;
buggyHunkSize = 0;
fixedHunkSize = 0;
}
int plusIndex = line.indexOf("+");
String lineNum = line.substring(4, plusIndex);
String[] nums = lineNum.split(",");
buggyStartLine = Integer.parseInt(nums[0].trim());
if (nums.length == 2) {
buggyRange = Integer.parseInt(nums[1].trim());
}
String lineNum2 = line.substring(plusIndex) .trim();
lineNum2 = lineNum2.substring(1, lineNum2.length() - 2);
String[] nums2 = lineNum2.split(",");
fixedStartLine = Integer.parseInt(nums2[0].trim());
if (nums2.length == 2) {
fixedRange = Integer.parseInt(nums2[1].trim());
}
continue;
} else if (sourceCode) {
if (line.startsWith("-")) {
buggyHunkSize ++;
} else if (line.startsWith("+")) {
fixedHunkSize ++;
}
hunk.append(line + "\n");
}
}
if (buggyStartLine > 0 && hunk.length() > 0) {
DiffEntryHunk diffEntryHunk = new DiffEntryHunk(buggyStartLine, fixedStartLine, buggyRange, fixedRange);
diffEntryHunk.setHunk(hunk.toString());
diffEntryHunk.setBuggyHunkSize(buggyHunkSize);
diffEntryHunk.setFixedHunkSize(fixedHunkSize);
diffEntryHunk.setFile(diffentryFile.getPath());
diffentryHunks.add(diffEntryHunk);
hunk.setLength(0);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (reader != null) {
reader.close();
reader = null;
}
} catch (IOException e) {
e.printStackTrace();
}
}
return diffentryHunks;
}
}
src/main/java/edu/lu/uni/serval/diffentry/RegExp.java
-20
View File
@@ -1,20 +0,0 @@
package edu.lu.uni.serval.diffentry;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class RegExp {
private static final String REGULAR_EXPRESSION = "^@@\\s\\-\\d+,*\\d*\\s\\+\\d+,*\\d*\\s@@$"; //@@ -21,0 +22,2 @@
private static Pattern pattern = Pattern.compile(REGULAR_EXPRESSION);
public static boolean filterSignal(String string) {
boolean flag = false;
Matcher res = pattern.matcher(string);
if (res.matches()) {
flag = true;
}
return flag;
}
}
src/main/java/edu/lu/uni/serval/fixminer/EnhancedASTDiff.java
+101
View File
@@ -0,0 +1,101 @@
package edu.lu.uni.serval.fixminer;
import akka.actor.ActorRef;
import akka.actor.ActorSystem;
import edu.lu.uni.serval.MultipleThreadsParser.MessageFile;
import edu.lu.uni.serval.MultipleThreadsParser.ParseFixPatternActor;
import edu.lu.uni.serval.MultipleThreadsParser.WorkMessage;
import edu.lu.uni.serval.utils.FileHelper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.File;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.Stream;
public class EnhancedASTDiff {
private static Logger log = LoggerFactory.getLogger(EnhancedASTDiff.class);
public static void main(String inputPath, String outputPath,String numOfWorkers,String project) {
String parameters = String.format("\nInput path %s \nOutput path %s",inputPath,outputPath);
log.info(parameters);
File folder = new File(inputPath);
File[] listOfFiles = folder.listFiles();
Stream<File> stream = Arrays.stream(listOfFiles);
List<File> folders = stream
.filter(x -> !x.getName().startsWith("."))
.collect(Collectors.toList());
for (File target : folders) {
String pjName = target.getName();
final List<MessageFile> msgFiles = getMessageFiles(target.toString() + "/"); //"/Users/anilkoyuncu/bugStudy/code/python/GumTreeInput/Apache/CAMEL/"
System.out.println(msgFiles.size());
if(msgFiles.size() == 0)
continue;
String GUM_TREE_OUTPUT = outputPath + "/"+ pjName + "/";
FileHelper.createDirectory(GUM_TREE_OUTPUT + "/UPD");
FileHelper.createDirectory(GUM_TREE_OUTPUT + "/INS");
FileHelper.createDirectory(GUM_TREE_OUTPUT + "/DEL");
FileHelper.createDirectory(GUM_TREE_OUTPUT + "/MOV");
FileHelper.createDirectory(GUM_TREE_OUTPUT + "/ALL");
int a = 0;
ActorSystem system = null;
ActorRef parsingActor = null;
final WorkMessage msg = new WorkMessage(0, msgFiles);
try {
log.info("Akka begins...");
system = ActorSystem.create("Mining-FixPattern-System");
parsingActor = system.actorOf(ParseFixPatternActor.props(Integer.valueOf(numOfWorkers), project), "mine-fix-pattern-actor");
parsingActor.tell(msg, ActorRef.noSender());
} catch (Exception e) {
system.shutdown();
e.printStackTrace();
}
}
}
private static List<MessageFile> getMessageFiles(String gumTreeInput) {
String inputPath = gumTreeInput; // prevFiles revFiles diffentryFile positionsFile
File revFilesPath = new File(inputPath + "revFiles/");
File[] revFiles = revFilesPath.listFiles(); // project folders
List<MessageFile> msgFiles = new ArrayList<>();
if (revFiles.length >= 0) {
for (File revFile : revFiles) {
String fileName = revFile.getName();
File prevFile = new File(gumTreeInput + "prevFiles/prev_" + fileName);// previous file
fileName = fileName.replace(".java", ".txt");
File diffentryFile = new File(gumTreeInput + "DiffEntries/" + fileName); // DiffEntry file
MessageFile msgFile = new MessageFile(revFile, prevFile, diffentryFile);
msgFiles.add(msgFile);
}
return msgFiles;
}
else{
return null;
}
}
}
src/main/java/edu/lu/uni/serval/fixminer/Launcher.java
+173
View File
@@ -0,0 +1,173 @@
package edu.lu.uni.serval.fixminer;
import edu.lu.uni.serval.fixminer.cluster.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.Properties;
/**
* Created by anilkoyuncu on 14/04/2018.
*/
public class Launcher {
private static Logger log = LoggerFactory.getLogger(Launcher.class);
public static void main(String[] args) throws IOException {
Properties appProps = new Properties();
// String appConfigPath = "/Users/kui.liu/Downloads/app.properties";//args[0];
// String appConfigPath = "/Users/anilkoyuncu/bugStudy/release/code/app.properties";
String appConfigPath = args[0];
appProps.load(new FileInputStream(appConfigPath));
String portInner = appProps.getProperty("portInner","6380");
// String serverWait = appProps.getProperty("serverWait", "50000");
String numOfWorkers = appProps.getProperty("numOfWorkers", "10");
String jobType = appProps.getProperty("jobType","ALL");
String portDumps = appProps.getProperty("portDumps","6399");
String pythonPath = appProps.getProperty("pythonPath","/Users/anilkoyuncu/bugStudy/code/python");
String datasetPath = appProps.getProperty("datasetPath","/Users/anilkoyuncu/bugStudy/dataset");
String pjName = appProps.getProperty("pjName","allDataset");
// String dbNo = appProps.getProperty("dbNo","0");
String actionType = appProps.getProperty("actionType","ALL");
String threshold = appProps.getProperty("threshold","1");
String cursor = appProps.getProperty("cursor","10000000");
String chunk = appProps.getProperty("chunk","1.txt");
String parameters = String.format("\nportInner %s " +
"\nnumOfWorkers %s " +
"\njobType %s \nport %s " +
"\npythonPath %s \ndatasetPath %s" +
"\npjName %s \nactionType %s \nthreshold %s \ncursor %s"
, portInner, numOfWorkers, jobType, portDumps, pythonPath,datasetPath,pjName,actionType,threshold,cursor);
log.info(parameters);
mainLaunch(portInner, numOfWorkers, jobType, portDumps, pythonPath,datasetPath,pjName,actionType,threshold,cursor,chunk);
}
public static void mainLaunch(String portInner, String numOfWorkers, String jobType, String portDumps, String pythonPath, String datasetPath, String pjName, String actionType, String threshold, String cursor, String chunk){
String dbDir;
String pairsPath;
String dumpsName;
String gumInput;
String gumOutput;
gumInput = datasetPath +"/"+pjName+"/";
gumOutput = datasetPath + "/EnhancedASTDiff" + pjName;
dbDir = datasetPath + "/redis";
pairsPath = datasetPath + "/pairsImport"+pjName;
dumpsName = "dumps-"+pjName+".rdb";
try {
switch (jobType) {
case "ENHANCEDASTDIFF":
EnhancedASTDiff.main(gumInput, gumOutput, numOfWorkers, pjName);
break;
case "CACHE":
StoreEDiffInCache.main(gumOutput, portDumps, dbDir, actionType+dumpsName,actionType);
break;
case "SI":
CalculatePairs.main(dbDir, actionType+dumpsName, portDumps, pairsPath+actionType, pjName+actionType);
ImportPairs2DB.main(pairsPath+actionType, portInner, dbDir,datasetPath);
break;
case "SIMI":
AkkaTreeLoader.main(portInner, dbDir, pjName +actionType+chunk+".rdb" , portDumps, actionType+dumpsName,pairsPath+actionType,numOfWorkers,cursor,chunk);
break;
case "LEVEL1":
level1(portInner, portDumps, pythonPath, datasetPath, pjName, actionType, threshold, dbDir, pairsPath, dumpsName, gumInput);
break;
//CALC python abstractPatch.py to from cluster folder
case "LEVEL2":
level2(portDumps, pythonPath, datasetPath, pjName, actionType, threshold, dbDir, dumpsName, gumInput);
break;
//CALC via python
case "LEVEL3":
level3(portDumps, pythonPath, datasetPath, pjName, actionType, threshold, dbDir, dumpsName, gumInput);
break;
case "EXTRACTPATTERN":
PatternExtractor.mainLaunch(portInner,numOfWorkers,jobType,portDumps,pythonPath,datasetPath,pjName,actionType,threshold);
break;
case "GETPATTERN":
PatternExtractor.mainLaunch(portInner,numOfWorkers,jobType,portDumps,pythonPath,datasetPath,pjName,actionType,threshold);
break;
}
} catch (Exception e) {
e.printStackTrace();
}
}
private static void level1(String portInner, String port, String pythonPath, String datasetPath, String pjName, String actionType, String threshold, String dbDir, String pairsPath, String dumpsName, String gumInput) throws Exception {
TreeLoaderClusterL1.main(portInner, port, dbDir, "level1-"+pjName+ actionType+".rdb", dbDir ,pjName + actionType);
CallShell cs1 =new CallShell();
String db1 = "bash "+dbDir + "/" + "startServer.sh" +" %s %s %s";
String db11 = String.format(db1, dbDir,"level1-"+pjName+ actionType+".rdb" ,Integer.valueOf(port));
cs1.runShell(db11, port);
String runpy = "bash "+datasetPath + "/" + "launchPy.sh" +" %s %s %s %s %s %s";
String formatRunPy = String.format(runpy,pythonPath +"/abstractPatch.py", gumInput, datasetPath + "/cluster"+pjName+ actionType, port, "matches" + pjName + actionType, threshold);
cs1.runShell(formatRunPy);
String stopServer = "bash "+dbDir + "/" + "stopServer.sh" +" %s";
stopServer = String.format(stopServer,Integer.valueOf(port));
cs1.runShell(stopServer, port);
}
private static void level2(String port, String pythonPath, String datasetPath, String pjName, String actionType, String threshold, String dbDir, String dumpsName, String gumInput) throws Exception {
String stopServer;
MultiThreadTreeLoaderCluster.calculatePairsOfClusters(datasetPath + "/cluster"+pjName+ actionType, datasetPath,actionType);
MultiThreadTreeLoaderCluster.mainCompare("6300", datasetPath+"/pairs"+actionType, datasetPath + "/redisSingleImport.sh", dbDir, "clusterl1-"+pjName+actionType+".rdb", actionType+dumpsName, "6301",actionType);
CallShell cs3 =new CallShell();
String db22 = "bash "+dbDir + "/" + "startServer.sh" +" %s %s %s";
String db1b = String.format(db22, dbDir,"clusterl1-"+pjName+actionType+".rdb",Integer.valueOf(port));
cs3.runShell(db1b, port);
String runpy2 = "bash "+datasetPath + "/" + "launchPy.sh" +" %s %s %s %s %s %s";
String formatRunPy1a = String.format(runpy2,pythonPath +"/abstractPatchCluster.py", gumInput, datasetPath + "/cluster"+pjName+ actionType, port, datasetPath + "/cluster-2l"+pjName+ actionType,threshold);
cs3.runShell(formatRunPy1a);
String stopServer1a = "bash "+dbDir + "/" + "stopServer.sh" +" %s";
stopServer = String.format(stopServer1a,Integer.valueOf(port));
cs3.runShell(stopServer, port);
}
private static void level3(String port, String pythonPath, String datasetPath, String pjName, String actionType, String threshold, String dbDir, String dumpsName, String gumInput) throws Exception {
String stopServer;
MultiThreadTreeLoaderCluster3.calculatePairsOfClusters(datasetPath + "/cluster-2l"+pjName+ actionType, datasetPath,actionType);
MultiThreadTreeLoaderCluster3.mainCompare("6300", datasetPath+"/pairs-2l"+actionType, datasetPath + "/redisSingleImport.sh", dbDir, "clusterl2-"+pjName+actionType+".rdb", actionType+dumpsName, "6301",actionType);
CallShell cs5 =new CallShell();
String dba = "bash "+dbDir + "/" + "startServer.sh" +" %s %s %s";
String dbaa = String.format(dba, dbDir,"clusterl2-"+pjName+actionType+".rdb",Integer.valueOf(port));
cs5.runShell(dbaa, port);
String runpya = "bash "+datasetPath + "/" + "launchPy.sh" +" %s %s %s %s %s %s";
String formatRunPya = String.format(runpya,pythonPath +"/abstractPatchClusterLevel3.py", gumInput, datasetPath + "/cluster-3l"+pjName+ actionType, port, datasetPath + "/cluster-2l"+pjName+ actionType,threshold);
cs5.runShell(formatRunPya);
String stopServera = "bash "+dbDir + "/" + "stopServer.sh" +" %s";
stopServer = String.format(stopServera,Integer.valueOf(port));
cs5.runShell(stopServer, port);
return;
}
}
src/main/java/edu/lu/uni/serval/fixminer/PatternExtractor.java
+441
View File
@@ -0,0 +1,441 @@
package edu.lu.uni.serval.fixminer;
import edu.lu.uni.serval.FixPattern.utils.Checker;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalActionSet;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.JedisPoolConfig;
import java.io.*;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Base64;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.Stream;
/**
* Created by anilkoyuncu on 02/08/2018.
*/
public class PatternExtractor {
private static Logger log = LoggerFactory.getLogger(PatternExtractor.class);
public static void mainLaunch(String portInner,String numOfWorkers,String jobType,String port, String pythonPath, String datasetPath, String pjName, String actionType,String threshold) {
String dbDir;
String pairsPath;
String dumpsName;
String gumInput;
String gumOutput;
gumInput = datasetPath + "/" + pjName + "/";
gumOutput = datasetPath + "/EnhancedASTDiff" + pjName;
dbDir = datasetPath + "/redis";
pairsPath = datasetPath + "/pairsImport" + pjName;
dumpsName = "dumps-" + pjName + ".rdb";
try {
String IDLIST_PATH =pythonPath + "/defects4jpatterns.txt";
List<String> fixes = readIdList(IDLIST_PATH);
switch (jobType) {
case "EXTRACTPATTERN":
loadDB(gumOutput, portInner, dbDir, actionType+dumpsName,actionType,fixes);
break;
case "GETPATTERN":
getPattern(fixes,actionType);
}
} catch (Exception e) {
e.printStackTrace();
}
}
public static void loadDB(String inputPath,String portInner,String dbDir,String chunkName,String operation,List<String> fixes) throws Exception {
// String inputPath;
// String portInner;
// String serverWait;
// String dbDir;
// String chunkName;
// String numOfWorkers;
// if (args.length > 0) {
// inputPath = args[0];
// portInner = args[1];
// serverWait = args[2];
// chunkName = args[3];
// numOfWorkers = args[4];
// dbDir = args[5];
// } else {
// inputPath = "/Users/anilkoyuncu/bugStudy/dataset/GumTreeOutput2";
// portInner = "6399";
// serverWait = "10000";
// chunkName ="dumps.rdb";
// dbDir = "/Users/anilkoyuncu/bugStudy/dataset/redis";
// numOfWorkers = "1";
// }
// String parameters = String.format("\nInput path %s \nportInner %s \nserverWait %s \nchunkName %s \ndbDir %s \noperation %s", inputPath, portInner, serverWait, chunkName, dbDir, operation);
// log.info(parameters);
// CallShell cs = new CallShell();
// String cmd = "bash " + dbDir + "/" + "startServer.sh" + " %s %s %s";
// cmd = String.format(cmd, dbDir, chunkName, Integer.valueOf(portInner));
//// loadRedis(cmd,serverWait);
// cs.runShell(cmd, serverWait);
JedisPool outerPool = new JedisPool(poolConfig, "127.0.0.1",Integer.valueOf(6399),20000000);
String clusterPath = "/Users/anilkoyuncu/bugStudy/release/code/clusterallDataset"+operation+"/";
String savePath = "/Users/anilkoyuncu/bugStudy/release/dataset/dumps"+operation+"/";
File patternsF = new File(clusterPath);
File[] listOfPatterns = patternsF.listFiles();
Stream<File> patterns = Arrays.stream(listOfPatterns);
List<File> patternsL = patterns
.filter(x -> !x.getName().startsWith("."))
// .filter(x-> x.getName().endsWith(".git"))
.collect(Collectors.toList());
for (File pattern:patternsL) {
File folder = new File(clusterPath + pattern.getName());
File[] listOfFiles = folder.listFiles();
Stream<File> stream = Arrays.stream(listOfFiles);
List<File> patches = stream
.filter(x -> !x.getName().startsWith("."))
// .filter(x-> x.getName().endsWith(".git"))
.collect(Collectors.toList());
for (File patch : patches) {
String fn = patch.getName();
String[] split = fn.split("_");
String project = split[split.length -1];
List<String> list = new ArrayList<String>(Arrays.asList(split));
list.remove(list.size() - 1);
String joinFN = String.join("_", list);
fn = project + "/" + operation + "/" + joinFN;
String saveFN = pattern.getName() + "_"+ project + "_" + operation + "_" + joinFN;
Jedis inner = null;
String s = null;
try {
inner = outerPool.getResource();
s = inner.get(fn);
if (s != null) {
HierarchicalActionSet actionSet = (HierarchicalActionSet) fromString(s);
BufferedWriter writer = new BufferedWriter(new FileWriter(savePath+saveFN));
writer.write(toString(actionSet));
writer.close();
}else{
log.error(fn);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (inner != null) {
inner.close();
}
}
}
}
}
public static void getPattern(List<String> fixes,String operation){
String clusterPath = "/Users/anilkoyuncu/bugStudy/release/dataset/output/clusterallDatasetUPD/";
// String savePath = "/Users/anilkoyuncu/bugStudy/release/dataset/dumps/";
String savePath = "/Volumes/anil.koyuncu/dumps/";
File patternsF = new File(clusterPath);
File[] listOfPatterns = patternsF.listFiles();
Stream<File> patterns = Arrays.stream(listOfPatterns);
List<File> patternsL = patterns
.filter(x -> !x.getName().startsWith("."))
// .filter(x -> !x.getName().endsWith(".git"))
.collect(Collectors.toList());
for (File pattern:patternsL) {
File folder = new File(clusterPath + pattern.getName());
File[] listOfFiles = folder.listFiles();
Stream<File> stream = Arrays.stream(listOfFiles);
List<File> patches = stream
.filter(x -> !x.getName().startsWith("."))
.filter(x -> !x.getName().endsWith(".git"))
.collect(Collectors.toList());
for (File patch : patches) {
String fn = patch.getName();
String[] split = fn.split("_");
String project = split[split.length - 1];
List<String> list = new ArrayList<String>(Arrays.asList(split));
list.remove(list.size() - 1);
String joinFN = String.join("_", list);
fn = project + "/" + operation + "/" + joinFN;
String saveFN = pattern.getName() + "_" + project + "_" + operation + "_" + joinFN;
try{
String content = new String(Files.readAllBytes(Paths.get(savePath + saveFN)));
HierarchicalActionSet actionSet = (HierarchicalActionSet) fromString(content);
int astType = actionSet.getNode().getType();
if (Checker.isStatement(astType) || astType == 23 //FieldDeclaration
|| astType == 31 //MethodDeclaration
|| astType == 55) {//TypeDeclaration
System.out.println(actionSet);
// ITree actionTree = MultiThreadTreeLoader.getActionTree(actionSet, null, null);
// ITree simpliedTree = getSimpliedTree(actionSet);
// System.out.println(new TreeToString().toString(simpliedTree));
// System.out.println(new TreeToString().toString(actionTree));
System.out.println("======");
}
//
// ITree simpliedTree = getSimpliedTree(actionSet);
// simpliedTree.toString();
// ITree simpliedTree = getSimpliedTree(actionSet);
//
// List<String> oldTokens = new ArrayList<>();
// List<String> newTokens = new ArrayList<>();
// if(secondValue.startsWith("/2/")){
// log.info("newss");
// }
// // 41 return statement
// oldTokens = getNames(actionSet.getNode(),oldTokens);
//
// simpliedTree.toString();
}catch (Exception e){
e.printStackTrace();
}
}
System.out.println("============");
}
}
// public static ITree getSimpliedTree(HierarchicalActionSet actionSet) {
//
// ITree tree = null;
// Jedis inner = null;
// try {
//
// ITree parent = null;
// ITree children = null;
// TreeContext tc = new TreeContext();
// tree = getASTTree(actionSet, parent, children, tc);
//// tree.setParent(null);
// tc.validate();
//
// } catch (Exception e) {
// e.printStackTrace();
// }
// return tree;
// }
static final JedisPoolConfig poolConfig = buildPoolConfig();
private static String toString( Serializable o ) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream( baos );
oos.writeObject( o );
oos.close();
return Base64.getEncoder().encodeToString(baos.toByteArray());
}
/** Read the object from Base64 string. */
private static Object fromString( String s ) throws IOException ,
ClassNotFoundException {
byte[] data = Base64.getDecoder().decode(s);
ObjectInputStream ois = new ObjectInputStream(
new ByteArrayInputStream(data));
Object o = ois.readObject();
ois.close();
return o;
}
private static JedisPoolConfig buildPoolConfig() {
final JedisPoolConfig poolConfig = new JedisPoolConfig();
poolConfig.setMaxTotal(128);
poolConfig.setMaxIdle(128);
poolConfig.setMinIdle(16);
poolConfig.setTestOnBorrow(false);
poolConfig.setTestOnReturn(false);
poolConfig.setTestWhileIdle(true);
poolConfig.setMinEvictableIdleTimeMillis(Duration.ofMinutes(60).toMillis());
poolConfig.setTimeBetweenEvictionRunsMillis(Duration.ofHours(30).toMillis());
// poolConfig.setNumTestsPerEvictionRun(3);
poolConfig.setBlockWhenExhausted(true);
return poolConfig;
}
public static List<String> readIdList(String fileName) {
try {
BufferedReader reader = new BufferedReader(new FileReader(new File(fileName)));
// n = Rownum ; m = Colnum
List<String> fixCommits = new ArrayList<String>();
String line = reader.readLine();
int i = 0, j = 0;
while (line != null) {
String strArray[] = line.split("\t");
if (!line.trim().isEmpty()) {
// GitTravellerDefects4J.FixCommit test = new GitTravellerDefects4J.FixCommit(strArray[1],strArray[0]);
fixCommits.add(strArray[0]);
// for (String s : strArray) {
// if (!s.trim().isEmpty()) {
// FixCommit test = new FixCommit("","");
// array[i][j++] = s;
// }
// }
line = reader.readLine();
// i++;
// j = 0;
}
}
reader.close();
return fixCommits;
} catch (IOException ex) {
System.out.println("Problems..");
}
return null;
}
public static class ProjectLiteral {
private String name;
private String file;
private String packageName;
private String className;
private String extend;
private String[] StringLit;
private String[] NumLit;
private String[] memberReference;
private String[] methodInvocation;
// pj, file, p['packageName'], p['className'],
// p['extends'], p['StringLiteral'], p['NumberLiteral']
ProjectLiteral(){}
ProjectLiteral(String name, String file,String packageName, String className,String extend, String sLit, String nLit, String mr ,String mi) {
this.name = name;
this.file = file;
this.packageName = packageName;
this.className = className;
this.extend = extend;
this.StringLit = sLit.split(";");
this.NumLit = nLit.split(";");
this.memberReference = mr.split(";");
this.methodInvocation = nLit.split(";");
}
// projectLiterals.stream().filter(x->x.className.equals("XYTitleAnnotation")).collect(Collectors.toList())
}
public static void main(String[] args) throws IOException {
List<ProjectLiteral> projectLiterals = readProjectCSV("/Users/anilkoyuncu/bugStudy/code/python/chart.csv");
System.out.print("");
}
public static List<ProjectLiteral> readProjectCSV(String fileName) {
try {
BufferedReader reader = new BufferedReader(new FileReader(new File(fileName)));
// n = Rownum ; m = Colnum
List<ProjectLiteral> fixCommits = new ArrayList<ProjectLiteral>();
String line = reader.readLine();
int i = 0, j = 0;
while (line != null) {
String strArray[] = line.split("\t");
if (!line.trim().isEmpty()) {
log.info(String.valueOf(strArray.length));
ProjectLiteral a = new ProjectLiteral();
if (strArray.length == 10) {
a = new ProjectLiteral(strArray[1], strArray[2], strArray[3], strArray[4], strArray[5], strArray[6], strArray[7],strArray[8],strArray[9]);
}else if (strArray.length == 9) {
a = new ProjectLiteral(strArray[1], strArray[2], strArray[3], strArray[4], strArray[5], strArray[6], strArray[7],strArray[8],"");
}else if (strArray.length == 8) {
a = new ProjectLiteral(strArray[1], strArray[2], strArray[3], strArray[4], strArray[5], strArray[6], strArray[7],"","");
}else if(strArray.length == 7){
a = new ProjectLiteral(strArray[1], strArray[2], strArray[3], strArray[4], strArray[5], strArray[6], "","","");
}else if(strArray.length == 6){
a = new ProjectLiteral(strArray[1], strArray[2], strArray[3], strArray[4], strArray[5], "", "","","");
}else if(strArray.length == 5){
a = new ProjectLiteral(strArray[1], strArray[2], strArray[3], strArray[4], "", "", "","","");
}else if(strArray.length == 4){
a = new ProjectLiteral(strArray[1], strArray[2], strArray[3], "", "", "", "","","");
}
else{
log.error("error");
}
// GitTravellerDefects4J.FixCommit test = new GitTravellerDefects4J.FixCommit(strArray[1],strArray[0]);
fixCommits.add(a);
// for (String s : strArray) {
// if (!s.trim().isEmpty()) {
// FixCommit test = new FixCommit("","");
// array[i][j++] = s;
// }
// }
line = reader.readLine();
// i++;
// j = 0;
}
}
reader.close();
return fixCommits;
} catch (IOException ex) {
System.out.println("Problems..");
}
return null;
}
}
src/main/java/edu/lu/uni/serval/fixminer/TestHunkParserSingleFile.java
+239
View File
@@ -0,0 +1,239 @@
package edu.lu.uni.serval.fixminer;
import akka.actor.ActorRef;
import akka.actor.ActorSystem;
import edu.lu.uni.serval.MultipleThreadsParser.MessageFile;
import edu.lu.uni.serval.MultipleThreadsParser.ParseFixPatternActor;
import edu.lu.uni.serval.MultipleThreadsParser.WorkMessage;
import edu.lu.uni.serval.utils.FileHelper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.File;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.Stream;
public class TestHunkParserSingleFile {
private static Logger log = LoggerFactory.getLogger(TestHunkParserSingleFile.class);
public static void main(String[] args) {
// input data
// String rootPath = "/Users/anilkoyuncu/bugStudy";
String inputPath;
String outputPath;
if(args.length > 0){
inputPath = args[1];
outputPath = args[0];
}else{
// inputPath = "/Users/anilkoyuncu/bugStudy/dataset/GumTreeInputBug4";
inputPath = "/Users/anilkoyuncu/bugStudy/dataset/allDataset";
// outputPath = "/Users/anilkoyuncu/bugStudy/code/python/GumTreeOutput2/";
outputPath = "/Users/anilkoyuncu/bugStudy/dataset/GumTreeOutputSingle";
}
//5d9d60_76f5be_components#camel-jaxb#src#test#java#org#apache#camel#jaxb#FallbackTypeConverterShouldNotThrowExceptionTest.txt_1_CAMEL
File folder = new File(inputPath);
File[] listOfFiles = folder.listFiles();
Stream<File> stream = Arrays.stream(listOfFiles);
List<File> folders = stream
.filter(x -> !x.getName().startsWith("."))
.collect(Collectors.toList());
// List<File> targetList = new ArrayList<File>();
// for (File f:folders){
// for(File f1 :f.listFiles()){
// if (!f1.getName().startsWith(".")){
// targetList.add(f1);
// }
// }
// }
// List<String> pjList = Arrays.asList("DATAJPA","ZXing","PDE","SWS","SWT", "SWF", "COLLECTIONS", "JDT");
List<String> files = new ArrayList<String>();
// files.add("5d9d60_76f5be_components#camel-jaxb#src#test#java#org#apache#camel#jaxb#FallbackTypeConverterShouldNotThrowExceptionTest.java");
files.add("d6c5e5_9f96d9_hbase-server#src#main#java#org#apache#hadoop#hbase#master#RegionStates.java");
for(String f : files){
String pjName = "HBASE";
// for (File target : folders) {
// String pjName = target.getName();
// if (!pjList.contains(pjName)){
// continue;
// }
// final List<MessageFile> msgFiles = getMessageFiles(target.toString() + "/"); //"/Users/anilkoyuncu/bugStudy/code/python/GumTreeInput/Apache/CAMEL/"
MessageFile messageFile = getMessageFile(inputPath + "/" + pjName +"/", f);
List<MessageFile> msgFiles = new ArrayList<>();
msgFiles.add(messageFile);
String GUM_TREE_OUTPUT = outputPath + "/"+ pjName + "/";
final String editScriptsFilePath = GUM_TREE_OUTPUT + "editScripts.list";
final String patchesSourceCodeFilePath =GUM_TREE_OUTPUT + "patchSourceCode.list";
final String buggyTokensFilePath = GUM_TREE_OUTPUT + "tokens.list";
final String editScriptSizesFilePath = GUM_TREE_OUTPUT + "editScriptSizes.csv";
final String alarmTypesFilePath = GUM_TREE_OUTPUT + "alarmTypes.list";
FileHelper.createDirectory(GUM_TREE_OUTPUT + "/ActionSetDumps");
FileHelper.deleteDirectory(editScriptsFilePath);
FileHelper.deleteDirectory(patchesSourceCodeFilePath);
FileHelper.deleteDirectory(buggyTokensFilePath);
FileHelper.deleteDirectory(editScriptSizesFilePath);
FileHelper.deleteDirectory(alarmTypesFilePath);
StringBuilder astEditScripts = new StringBuilder();
StringBuilder tokens = new StringBuilder();
StringBuilder sizes = new StringBuilder();
StringBuilder patches = new StringBuilder();
StringBuilder alarmTypes = new StringBuilder();
// int a = 0;
ActorSystem system = null;
ActorRef parsingActor = null;
final WorkMessage msg = new WorkMessage(0, msgFiles);
try {
log.info("Akka begins...");
system = ActorSystem.create("Mining-FixPattern-System");
parsingActor = system.actorOf(ParseFixPatternActor.props(1, "dataset"), "mine-fix-pattern-actor");
parsingActor.tell(msg, ActorRef.noSender());
} catch (Exception e) {
system.shutdown();
e.printStackTrace();
}
// int counter = 0;
// for (MessageFile msgFile : msgFiles) {
// FixedViolationHunkParser parser = new FixedViolationHunkParser();
//
// final ExecutorService executor = Executors.newSingleThreadExecutor();
// // schedule the work
// final Future<?> future = executor.submit(new RunnableParser(msgFile.getPrevFile(),
// msgFile.getRevFile(), msgFile.getDiffEntryFile(), parser));
// try {
// // where we wait for task to complete
// future.get(Configuration.SECONDS_TO_WAIT, TimeUnit.SECONDS);
// String editScripts = parser.getAstEditScripts();
// if (!editScripts.equals("")) {
// astEditScripts.append(editScripts);
// tokens.append(parser.getTokensOfSourceCode());
// sizes.append(parser.getSizes());
// patches.append(parser.getPatchesSourceCode());
// alarmTypes.append(parser.getAlarmTypes());
//
// a++;
// if (a % 100 == 0) {
// FileHelper.outputToFile(editScriptsFilePath, astEditScripts, true);
// FileHelper.outputToFile(buggyTokensFilePath, tokens, true);
// FileHelper.outputToFile(editScriptSizesFilePath, sizes, true);
// FileHelper.outputToFile(patchesSourceCodeFilePath, patches, true);
// FileHelper.outputToFile(alarmTypesFilePath, alarmTypes, true);
// astEditScripts.setLength(0);
// tokens.setLength(0);
// sizes.setLength(0);
// patches.setLength(0);
// alarmTypes.setLength(0);
// System.out.println("Finish of parsing " + a + " files......");
// }
// }
// } catch (TimeoutException e) {
// err.println("task timed out");
// future.cancel(true /* mayInterruptIfRunning */);
// } catch (InterruptedException e) {
// err.println("task interrupted");
// } catch (ExecutionException e) {
// err.println("task aborted");
// } finally {
// executor.shutdownNow();
// }
// }
FileHelper.outputToFile(editScriptsFilePath, astEditScripts, true);
FileHelper.outputToFile(buggyTokensFilePath, tokens, true);
FileHelper.outputToFile(editScriptSizesFilePath, sizes, true);
FileHelper.outputToFile(patchesSourceCodeFilePath, patches, true);
FileHelper.outputToFile(alarmTypesFilePath, alarmTypes, true);
astEditScripts.setLength(0);
tokens.setLength(0);
sizes.setLength(0);
patches.setLength(0);
alarmTypes.setLength(0);
// classifyByAlarmTypes();
}
}
private static List<MessageFile> getMessageFiles(String gumTreeInput) {
String inputPath = gumTreeInput; // prevFiles revFiles diffentryFile positionsFile
File revFilesPath = new File(inputPath + "revFiles/");
File[] revFiles = revFilesPath.listFiles(); // project folders
List<MessageFile> msgFiles = new ArrayList<>();
// gumTreeInput = /Volumes/data/bugStudy_backup/dataset/GumTreeInputBug4/AMQP/
// fileName = 01534a_df5570_spring-rabbit#src#test#java#org#springframework#amqp#rabbit#listener#LocallyTransactedTests.java
if (revFiles.length >= 0) {
for (File revFile : revFiles) {
// if (revFile.getName().endsWith(".java")) {
String fileName = revFile.getName();
File prevFile = new File(gumTreeInput + "prevFiles/prev_" + fileName);// previous file
fileName = fileName.replace(".java", ".txt");
File diffentryFile = new File(gumTreeInput + "DiffEntries/" + fileName); // DiffEntry file
File positionFile = new File(gumTreeInput + "positions/" + fileName); // position file
MessageFile msgFile = new MessageFile(revFile, prevFile, diffentryFile);
msgFile.setPositionFile(positionFile);
msgFiles.add(msgFile);
// }
}
return msgFiles;
}
else{
return null;
}
}
private static MessageFile getMessageFile(String gumTreeInput, String fileName) {
// String inputPath = gumTreeInput; // prevFiles revFiles diffentryFile positionsFile
// File revFilesPath = new File(inputPath + "revFiles/");
// File[] revFiles = revFilesPath.listFiles(); // project folders
// List<MessageFile> msgFiles = new ArrayList<>();
// gumTreeInput = /Volumes/data/bugStudy_backup/dataset/GumTreeInputBug4/AMQP/
// fileName = 01534a_df5570_spring-rabbit#src#test#java#org#springframework#amqp#rabbit#listener#LocallyTransactedTests.java
// if (revFiles.length >= 0) {
// for (File revFile : revFiles) {
// if (revFile.getName().endsWith(".java")) {
// String fileName = revFile.getName();
File revFile = new File(gumTreeInput + "revFiles/"+fileName);
File prevFile = new File(gumTreeInput + "prevFiles/prev_" + fileName);// previous file
fileName = fileName.replace(".java", ".txt");
File diffentryFile = new File(gumTreeInput + "DiffEntries/" + fileName); // DiffEntry file
// File positionFile = new File(gumTreeInput + "positions/" + fileName); // position file
MessageFile msgFile = new MessageFile(revFile, prevFile, diffentryFile);
return msgFile;
// msgFile.setPositionFile(positionFile);
// msgFiles.add(msgFile);
// }
// }
//
// return msgFiles;
// }
// else{
// return null;
// }
}
}
src/main/java/edu/lu/uni/serval/fixminer/TestTreeLoader.java
+230
View File
@@ -0,0 +1,230 @@
package edu.lu.uni.serval.fixminer;
import com.github.gumtreediff.actions.ActionGenerator;
import com.github.gumtreediff.actions.model.Action;
import com.github.gumtreediff.matchers.Matcher;
import com.github.gumtreediff.matchers.Matchers;
import com.github.gumtreediff.tree.ITree;
import java.io.*;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.Stream;
/**
* Created by anilkoyuncu on 17/03/2018.
*/
public class TestTreeLoader {
public static void main(String[] args) {
// String inputPath = "/Users/anilkoyuncu/bugStudy/dataset/GumTreeOutput/ENTESB/ASTDumps";
String inputPath = "/Users/anilkoyuncu/bugStudy/dataset/GumTreeOutput2/";
File folder = new File(inputPath);
File[] listOfFiles = folder.listFiles();
Stream<File> stream = Arrays.stream(listOfFiles);
List<File> pjs = stream
.filter(x -> !x.getName().startsWith("."))
.collect(Collectors.toList());
List<File> fileToCompare = new ArrayList<>();
for(File pj:pjs){
File[] files = pj.listFiles(new FilenameFilter() {
public boolean accept(File dir, String name) {
return name.startsWith("ASTDumps");
}
});
Collections.addAll(fileToCompare, files[0].listFiles());
}
System.out.println("a");
// compareAll(fileToCompare);
memoryFriendlyCompare(fileToCompare);
}
public static ITree getSimpliedTree(String fn) {
ITree tree = null;
try {
FileInputStream fi = new FileInputStream(new File(fn));
ObjectInputStream oi = new ObjectInputStream(fi);
tree = (ITree) oi.readObject();
oi.close();
fi.close();
} catch (FileNotFoundException e) {
System.out.println("File not found");
} catch (IOException e) {
System.out.println("Error initializing stream");
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
tree.setLabel("");
tree.setParent(null);
List<ITree> descendants = tree.getDescendants();
for (ITree descendant : descendants) {
descendant.setLabel("");
}
return tree;
}
public static void memoryFriendlyCompare(List<File> folders){
List<String> treesFileNames = new ArrayList<>();
// HashMap<Integer, String> hmap = new HashMap<Integer, String>();
for (File target : folders) {
// hmap.put(folders.indexOf(target), target.toString());
treesFileNames.add(target.toString());
}
for (int i = 0; i < treesFileNames.size(); i++) {
for (int j = i + 1; j < treesFileNames.size(); j++) {
// compare list.get(i) and list.get(j)
try {
BufferedWriter writer = new BufferedWriter(new FileWriter("output2.txt", true));
ITree oldTree = getSimpliedTree(treesFileNames.get(i));
ITree newTree = getSimpliedTree(treesFileNames.get(j));
Matcher m = Matchers.getInstance().getMatcher(oldTree, newTree);
m.match();
ActionGenerator ag = new ActionGenerator(oldTree, newTree, m.getMappings());
ag.generate();
List<Action> actions = ag.getActions();
writer.write(String.valueOf(i));
writer.write("\t");
writer.write(String.valueOf(j));
writer.write("\t");
writer.write(String.format("%1.2f", m.chawatheSimilarity(oldTree, newTree)));
writer.write("\t");
writer.write(String.format("%1.2f", m.diceSimilarity(oldTree, newTree)));
writer.write("\t");
writer.write(String.format("%1.2f", m.jaccardSimilarity(oldTree, newTree)));
writer.write("\t");
writer.write(String.valueOf(actions.size()));
writer.write("\t");
writer.write(treesFileNames.get(i));
writer.write("\t");
writer.write(treesFileNames.get(j));
writer.write("\n");
writer.close();
} catch (FileNotFoundException e) {
System.out.println("File not found");
} catch (IOException e) {
System.out.println("Error initializing stream");
}
}
}
}
// public static void compareAll(List<File> folders){
// List<ITree> trees = new ArrayList<>();
// HashMap<Integer, String> hmap = new HashMap<Integer, String>();
// for (File target : folders) {
//
// try {
// FileInputStream fi = new FileInputStream(new File(target.toString()));
// ObjectInputStream oi = new ObjectInputStream(fi);
// ITree pr1 = (ITree) oi.readObject();
// oi.close();
// fi.close();
// trees.add(pr1);
// hmap.put(folders.indexOf(target), target.toString());
//
// } catch (FileNotFoundException e) {
// System.out.println("File not found");
// } catch (IOException e) {
// System.out.println("Error initializing stream");
// } catch (ClassNotFoundException e) {
// // TODO Auto-generated catch block
// e.printStackTrace();
// }
// }
//
// for (ITree tree : trees) {
//// SimplifyTree simplifyTree = new SimplifyTree();
//// simplifyTree.canonicalizeSourceCodeTree(tree);
// tree.setLabel("");
// tree.setParent(null);
// List<ITree> descendants = tree.getDescendants();
// for (ITree descendant : descendants) {
// descendant.setLabel("");
// }
//
// }
// System.out.println("a");
//
// try {
//
//
//
// for (int i = 0; i < trees.size(); i++) {
// for (int j = i + 1; j < trees.size(); j++) {
// // compare list.get(i) and list.get(j)
// BufferedWriter writer = new BufferedWriter(new FileWriter("output.txt",true));
// ITree oldTree = trees.get(i);
//
// ITree newTree = trees.get(j);
//
// Matcher m = Matchers.getInstance().getMatcher(oldTree, newTree);
// m.match();
//
// ActionGenerator ag = new ActionGenerator(oldTree, newTree, m.getMappings());
// ag.generate();
// List<Action> actions = ag.getActions();
// writer.write(String.valueOf(i));
// writer.write("\t");
// writer.write(String.valueOf(j));
// writer.write("\t");
//
// writer.write(String.format("%1.2f", m.chawatheSimilarity(oldTree, newTree)));
// writer.write("\t");
// writer.write(String.format("%1.2f", m.diceSimilarity(oldTree, newTree)));
// writer.write("\t");
// writer.write(String.format("%1.2f", m.jaccardSimilarity(oldTree, newTree)));
// writer.write("\t");
// writer.write(String.valueOf(actions.size()));
// writer.write("\t");
// writer.write(hmap.get(i));
// writer.write("\t");
// writer.write(hmap.get(j));
// writer.write("\n");
//
// writer.close();
// }
// }
//
//
// } catch (FileNotFoundException e) {
// System.out.println("File not found");
// } catch (IOException e) {
// System.out.println("Error initializing stream");
//
// }
//
//
//// if (actions.size() > 1) {
//// Matcher m1 = Matchers.getInstance().getMatcher(actions.get(0).getNode(), actions.get(0).getNode());
//// m1.match();
//// Set<Mapping> mappingSet1 = m1.getMappingSet();
////
//// }
// }
}
src/main/java/edu/lu/uni/serval/fixminer/cluster/AkkaTreeLoader.java
+114
View File
@@ -0,0 +1,114 @@
package edu.lu.uni.serval.fixminer.cluster;
import edu.lu.uni.serval.FixPattern.utils.PoolBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.regex.Pattern;
import static edu.lu.uni.serval.fixminer.cluster.akka.AkkaTreeParser.akkaCompare;
/**
* Created by anilkoyuncu on 19/03/2018.
*/
public class AkkaTreeLoader {
private static Logger log = LoggerFactory.getLogger(AkkaTreeLoader.class);
public static void main(String portInner, String dbDir, String chunkName, String port, String dumpsName, String pairsPath, String numOfWorkers, String cursor,String chunk) throws Exception {
String parameters = String.format("\nportInner %s \nchunkName %s \ndbDir %s \ndumpsName %s",portInner,chunkName,dbDir,dumpsName);
log.info(parameters);
CallShell cs = new CallShell();
String cmd = "bash "+dbDir + "/" + "startServer.sh" +" %s %s %s";
String cmd1 = String.format(cmd, dbDir,dumpsName,Integer.valueOf(port));
//
cs.runShell(cmd1,port);
String cmdInner = "bash "+dbDir + "/" + "startServer.sh" +" %s %s %s";
String cmd2 = String.format(cmdInner, dbDir,chunkName,Integer.valueOf(portInner));
log.info(cmd1);
log.info(cmd2);
//
cs.runShell(cmd2, portInner);
JedisPool outerPool = new JedisPool(PoolBuilder.getPoolConfig(), "127.0.0.1",Integer.valueOf(port),20000000);
JedisPool innerPool = new JedisPool(PoolBuilder.getPoolConfig(), "127.0.0.1",Integer.valueOf(portInner),20000000);
String pairsIndexFile = pairsPath + "/"+ chunkName;
//0.txt.rdb
pairsIndexFile = pairsIndexFile.replace(chunk+".rdb",".index");
Pattern pattern = Pattern.compile(",");
String csvFile = pairsIndexFile;
try {
try (BufferedReader in = new BufferedReader(new FileReader(csvFile));){
Map<String,String> namefreq = in
.lines()
.map(x -> pattern.split(x))
.collect(HashMap::new, (map, x) ->
map.put(x[0], x[1]),
Map::putAll);
Jedis inner = null;
try {
inner = outerPool.getResource();
for (Map.Entry<String, String> entry : namefreq.entrySet()) {
String key = entry.getKey();
String value = entry.getValue();
inner.select(1);
inner.set(key,value);
}
}finally {
if (inner != null) {
inner.close();
}
}
}
} catch (IOException e) {
e.printStackTrace();
}
akkaCompare(innerPool,outerPool,numOfWorkers,cursor);
}
}
src/main/java/edu/lu/uni/serval/fixminer/cluster/CalculatePairs.java
+136
View File
@@ -0,0 +1,136 @@
package edu.lu.uni.serval.fixminer.cluster;
import edu.lu.uni.serval.FixPattern.utils.PoolBuilder;
import edu.lu.uni.serval.utils.FileHelper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.ScanParams;
import redis.clients.jedis.ScanResult;
import java.io.*;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.util.List;
/**
* Created by anilkoyuncu on 05/04/2018.
*/
public class CalculatePairs {
private static Logger log = LoggerFactory.getLogger(CalculatePairs.class);
public static void main(String dbDir,String chunkName,String port,String outputPath,String pjName) throws Exception {
String parameters = String.format("\nport %s \nchunkName %s \ndbDir %s",port,chunkName,dbDir);
log.info(parameters);
CallShell cs =new CallShell();
String cmd = "bash "+dbDir + "/" + "startServer.sh" +" %s %s %s";
cmd = String.format(cmd, dbDir,chunkName,Integer.valueOf(port));
cs.runShell(cmd, port);
FileHelper.createDirectory(outputPath);
JedisPool outerPool = new JedisPool(PoolBuilder.getPoolConfig(), "127.0.0.1",Integer.valueOf(port),20000000);
ScanResult<String> scan;
try (Jedis outer = outerPool.getResource()) {
while (outer.ping()== "PONG"){
log.info("wait");
}
ScanParams sc = new ScanParams();
//150000000
sc.count(1500000000);
sc.match("*");
scan = outer.scan("0", sc);
int size = scan.getResult().size();
log.info("Scanning " + String.valueOf(size));
}
List<String> result = scan.getResult();
byte [] buf = new byte[0];
String line = null;
try {
// FileOutputStream fos = new FileOutputStream(outputPath + "/" +pjName+".csv");
// DataOutputStream outStream = new DataOutputStream(new BufferedOutputStream(fos));
//
FileOutputStream fosIndex = new FileOutputStream(outputPath + "/" +pjName+".index");
DataOutputStream outStreamIndex = new DataOutputStream(new BufferedOutputStream(fosIndex));
//
//
//
//
// for (int i = 0; i < result.size(); i++) {
// line = String.valueOf(i) +"," + result.get(i)+"\n";
// outStreamIndex.write(line.getBytes());
//
// for (int j = i + 1; j < result.size(); j++) {
//
//
//
// line = String.valueOf(i) +"," + String.valueOf(j)+"\n"; // + "," + result.get(i) + "," + result.get(j)+"\n";
// outStream.write(line.getBytes());
//
// }
// }
// outStream.close();
// outStreamIndex.close();
int fileCounter = 0;
FileChannel rwChannel = new RandomAccessFile(outputPath + "/" +pjName +String.valueOf(fileCounter)+".txt", "rw").getChannel();
int maxSize = 500*1000000;
ByteBuffer wrBuf = rwChannel.map(FileChannel.MapMode.READ_WRITE, 0, maxSize);
for (int i = 0; i < result.size(); i++) {
line = String.valueOf(i) +"," + result.get(i)+"\n";
outStreamIndex.write(line.getBytes());
for (int j = i + 1; j < result.size(); j++) {
// line = String.valueOf(i) +"\t" + String.valueOf(j)+"\n"; // + "\t" + result.get(i) + "\t" + result.get(j)+"\n";
line = String.valueOf(i) +"," + String.valueOf(j)+"\n"; // + "," + result.get(i) + "," + result.get(j)+"\n";
buf = line.getBytes();
if(wrBuf.remaining() > 500) {
wrBuf.put(buf);
}else{
log.info("Next pair dump");
fileCounter++;
rwChannel = new RandomAccessFile(outputPath+"/" +pjName+String.valueOf(fileCounter)+".txt", "rw").getChannel();
wrBuf = rwChannel.map(FileChannel.MapMode.READ_WRITE, 0, maxSize);
}
}
}
outStreamIndex.close();
rwChannel.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}catch (java.nio.BufferOverflowException e) {
log.error(line);
log.error(String.valueOf(buf.length));
e.printStackTrace();
}
// String stopServer = "bash "+dbDir + "/" + "stopServer.sh" +" %s";
// String stopServer2 = String.format(stopServer,Integer.valueOf(port));
//// loadRedis(stopServer2,serverWait);
// cs.runShell(stopServer2,serverWait);
log.info("Done pairs");
}
}
src/main/java/edu/lu/uni/serval/fixminer/cluster/CallShell.java
+111
View File
@@ -0,0 +1,111 @@
package edu.lu.uni.serval.fixminer.cluster;
/**
* Created by anilkoyuncu on 17/04/2018.
*/
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.concurrent.TimeUnit;
public class CallShell {
public void runShell(String command) throws Exception {
Process process = Runtime.getRuntime().exec(command);
BufferedReader reader = new BufferedReader(new InputStreamReader(
process.getInputStream()));
String s;
while ((s = reader.readLine()) != null) {
System.out.println("Script output: " + s);
}
}
public static void runShell(String command, String port) throws Exception {
Process process = Runtime.getRuntime().exec(command);
BufferedReader reader = new BufferedReader(new InputStreamReader(
process.getInputStream()));
String s;
while ((s = reader.readLine()) != null) {
System.out.println("Script output: " + s);
}
// Thread.sleep(Integer.valueOf(serverWait));
String cmd = "redis-cli -p %s ping";
String cmd1 = String.format(cmd,Integer.valueOf(port));
runPing(cmd1);
}
public static void runPing(String command) throws Exception {
try{
StringBuffer output = new StringBuffer();
Process process = Runtime.getRuntime().exec(command);
BufferedReader reader = new BufferedReader(new InputStreamReader(
process.getInputStream()));
String s;
BufferedReader stdError = new BufferedReader(new
InputStreamReader(process.getErrorStream()));
// read the output from the command
// System.out.println("Here is the standard output of the command:\n");
// while ((s = reader.readLine()) != null && !s.equals("PONG")) {
// System.out.println(s);
// }
// while(true){
s = reader.readLine();
if(s !=null && s.equals("PONG")){
System.out.println(s);
}else{
String e;
if((e = stdError.readLine()) == null) {
TimeUnit.MINUTES.sleep(1);
runPing(command);
}else{
System.out.println(e);
}
}
// System.out.println(s);
// }
// read any errors from the attempted command
// System.out.println("Here is the standard error of the command (if any):\n");
while ((s = stdError.readLine()) != null) {
System.out.print(s);
}
// System.exit(0);
}
catch (IOException e) {
System.out.println("exception happened - here's what I know: ");
e.printStackTrace();
System.exit(-1);
}
}
public static void main(String[] args) throws Exception {
// runPing("redis-cli -p 6380 ping");
runShell("bash /Users/anilkoyuncu/bugStudy/release/code/redis/startServer.sh /Users/anilkoyuncu/bugStudy/release/code/redis Defects4JALL0.txt.rdb 6380","6380");
}
}
src/main/java/edu/lu/uni/serval/fixminer/cluster/Compare.java
+101
View File
@@ -0,0 +1,101 @@
package edu.lu.uni.serval.fixminer.cluster;
import com.github.gumtreediff.actions.ActionGenerator;
import com.github.gumtreediff.actions.model.Action;
import com.github.gumtreediff.matchers.Matcher;
import com.github.gumtreediff.matchers.Matchers;
import com.github.gumtreediff.tree.ITree;
import edu.lu.uni.serval.FixPattern.utils.EDiff;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import java.util.List;
import java.util.Map;
/**
* Created by anilkoyuncu on 03/04/2018.
*/
public class Compare {
private Logger log = LoggerFactory.getLogger(Compare.class);
public void coreCompare(String name, JedisPool innerPool, JedisPool outerPool) {
Map<String, String> resultMap;
Jedis jedis = null;
ITree oldTree = null;
ITree newTree = null;
try {
jedis = innerPool.getResource();
// resultMap = jedis.hgetAll(name);
String[] split = name.split("_");
String i = split[1];
String j = split[2];
// String firstValue = resultMap.get("0");
// String secondValue = resultMap.get("1");
oldTree = EDiff.getSimpliedTree(i,outerPool);
newTree = EDiff.getSimpliedTree(j,outerPool);
Matcher m = Matchers.getInstance().getMatcher(oldTree, newTree);
m.match();
ActionGenerator ag = new ActionGenerator(oldTree, newTree, m.getMappings());
ag.generate();
List<Action> actions = ag.getActions();
double chawatheSimilarity1 = m.chawatheSimilarity(oldTree, newTree);
String chawatheSimilarity = String.format("%1.2f", chawatheSimilarity1);
double diceSimilarity1 = m.diceSimilarity(oldTree, newTree);
String diceSimilarity = String.format("%1.2f", diceSimilarity1);
double jaccardSimilarity1 = m.jaccardSimilarity(oldTree, newTree);
String jaccardSimilarity = String.format("%1.2f", jaccardSimilarity1);
String editDistance = String.valueOf(actions.size());
String result = i + "," + j + "," + chawatheSimilarity + "," + diceSimilarity + "," + jaccardSimilarity + "," + editDistance;
if (((Double) chawatheSimilarity1).equals(1.0) || ((Double) diceSimilarity1).equals(1.0)
|| ((Double) jaccardSimilarity1).equals(1.0) || actions.size() == 0) {
String matchKey = "match_" + (String.valueOf(i)) + "_" + String.valueOf(j);
// log.info(matchKey);
jedis.select(1);
jedis.set(matchKey, result);
}
jedis.select(0);
jedis.del("pair_" + (String.valueOf(i)) + "_" + String.valueOf(j));
} catch (Exception e) {
log.error(e.toString() + " {}", name);
// e.printStackTrace();
}finally {
if (jedis != null) {
jedis.close();
}
}
}
}
src/main/java/edu/lu/uni/serval/fixminer/cluster/ImportPairs2DB.java
+63
View File
@@ -0,0 +1,63 @@
package edu.lu.uni.serval.fixminer.cluster;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.File;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.Stream;
/**
* Created by anilkoyuncu on 05/04/2018.
*/
public class ImportPairs2DB {
private static Logger log = LoggerFactory.getLogger(ImportPairs2DB.class);
public static void main(String csvInputPath,String portInner,String dbDir,String datasetPath) throws Exception {
String parameters = String.format("\nInput path %s \nportInner %s \ndbDir %s",csvInputPath,portInner,dbDir);
log.info(parameters);
File folder = new File(csvInputPath);
File[] subFolders = folder.listFiles();
Stream<File> stream = Arrays.stream(subFolders);
List<File> pjs = stream
.filter(x -> x.getName().endsWith(".txt"))
.collect(Collectors.toList());
Integer portInt = Integer.valueOf(portInner);
for (File pj : pjs) {
String cmd = "bash "+dbDir + "/" + "startServer.sh" +" %s %s %s";
cmd = String.format(cmd, dbDir,pj.getName() +".rdb", portInt);
log.info(cmd);
CallShell cs = new CallShell();
cs.runShell(cmd, portInner);
cmd = "bash "+datasetPath + "/redisSingleImport.sh" +" %s %s";
cmd = String.format(cmd, pj.getPath(), portInt);
log.info(cmd);
cs.runShell(cmd);
String stopServer = "bash "+dbDir + "/" + "stopServer.sh" +" %s";
String stopServer2 = String.format(stopServer,portInt);
cs.runShell(stopServer2, portInner);
portInt++;
}
log.info(parameters);
}
}
src/main/java/edu/lu/uni/serval/fixminer/cluster/MultiThreadTreeLoader.java
+600
View File
@@ -0,0 +1,600 @@
package edu.lu.uni.serval.fixminer.cluster;
import com.github.gumtreediff.actions.ActionGenerator;
import com.github.gumtreediff.actions.model.Action;
import com.github.gumtreediff.matchers.Matcher;
import com.github.gumtreediff.matchers.Matchers;
import com.github.gumtreediff.tree.ITree;
import com.github.gumtreediff.tree.Tree;
import edu.lu.uni.serval.FixPattern.utils.ASTNodeMap;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalActionSet;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.*;
import java.io.*;
import java.nio.file.Files;
import java.time.Duration;
import java.util.*;
import java.util.concurrent.Executors;
import java.util.function.Consumer;
import java.util.stream.Collectors;
import java.util.stream.Stream;
/**
* Created by anilkoyuncu on 19/03/2018.
*/
public class MultiThreadTreeLoader {
private static class StreamGobbler implements Runnable {
private InputStream inputStream;
private Consumer<String> consumer;
public StreamGobbler(InputStream inputStream, Consumer<String> consumer) {
this.inputStream = inputStream;
this.consumer = consumer;
}
@Override
public void run() {
new BufferedReader(new InputStreamReader(inputStream)).lines()
.forEach(consumer);
}
}
private static Logger log = LoggerFactory.getLogger(MultiThreadTreeLoader.class);
public static void loadRedis(String cmd,String serverWait){
Process process;
try {
// String comd = String.format(cmd, f.getAbsoluteFile());
process = Runtime.getRuntime()
.exec(cmd);
StreamGobbler streamGobbler =
new StreamGobbler(process.getInputStream(), System.out::println);
Executors.newSingleThreadExecutor().submit(streamGobbler);
// int exitCode = process.waitFor();
// assert exitCode == 0;
Thread.sleep(Integer.valueOf(serverWait));
} catch (IOException e) {
e.printStackTrace();
}
catch (InterruptedException e) {
e.printStackTrace();
}
log.info("Load done");
}
public static void main(String[] args) {
String inputPath;
String outputPath;
String port;
String portInner;
String pairsCSVPath;
String importScript;
String pairsCompletedPath;
String serverWait;
String option;
String dbDir;
String chunkName;
if (args.length > 0) {
inputPath = args[0];
port = args[1];
portInner = args[2];
serverWait = args[3];
option = args[4];
chunkName = args[5];
dbDir = args[6];
// pairsCSVPath = args[3];
// importScript = args[4];
// pairsCompletedPath = args[3];
} else {
inputPath = "/Users/anilkoyuncu/bugStudy/dataset/GumTreeOutput2";
outputPath = "/Users/anilkoyuncu/bugStudy/dataset/";
port = "6379";
portInner = "6380";
serverWait = "10000";
option = "COMP";
pairsCSVPath = "/Users/anilkoyuncu/bugStudy/dataset/pairs/test";
importScript = "/Users/anilkoyuncu/bugStudy/dataset/pairs/test2.sh";
pairsCompletedPath = "/Users/anilkoyuncu/bugStudy/dataset/pairs_completed";
chunkName ="chunk5.rdb";
dbDir = "/Users/anilkoyuncu/bugStudy/dataset/redis";
}
if (option.equals("CALC")) {
calculatePairs(inputPath, port);
log.info("Calculate pairs done");
}else {
comparePairs(port, inputPath, portInner, serverWait,chunkName,dbDir);
}
}
public static void comparePairs(String port,String inputPath, String innerPort,String serverWait,String chunkName, String dbDir){
// String cmd;
// cmd = "bash " + importScript +" %s";
List<String> dir;
List<String> path;
String orgDbname;
File files = new File(dbDir);
File[] listFiles = files.listFiles();
Stream<File> stream = Arrays.stream(listFiles);
List<File> folders = stream
.filter(x -> x.getName().equals(chunkName))
.collect(Collectors.toList());
for (File folder : folders) {
String cmd = "bash "+dbDir + "/" + "startServer.sh" +" %s %s %s";
cmd = String.format(cmd, dbDir,folder.getName(),Integer.valueOf(innerPort));
loadRedis(cmd,serverWait);
JedisPool pool = new JedisPool(new JedisPoolConfig(), "127.0.0.1", Integer.valueOf(innerPort), 20000000);
ScanResult<String> scan;
try (Jedis inner = pool.getResource()) {
while (inner.ping()== "PONG"){
log.info("wait");
}
ScanParams sc = new ScanParams();
sc.count(150000000);
sc.match("pair_*");
scan = inner.scan("0", sc);
int size = scan.getResult().size();
log.info("Scanning " + String.valueOf(size));
}
scan.getResult().parallelStream()
.forEach(m -> coreCompare(m, inputPath, innerPort));
String stopServer = "bash "+dbDir + "/" + "stopServer.sh" +" %s";
stopServer = String.format(stopServer,Integer.valueOf(innerPort));
loadRedis(stopServer,serverWait);
}
}
private static void coreCompare(String name , String inputPath, String innerPort) {
JedisPool pool = new JedisPool(new JedisPoolConfig(), "127.0.0.1", Integer.valueOf(innerPort), 20000000);
Map<String, String> resultMap;
try (Jedis jedis = pool.getResource()) {
resultMap = jedis.hgetAll(name);
}
String[] split = name.split("_");
String i = split[1];
String j = split[2];
String firstValue = resultMap.get("0");
String secondValue = resultMap.get("1");
String[] firstValueSplit = firstValue.split("GumTreeOutput2");
String[] secondValueSplit = secondValue.split("GumTreeOutput2");
if (firstValueSplit.length == 1) {
firstValue = inputPath + firstValueSplit[0];
} else {
firstValue = inputPath + firstValueSplit[1];
}
if (secondValueSplit.length == 1) {
secondValue = inputPath + secondValueSplit[0];
} else {
secondValue = inputPath + secondValueSplit[1];
}
try {
ITree oldTree = getSimpliedTree(firstValue);
ITree newTree = getSimpliedTree(secondValue);
Matcher m = Matchers.getInstance().getMatcher(oldTree, newTree);
m.match();
ActionGenerator ag = new ActionGenerator(oldTree, newTree, m.getMappings());
ag.generate();
List<Action> actions = ag.getActions();
double chawatheSimilarity1 = m.chawatheSimilarity(oldTree, newTree);
String chawatheSimilarity = String.format("%1.2f", chawatheSimilarity1);
double diceSimilarity1 = m.diceSimilarity(oldTree, newTree);
String diceSimilarity = String.format("%1.2f", diceSimilarity1);
double jaccardSimilarity1 = m.jaccardSimilarity(oldTree, newTree);
String jaccardSimilarity = String.format("%1.2f", jaccardSimilarity1);
String editDistance = String.valueOf(actions.size());
String result = resultMap.get("0") + "," + resultMap.get("1") + "," + chawatheSimilarity + "," + diceSimilarity + "," + jaccardSimilarity + "," + editDistance;
if (((Double) chawatheSimilarity1).equals(1.0) || ((Double) diceSimilarity1).equals(1.0)
|| ((Double) jaccardSimilarity1).equals(1.0) || actions.size() == 0) {
String matchKey = "match_" + (String.valueOf(i)) + "_" + String.valueOf(j);
try (Jedis jedis = pool.getResource()) {
jedis.select(1);
jedis.set(matchKey, result);
}
}
try (Jedis jedis = pool.getResource()) {
jedis.del("pair_"+ (String.valueOf(i)) + "_" + String.valueOf(j));
}
}catch (Exception e){
log.warn(e.toString() + " {}",(name));
}
}
public static void calculatePairs(String inputPath,String port) {
File folder = new File(inputPath);
File[] listOfFiles = folder.listFiles();
Stream<File> stream = Arrays.stream(listOfFiles);
List<File> pjs = stream
.filter(x -> !x.getName().startsWith("."))
.collect(Collectors.toList());
List<File> fileToCompare = new ArrayList<>();
for (File pj : pjs) {
File[] files = pj.listFiles(new FilenameFilter() {
public boolean accept(File dir, String name) {
return name.startsWith("ActionSetDumps");
}
});
Collections.addAll(fileToCompare, files[0].listFiles());
}
System.out.println("a");
// compareAll(fileToCompare);
readMessageFiles(fileToCompare,port);
}
// public static void processMessages(String inputPath, String outputPath) {
// File folder = new File(outputPath + "pairs_splitted/");
// File[] listOfFiles = folder.listFiles();
// Stream<File> stream = Arrays.stream(listOfFiles);
// List<File> pjs = stream
// .filter(x -> !x.getName().startsWith("."))
// .collect(Collectors.toList());
// FileHelper.createDirectory(outputPath + "comparison_splitted/");
// pjs.parallelStream()
// .forEach(m -> coreLoop(m, outputPath,inputPath));
// }
public static ITree getSimpliedTree(String fn) {
HierarchicalActionSet actionSet = null;
try {
FileInputStream fi = new FileInputStream(new File(fn));
ObjectInputStream oi = new ObjectInputStream(fi);
actionSet = (HierarchicalActionSet) oi.readObject();
oi.close();
fi.close();
} catch (FileNotFoundException e) {
log.error("File not found");
e.printStackTrace();
} catch (IOException e) {
log.error("Error initializing stream");
e.printStackTrace();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
ITree parent = null;
ITree children =null;
ITree tree = getASTTree(actionSet, parent, children);
tree.setParent(null);
return tree;
}
public static <T, E> List<T> getKeysByValue(Map<T, E> map, E value) {
return map.entrySet()
.stream()
.filter(entry -> Objects.equals(entry.getValue(), value))
.map(Map.Entry::getKey)
.collect(Collectors.toList());
}
public static ITree getASTTree(HierarchicalActionSet actionSet, ITree parent, ITree children){
int newType = 0;
String astNodeType = actionSet.getAstNodeType();
List<Integer> keysByValue = getKeysByValue(ASTNodeMap.map, astNodeType);
if(keysByValue.size() != 1){
log.error("Birden cok astnodemapmapping");
}
newType = keysByValue.get(0);
if(actionSet.getParent() == null){
//root
parent = new Tree(newType,"");
}else{
children = new Tree(newType,"");
parent.addChild(children);
}
List<HierarchicalActionSet> subActions = actionSet.getSubActions();
if (subActions.size() != 0){
for (HierarchicalActionSet subAction : subActions) {
if(actionSet.getParent() == null){
children = parent;
}
getASTTree(subAction,children,null);
}
}
return parent;
}
// public static ITree getActionTree(HierarchicalActionSet actionSet, ITree parent, ITree children){
//
// int newType = 0;
//
// Action action = actionSet.getAction();
// if (action instanceof Update){
// newType = 101;
// }else if(action instanceof Insert){
// newType =100;
// }else if(action instanceof Move){
// newType = 102;
// }else if(action instanceof Delete){
// newType=103;
// }else{
// new Exception("unknow action");
// }
// if(actionSet.getParent() == null){
// //root
//
// parent = new Tree(newType,"");
// }else{
// children = new Tree(newType,"");
// parent.addChild(children);
// }
// List<HierarchicalActionSet> subActions = actionSet.getSubActions();
// if (subActions.size() != 0){
// for (HierarchicalActionSet subAction : subActions) {
//
// if(actionSet.getParent() == null){
// children = parent;
// }
// getActionTree(subAction,children,null);
//
// }
//
//
// }
// return parent;
// }
private static void coreLoop(File mes, String outputPath,String inputPath) {
try {
log.info("Starting in coreLoop");
BufferedReader br = null;
String sCurrentLine = null;
BufferedWriter writer = new BufferedWriter(new FileWriter(outputPath + "comparison_splitted/" + "output_" + mes.getName()));
br = new BufferedReader(
new FileReader(mes));
while ((sCurrentLine = br.readLine()) != null) {
String currentLine = sCurrentLine;
String[] split = currentLine.split("\t");
String i = split[0];
String j = split[1];
String firstValue = split[2];
String secondValue = split[3];
firstValue = inputPath + firstValue.split("GumTreeOutput2")[1];
secondValue = inputPath + secondValue.split("GumTreeOutput2")[1];
ITree oldTree = getSimpliedTree(firstValue);
ITree newTree = getSimpliedTree(secondValue);
Matcher m = Matchers.getInstance().getMatcher(oldTree, newTree);
m.match();
ActionGenerator ag = new ActionGenerator(oldTree, newTree, m.getMappings());
ag.generate();
List<Action> actions = ag.getActions();
writer.write(String.valueOf(i));
writer.write("\t");
writer.write(String.valueOf(j));
writer.write("\t");
writer.write(String.format("%1.2f", m.chawatheSimilarity(oldTree, newTree)));
writer.write("\t");
writer.write(String.format("%1.2f", m.diceSimilarity(oldTree, newTree)));
writer.write("\t");
writer.write(String.format("%1.2f", m.jaccardSimilarity(oldTree, newTree)));
writer.write("\t");
writer.write(String.valueOf(actions.size()));
writer.write("\t");
writer.write(firstValue);
writer.write("\t");
writer.write(secondValue);
writer.write("\n");
}
writer.close();
} catch (FileNotFoundException e) {
log.error("File not found");
e.printStackTrace();
} catch (IOException e) {
log.error("Error initializing stream");
e.printStackTrace();
}
log.info("Completed output_" + mes.getName());
}
private static void readMessageFiles(List<File> folders,String port) {
List<String> treesFileNames = new ArrayList<>();
for (File target : folders) {
treesFileNames.add(target.toString());
}
// FileHelper.createDirectory(outputPath + "pairs/");
log.info("Calculating pairs");
// treesFileNames = treesFileNames.subList(0,100);
byte [] buf = new byte[0];
String line = null;
// FileChannel rwChannel = new RandomAccessFile(outputPath + "pairs/" +"textfile.txt", "rw").getChannel();
// ByteBuffer wrBuf = rwChannel.map(FileChannel.MapMode.READ_WRITE, 0, Integer.MAX_VALUE);
int fileCounter = 0;
JedisPool jedisPool = new JedisPool(poolConfig, "127.0.0.1",Integer.valueOf(port),20000000);
try (Jedis jedis = jedisPool.getResource()) {
List<String> dir = null;
List<String> path = null;
for (int i = 0; i < treesFileNames.size(); i++) {
for (int j = i + 1; j < treesFileNames.size(); j++) {
// do operations with jedis resource
String key = "pair_" + String.valueOf(i) + "_" + String.valueOf(j);
// String value = treesFileNames.get(i).split("GumTreeOutput2")[1] +","+treesFileNames.get(j).split("GumTreeOutput2")[1];
// jedis.set(key,value);
jedis.hset(key,"0",treesFileNames.get(i).split("GumTreeOutput2")[1]);
jedis.hset(key,"1",treesFileNames.get(j).split("GumTreeOutput2")[1]);
//10000000
if(Integer.compare(jedis.dbSize().intValue(),10000000) == 0){
dir = jedis.configGet("dir");
path = jedis.configGet("dbfilename");
File dbPath = new File(dir.get(1)+"/"+path.get(1));
File savePath = new File(dir.get(1)+"/"+"chunk"+String.valueOf(fileCounter)+ ".rdb");
try {
jedis.save();
while (jedis.ping()== "PONG"){
log.info("wait");
}
Files.copy(dbPath.toPath(),savePath.toPath());
} catch (IOException e) {
e.printStackTrace();
}
fileCounter++;
jedis.flushDB();
}
}
}
jedis.save();
fileCounter++;
File dbPath = new File(dir.get(1)+"/"+path.get(1));
File savePath = new File(dir.get(1)+"/"+"chunk"+String.valueOf(fileCounter)+ ".rdb");
try {
while (jedis.ping()== "PONG"){
log.info("wait");
}
Files.copy(dbPath.toPath(),savePath.toPath());
} catch (IOException e) {
e.printStackTrace();
}
jedis.flushDB();
}
log.info("Done pairs");
}
static final JedisPoolConfig poolConfig = buildPoolConfig();
private static JedisPoolConfig buildPoolConfig() {
final JedisPoolConfig poolConfig = new JedisPoolConfig();
poolConfig.setMaxTotal(128);
poolConfig.setMaxIdle(128);
poolConfig.setMinIdle(16);
poolConfig.setTestOnBorrow(true);
poolConfig.setTestOnReturn(true);
poolConfig.setTestWhileIdle(true);
poolConfig.setMinEvictableIdleTimeMillis(Duration.ofMinutes(60).toMillis());
poolConfig.setTimeBetweenEvictionRunsMillis(Duration.ofHours(30).toMillis());
poolConfig.setNumTestsPerEvictionRun(3);
poolConfig.setBlockWhenExhausted(true);
return poolConfig;
}
// return msgFiles;
}
src/main/java/edu/lu/uni/serval/fixminer/cluster/MultiThreadTreeLoaderCluster.java
+342
View File
@@ -0,0 +1,342 @@
package edu.lu.uni.serval.fixminer.cluster;
import com.github.gumtreediff.actions.ActionGenerator;
import com.github.gumtreediff.actions.model.Action;
import com.github.gumtreediff.matchers.Matcher;
import com.github.gumtreediff.matchers.Matchers;
import com.github.gumtreediff.tree.ITree;
import com.github.gumtreediff.tree.TreeContext;
import edu.lu.uni.serval.FixPattern.utils.EDiff;
import edu.lu.uni.serval.FixPattern.utils.EDiffHelper;
import edu.lu.uni.serval.FixPattern.utils.PoolBuilder;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalActionSet;
import edu.lu.uni.serval.utils.FileHelper;
import org.javatuples.Pair;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.ScanParams;
import redis.clients.jedis.ScanResult;
import java.io.*;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
import java.util.stream.Stream;
/**
* Created by anilkoyuncu on 19/03/2018.
*/
public class MultiThreadTreeLoaderCluster {
private static Logger log = LoggerFactory.getLogger(MultiThreadTreeLoaderCluster.class);
public static void mainCompare(String port,String pairsCSVPath,String importScript,String dbDir,String chunkName,String dumpName,String portInner,String type) throws Exception {
CallShell cs = new CallShell();
String cmd1 = "bash "+dbDir + "/" + "startServer.sh" +" %s %s %s";
cmd1 = String.format(cmd1, dbDir,chunkName,Integer.valueOf(portInner));
cs.runShell(cmd1, port);
String cmd2 = "bash "+dbDir + "/" + "startServer.sh" +" %s %s %s";
cmd2 = String.format(cmd2, dbDir,dumpName,Integer.valueOf(port));
cs.runShell(cmd2, port);
String cmd3;
cmd3 = "bash " + importScript +" %s %s";
JedisPool jedisPool = new JedisPool(PoolBuilder.getPoolConfig(), "127.0.0.1",Integer.valueOf(portInner),20000000);
JedisPool outerPool = new JedisPool(PoolBuilder.getPoolConfig(), "127.0.0.1",Integer.valueOf(port),20000000);
File folder = new File(pairsCSVPath);
File[] listOfFiles = folder.listFiles();
Stream<File> stream = Arrays.stream(listOfFiles);
List<File> folders = stream
.filter(x -> !x.getName().startsWith("."))
.collect(Collectors.toList());
for (File f:folders){
// if(f.getName().startsWith("cluster0")) {
try (Jedis jedis = jedisPool.getResource()) {
// do operations with jedis resource
ScanParams sc = new ScanParams();
sc.count(150000000);
sc.match("pair_[0-9]*");
log.info("Scanning");
ScanResult<String> scan = jedis.scan("0", sc);
int size = scan.getResult().size();
if (size == 0) {
String comd = String.format(cmd3,f.getPath(),portInner);
// loadRedis(comd);
cs.runShell(comd);
scan = jedis.scan("0", sc);
size = scan.getResult().size();
}
log.info("Scanned " + String.valueOf(size));
String clusterName = f.getName().replaceAll("[^0-9]+", "");
//76
scan.getResult().parallelStream()
.forEach(m -> coreCompare(m, jedisPool, clusterName,outerPool,type));
jedis.save();
}
// }
}
String stopServer = "bash "+dbDir + "/" + "stopServer.sh" +" %s";
String stopServer1 = String.format(stopServer,Integer.valueOf(portInner));
cs.runShell(stopServer1, port);
String stopServer2 = String.format(stopServer,Integer.valueOf(port));
cs.runShell(stopServer2, port);
}
public static Pair<ITree,String> getTree(String firstValue, JedisPool outerPool,String type){
ITree tree = null;
Jedis inner = null;
String[] split2 = firstValue.split("/");
String fullFileName = split2[split2.length-1];
String[] split = fullFileName.split(".txt_");
String pureFileName = split[0];
String[] splitPJ = split[1].split("_");
String project = splitPJ[1];
String actionSetPosition = splitPJ[0];
try {
inner = outerPool.getResource();
String filename = project + "/"+type+"/" + pureFileName + ".txt_" + actionSetPosition;
String si= inner.get(filename);
HierarchicalActionSet actionSet = (HierarchicalActionSet) EDiffHelper.fromString(si);
ITree parent = null;
ITree children =null;
TreeContext tc = new TreeContext();
tree = EDiff.getActionTree(actionSet, parent, children,tc);
tree.setParent(null);
tc.validate();
} catch (IOException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
}finally {
if (inner != null) {
inner.close();
}
}
Pair<ITree, String> pair = new Pair<>(tree,project);
return pair;
}
private static void coreCompare(String name , JedisPool jedisPool,String clusterName,JedisPool outerPool,String type) {
try (Jedis jedis = jedisPool.getResource()) {
Map<String, String> resultMap = jedis.hgetAll(name);
resultMap.get("0");
String[] split = name.split("_");
String i = null;
String j =null;
try {
i = split[1];
j = split[2];
}
catch (Exception e){
e.printStackTrace();
}
String firstValue = resultMap.get("0");
String secondValue = resultMap.get("1");
try {
Pair<ITree, String> oldPair = getTree(firstValue, outerPool,type);
Pair<ITree, String> newPair = getTree(secondValue, outerPool,type);
ITree oldTree = oldPair.getValue0();
ITree newTree = newPair.getValue0();
String oldProject = oldPair.getValue1();
String newProject = newPair.getValue1();
Matcher m = Matchers.getInstance().getMatcher(oldTree, newTree);
m.match();
ActionGenerator ag = new ActionGenerator(oldTree, newTree, m.getMappings());
ag.generate();
List<Action> actions = ag.getActions();
String resultKey = "result_" + (String.valueOf(i)) + "_" + String.valueOf(j);
double chawatheSimilarity1 = m.chawatheSimilarity(oldTree, newTree);
String chawatheSimilarity = String.format("%1.2f", chawatheSimilarity1);
double diceSimilarity1 = m.diceSimilarity(oldTree, newTree);
String diceSimilarity = String.format("%1.2f", diceSimilarity1);
double jaccardSimilarity1 = m.jaccardSimilarity(oldTree, newTree);
String jaccardSimilarity = String.format("%1.2f", jaccardSimilarity1);
String editDistance = String.valueOf(actions.size());
// jedis.select(1);
String result = resultMap.get("0") + "," + oldProject +"," + resultMap.get("1") + "," +newProject+ "," + chawatheSimilarity + "," + diceSimilarity + "," + jaccardSimilarity + "," + editDistance;
// jedis.set(resultKey, result);
if (((Double) chawatheSimilarity1).equals(1.0) || ((Double) diceSimilarity1).equals(1.0)
|| ((Double) jaccardSimilarity1).equals(1.0) || actions.size() == 0) {
String matchKey = "match-"+clusterName+"_" + (String.valueOf(i)) + "_" + String.valueOf(j);
jedis.select(1);
jedis.set(matchKey, result);
}
jedis.select(0);
String pairKey = "pair_" + (String.valueOf(i)) + "_" + String.valueOf(j);
jedis.del(pairKey);
// log.info("Completed " + resultKey);
}catch (Exception e){
log.warn(e.toString() + " {}",(name));
}
}
}
/*
pairs of each cluster
*/
public static void calculatePairsOfClusters(String inputPath, String outputPath,String type) {
File folder = new File(inputPath);
File[] listOfFiles = folder.listFiles();
Stream<File> stream = Arrays.stream(listOfFiles);
List<File> pjs = stream
.filter(x -> !x.getName().startsWith("."))
.filter(x-> x.isDirectory())
.collect(Collectors.toList());
FileHelper.createDirectory(outputPath + "/pairs"+type);
for (File pj : pjs) {
File[] files = pj.listFiles();
List<File> fileList = Arrays.asList(files);
readMessageFilesCluster(fileList, outputPath,inputPath,pj.getName(),type);
}
}
private static void readMessageFilesCluster(List<File> folders, String outputPath,String inputPath,String cluster,String type) {
List<String> treesFileNames = new ArrayList<>();
for (File target : folders) {
treesFileNames.add(target.toString());
}
log.info("Calculating pairs");
// treesFileNames = treesFileNames.subList(0,100);
String filename = "cluster" + cluster;
byte [] buf = new byte[0];
String line = null;
try {
FileOutputStream fos = new FileOutputStream(outputPath + "/pairs"+type+"/" +filename+".csv");
DataOutputStream outStream = new DataOutputStream(new BufferedOutputStream(fos));
for (int i = 0; i < treesFileNames.size(); i++) {
for (int j = i + 1; j < treesFileNames.size(); j++) {
line = String.valueOf(i) +"," + String.valueOf(j) + "," + treesFileNames.get(i).replace(inputPath,"") + "," + treesFileNames.get(j).replace(inputPath,"")+"\n";
outStream.write(line.getBytes());
}
}
outStream.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}catch (java.nio.BufferOverflowException e) {
log.error(line);
log.error(String.valueOf(buf.length));
e.printStackTrace();
}
log.info("Done pairs");
}
// return msgFiles;
}
src/main/java/edu/lu/uni/serval/fixminer/cluster/MultiThreadTreeLoaderCluster3.java
+620
View File
@@ -0,0 +1,620 @@
package edu.lu.uni.serval.fixminer.cluster;
import com.github.gumtreediff.tree.ITree;
import com.github.gumtreediff.tree.TreeContext;
import edu.lu.uni.serval.FixPattern.utils.EDiffHelper;
import edu.lu.uni.serval.FixPattern.utils.PoolBuilder;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalActionSet;
import edu.lu.uni.serval.utils.FileHelper;
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.text.similarity.JaroWinklerDistance;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.ScanParams;
import redis.clients.jedis.ScanResult;
import java.io.*;
import java.util.*;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import java.util.stream.Stream;
/**
* Created by anilkoyuncu on 19/03/2018.
*/
public class MultiThreadTreeLoaderCluster3 {
private static Logger log = LoggerFactory.getLogger(MultiThreadTreeLoaderCluster3.class);
public static void mainCompare(String port,String pairsCSVPath,String importScript,String dbDir,String chunkName,String dumpName,String portInner,String type) throws Exception {
CallShell cs = new CallShell();
String cmd1 = "bash "+dbDir + "/" + "startServer.sh" +" %s %s %s";
cmd1 = String.format(cmd1, dbDir,chunkName,Integer.valueOf(portInner));
cs.runShell(cmd1, port);
String cmd2 = "bash "+dbDir + "/" + "startServer.sh" +" %s %s %s";
cmd2 = String.format(cmd2, dbDir,dumpName,Integer.valueOf(port));
cs.runShell(cmd2, port);
String cmd3;
cmd3 = "bash " + importScript +" %s %s";
JedisPool jedisPool = new JedisPool(PoolBuilder.getPoolConfig(), "127.0.0.1",Integer.valueOf(portInner),20000000);
JedisPool outerPool = new JedisPool(PoolBuilder.getPoolConfig(), "127.0.0.1",Integer.valueOf(port),20000000);
File folder = new File(pairsCSVPath);
File[] listOfFiles = folder.listFiles();
Stream<File> stream = Arrays.stream(listOfFiles);
List<File> folders = stream
.filter(x -> !x.getName().startsWith("."))
.collect(Collectors.toList());
for (File f:folders){
//36_0,119_0,4_0
// if(f.getName().startsWith("cluster1_0")) {
try (Jedis jedis = jedisPool.getResource()) {
// do operations with jedis resource
ScanParams sc = new ScanParams();
sc.count(150000000);
sc.match("pair_[0-9]*");
log.info("Scanning");
ScanResult<String> scan = jedis.scan("0", sc);
int size = scan.getResult().size();
if (size == 0) {
String comd = String.format(cmd3, f.getPath(), portInner);
cs.runShell(comd);
scan = jedis.scan("0", sc);
size = scan.getResult().size();
}
log.info("Scanned " + String.valueOf(size));
String clusterName = f.getName().split("\\.")[0].replace("cluster", "");
scan.getResult().parallelStream()
.forEach(m -> coreCompare(m, jedisPool, clusterName, outerPool,type));
jedis.save();
}
}
String stopServer = "bash "+dbDir + "/" + "stopServer.sh" +" %s";
String stopServer1 = String.format(stopServer,Integer.valueOf(portInner));
cs.runShell(stopServer1, port);
String stopServer2 = String.format(stopServer,Integer.valueOf(port));
cs.runShell(stopServer2, port);
}
public static ITree getTree(String firstValue, JedisPool outerPool,String type){
// String gumTreeInput = "/Volumes/data/bugStudy_backup/dataset/GumTreeInputBug4/";
String[] split2 = firstValue.split("/");
String filename = split2[split2.length-1];
// String filename = split2[3];
String[] split1= filename.split(".txt_");
String s = split1[0];
String[] splitPJ = split1[1].split("_");
String project = splitPJ[1];
String actionSetPosition = splitPJ[0];
Integer asp = Integer.valueOf(actionSetPosition);
// if (asp > 1){
// return null;
// }
ITree tree = null;
Jedis inner = null;
try {
inner = outerPool.getResource();
String fn = project + "/"+type+"/" + s + ".txt_" + actionSetPosition;
String si= inner.get(fn);
HierarchicalActionSet actionSet = (HierarchicalActionSet) EDiffHelper.fromString(si);
ITree parent = null;
ITree children =null;
TreeContext tc = new TreeContext();
tree = EDiffHelper.getASTTree(actionSet, parent, children,tc);
// tree.setParent(null);
tc.validate();
} catch (IOException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
}finally {
if (inner != null) {
inner.close();
}
}
return tree;
}
public static List<String> getNames(ITree oldTree, List<String> oldTokens){
List<ITree> descendants = oldTree.getDescendants();
descendants.add(0,oldTree);
boolean upd=false;
for (ITree oldDescendant : descendants) {
boolean addToken = false;
int type = oldDescendant.getType();
String sType = String.valueOf(type);
if((sType.equals("42") || oldDescendant.getChildren().size() ==0)||
(sType.equals("32") && oldDescendant.getHeight() == 1 && oldDescendant.getChildren().size() == 1) ||
(sType.equals("59") && oldDescendant.getHeight() == 1 && oldDescendant.getChildren().size() == 1) ||
(sType.equals("43") && oldDescendant.getHeight() == 0 && oldDescendant.getChildren().size() == 0) ||
(sType.equals("53") )|| //&& oldDescendant.getHeight() == 1 && oldDescendant.getChildren().size() == 1) ||
(sType.equals("7") && oldDescendant.getHeight() == 1 && oldDescendant.getChildren().size() == 1) ||
(sType.equals("41") && oldDescendant.getHeight() == 1 && oldDescendant.getChildren().size() == 1) ||
(sType.equals("40") && oldDescendant.getHeight() == 1 && oldDescendant.getChildren().size() == 1)
){
// int depth = oldDescendant.getChildren().size();
String label = oldDescendant.getLabel();
// if(sType.equals("32") && oldDescendant.getHeight() == 1 && oldDescendant.getChildren().size() == 1){
// log.info("");
// }
label = label.split("@AT@")[0];
String[] split = label.split(" "+sType);
String[] split2 = split[1].split("@");
List<String> m = new ArrayList<String>();
if(label.startsWith("UPD")){
upd = true;
java.util.regex.Matcher matcher;
if(sType.equals("53")){
String timeRegex = ".*@@(ClassInstanceCreation:new [a-zA-Z0-9]+).*@TO@\\s(ClassInstanceCreation:new [a-zA-Z0-9]+).*";
Pattern pattern = Pattern.compile(timeRegex, Pattern.DOTALL);
matcher= pattern.matcher(split[1]);
}else {
String timeRegex = "@@(.*)@TO@(.*)";
Pattern pattern = Pattern.compile(timeRegex, Pattern.DOTALL);
matcher = pattern.matcher(split[1]);
}
if (matcher.matches()) {
String hours = matcher.group(1);
String to = matcher.group(2);
if(sType.equals("31")){
String commonPrefix = StringUtils.getCommonPrefix(hours.trim(), to.trim());
if(commonPrefix.isEmpty()){
log.info("PREFIX EMPTY");
}else {
String s = hours.trim().replace(commonPrefix, "");
String s1 = to.trim().replace(commonPrefix, "");
String[] split1 = s.split(",");
String[] split3 = s1.split(",");
Set<String> set = new TreeSet<>();
for (String s2 : split1) {
if(!s2.isEmpty()) {
set.add(s2.trim());
}
}
for (String s2 : split3) {
if(!s2.isEmpty()) {
set.add(s2.trim());
}
}
List<String> list = set.stream().collect(Collectors.toList());
m.addAll(list);
addToken = true;
}
}else if(sType.equals("7")){//assignment
m.add(hours.trim().split("=")[0]);
m.add(to.trim().split("=")[0]);
addToken = true;
}else if(sType.equals("43") && oldDescendant.getParent() != null && oldDescendant.getParent().getParent() != null){// catch clause
if(oldDescendant.getParent().getType() == 44 && oldDescendant.getParent().getParent().getType() == 12){
m.add(hours.trim());
m.add(to.trim());
addToken = true;
}
}else if(sType.equals("59") && oldDescendant.getChildren().size()==1 && oldDescendant.getHeight()==1 ){
if( oldDescendant.getChild(0).getType() == 34 || oldDescendant.getChild(0).getType() == 45) {
m.add(hours.trim().split("=")[0]);
m.add(to.trim().split("=")[0]);
addToken = true;
}
}else if(sType.equals("42") && oldDescendant.getParent() != null && oldDescendant.getParent().getType() == 27 ){//infix varuable change
m.add(hours.trim());
m.add(to.trim());
addToken = true;
}else if(sType.equals("40") && oldDescendant.getChildren().size()==1 && oldDescendant.getHeight()==1 ){//qualified name
m.add(hours.trim());
m.add(to.trim());
addToken = true;
}else if(sType.equals("-1") && oldDescendant.getChildren().size()==0 && oldDescendant.getHeight()==0 ){//operator
m.add(hours.trim());
m.add(to.trim());
addToken = true;
}else if(sType.equals("83") && oldDescendant.getChildren().size()==0 && oldDescendant.getHeight()==0 ){//modifier
m.add(hours.trim());
m.add(to.trim());
addToken = true;
}else if(sType.equals("39") && oldDescendant.getChildren().size()==0 && oldDescendant.getHeight()==0 ){//primitive type
m.add(hours.trim());
m.add(to.trim());
addToken = true;
}else{
m.add(hours.trim());
m.add(to.trim());
}
}
}else if(label.startsWith("INS") && upd == false){
String timeRegex = "@@(.*)@TO@(.*)";
Pattern pattern = Pattern.compile(timeRegex, Pattern.DOTALL);
java.util.regex.Matcher matcher = pattern.matcher(split[1]);
if (matcher.matches()) {
String hours = matcher.group(1);
if (hours.startsWith("MethodName:")) {
String methodName = hours.split(":")[1];
oldTokens.add(methodName);
// }else if(sType.equals("34")){
// oldTokens.add("NumberLiteral");
}else {
oldTokens.add(hours.trim());
}
}
}else if(label.startsWith("DEL") && upd == false){
String timeRegex = "@@(.*)";
Pattern pattern = Pattern.compile(timeRegex, Pattern.DOTALL);
java.util.regex.Matcher matcher = pattern.matcher(split[1]);
if (matcher.matches()) {
String hours = matcher.group(1);
if (hours.startsWith("MethodName:")){
String methodName = hours.split(":")[1];
oldTokens.add(methodName);
}else {
oldTokens.add(hours.trim());
}
}
}else if(label.startsWith("MOV") && upd == false){
String timeRegex = "@@(.*)@TO@(.*)";
Pattern pattern = Pattern.compile(timeRegex, Pattern.DOTALL);
java.util.regex.Matcher matcher = pattern.matcher(split[1]);
if (matcher.matches()) {
String hours = matcher.group(1);
if (hours.startsWith("MethodName:")){
String methodName = hours.split(":")[1];
oldTokens.add(methodName);
}else {
oldTokens.add(hours.trim());
}
}
}
boolean alreadyAddParentMethodName = false;
for (String s : m) {
// if(s.isEmpty()){
// continue;
// }
//TODO remove 44
if(s.startsWith("SimpleName:") || s.startsWith("Name:")){
String methodName = s.split(":")[1];
oldTokens.add(methodName);
}else if (s.startsWith("MethodName:")){
String methodName = s.split(":")[1];
ITree parent = oldDescendant.getParent();
// if(parent!= null && parent.getType() == 32 && !alreadyAddParentMethodName){ //parent is method invocation statement
// String parentLabel = parent.getLabel();
// String[] pns = parentLabel.split("\\." + methodName);
// if(pns.length > 1) {
// String parentName = pns[0];
// String[] parentNameSplit = parentName.split("@@");
// if (parentNameSplit.length > 1) {
// String parentMethodName = parentNameSplit[1];
// String s1 = parentMethodName.split("@TO@")[0];
// oldTokens.add(s1.trim());
// alreadyAddParentMethodName = true;
// }
// }
//
// }
oldTokens.add(methodName);
}else if( sType.equals("59") || sType.equals("43")|| sType.equals("53") || sType.equals("7") || sType.equals("27") || sType.equals("83") || sType.equals("44") ||sType.equals("78") || sType.equals("41") || addToken){
// if(sType.equals("27") || sType.equals("83") || sType.equals("44")){
// String parentLabel = oldDescendant.getParent().getLabel();
// List<String> parentM = new ArrayList<String>();
// if(parentLabel.startsWith("UPD")) {
// upd = true;
// String timeRegex = "@@(.*)@TO@(.*)";
// Pattern pattern = Pattern.compile(timeRegex, Pattern.DOTALL);
// java.util.regex.Matcher matcher = pattern.matcher(split[1]);
//
// if (matcher.matches()) {
// String hours = matcher.group(1);
// String to = matcher.group(2);
// parentM.add(hours.trim());
// parentM.add(to.trim());
//
// }
// }
//
//
// oldTokens.add(s);
// }else
if(sType.equals("53") || sType.equals("78") || addToken){//sType.equals("41") ||
oldTokens.add(s);
}
// else {
// String s1 = s.split("=")[0];
// oldTokens.add(s1);
// }
}
// else
// if(oldTokens.size() < 2){
// oldTokens.add(s);
// }
}
}
}
// if (oldTokens.size() == 0 ) {// && (oldTree.getType() != 41 && oldTree.getType() != 21 && oldTree.getType() !=17 && oldTree.getType()!=60 && oldTree.getType() != 46)){
// log.info("dur bakalim nereye!???");
// }
return oldTokens;
}
private static void coreCompare(String name , JedisPool jedisPool,String clusterName,JedisPool outerPool,String type) {
try (Jedis jedis = jedisPool.getResource()) {
Map<String, String> resultMap = jedis.hgetAll(name);
String[] split = name.split("_");
String i = null;
String j =null;
try {
i = split[1];
j = split[2];
}
catch (Exception e){
e.printStackTrace();
}
String firstValue = resultMap.get("0");
String secondValue = resultMap.get("1");
try {
ITree oldTree = getTree(firstValue,outerPool,type);
ITree newTree = getTree(secondValue,outerPool,type);
if(oldTree == null || newTree == null) {
jedis.select(0);
String pairKey = "pair_" + (String.valueOf(i)) + "_" + String.valueOf(j);
jedis.del(pairKey);
return;
}
List<String> oldTokens = new ArrayList<>();
List<String> newTokens = new ArrayList<>();
oldTokens = getNames(oldTree,oldTokens);
newTokens = getNames(newTree,newTokens);
CharSequence[] oldSequences = oldTokens.toArray(new CharSequence[oldTokens.size()]);
CharSequence[] newSequences = newTokens.toArray(new CharSequence[newTokens.size()]);
JaroWinklerDistance jwd = new JaroWinklerDistance();
// LevenshteinDistance ld = new LevenshteinDistance();
Double overallSimi = Double.valueOf(0);
if(oldSequences.length > 0 && (oldSequences.length == newSequences.length)){
for (int idx = 0; idx < newSequences.length; idx++) {
Double simi = jwd.apply(newSequences[idx], oldSequences[idx]);
overallSimi = simi + overallSimi;
}
overallSimi = overallSimi / oldSequences.length;
}else{
overallSimi = Double.valueOf(0);
// if(oldSequences.length != 0) {
// log.info("ERROR");
// }
}
int retval = Double.compare(overallSimi, Double.valueOf(0.8));
if(retval >= 0){
String matchKey = "match-"+clusterName+"_" + (String.valueOf(i)) + "_" + String.valueOf(j);
String result = firstValue + "," + secondValue + ","+String.join(",", oldTokens);
jedis.select(1);
jedis.set(matchKey, result);
}
jedis.select(0);
String pairKey = "pair_" + (String.valueOf(i)) + "_" + String.valueOf(j);
jedis.del(pairKey);
}catch (Exception e){
log.warn(e.toString() + " {}",(name));
}
}
}
/*
pairs of each cluster
*/
public static void calculatePairsOfClusters(String inputPath, String outputPath,String type) {
File folder = new File(inputPath);
File[] listOfFiles = folder.listFiles();
Stream<File> stream = Arrays.stream(listOfFiles);
List<File> pjs = stream
.filter(x -> !x.getName().startsWith("."))
.filter(x->x.isDirectory())
.collect(Collectors.toList());
FileHelper.createDirectory(outputPath + "/pairs-2l"+type);
for (File pj : pjs) {
File[] files = pj.listFiles();
List<File> fileList = Arrays.asList(files);
for (File cluster:fileList) {
if (cluster.getName().startsWith(".")){
continue;
}
File[] clusterFiles = cluster.listFiles();
List<File> clusterFilesL = Arrays.asList(clusterFiles);
readMessageFilesCluster(clusterFilesL, outputPath, inputPath, pj.getName(), cluster.getName(),type);
}
}
}
private static void readMessageFilesCluster(List<File> folders, String outputPath,String inputPath,String cluster, String subCluster,String type) {
List<String> treesFileNames = new ArrayList<>();
for (File target : folders) {
treesFileNames.add(target.toString());
}
log.info("Calculating pairs");
// treesFileNames = treesFileNames.subList(0,100);
String filename = "cluster" + cluster + "_" + subCluster;
byte [] buf = new byte[0];
String line = null;
try {
// FileChannel rwChannel = new RandomAccessFile(outputPath + "pairs-2l/" +filename+".txt", "rw").getChannel();
// ByteBuffer wrBuf = rwChannel.map(FileChannel.MapMode.READ_WRITE, 0, 1000*treesFileNames.size()*treesFileNames.size());
// int fileCounter = 0;
FileOutputStream fos = new FileOutputStream(outputPath + "/pairs-2l"+type+"/" +filename+".csv");
DataOutputStream outStream = new DataOutputStream(new BufferedOutputStream(fos));
for (int i = 0; i < treesFileNames.size(); i++) {
for (int j = i + 1; j < treesFileNames.size(); j++) {
line = String.valueOf(i) +"," + String.valueOf(j) + "," + treesFileNames.get(i).replace(inputPath,"") + "," + treesFileNames.get(j).replace(inputPath,"")+"\n";
outStream.write(line.getBytes());
// buf = line.getBytes();
// if(wrBuf.remaining() > 500) {
// wrBuf.put(buf);
// }else{
// log.info("Next pair dump");
// fileCounter++;
// rwChannel = new RandomAccessFile(outputPath+"pairs/" +filename+String.valueOf(fileCounter)+".txt", "rw").getChannel();
// wrBuf = rwChannel.map(FileChannel.MapMode.READ_WRITE, 0, Integer.MAX_VALUE);
// }
}
}
outStream.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}catch (java.nio.BufferOverflowException e) {
log.error(line);
log.error(String.valueOf(buf.length));
e.printStackTrace();
}
log.info("Done pairs");
}
}
src/main/java/edu/lu/uni/serval/fixminer/cluster/RunnableCompare.java
+23
View File
@@ -0,0 +1,23 @@
package edu.lu.uni.serval.fixminer.cluster;
import redis.clients.jedis.JedisPool;
public class RunnableCompare implements Runnable {
private String name;
private JedisPool innerPool;
private JedisPool outerPool;
private Compare comparer;
public RunnableCompare(String name , JedisPool innerPool, JedisPool outerPool, Compare comp) {
this.name = name;
this.innerPool = innerPool;
this.outerPool = outerPool;
this.comparer = comp;
}
@Override
public void run() {
comparer.coreCompare(name, innerPool, outerPool);
}
}
src/main/java/edu/lu/uni/serval/fixminer/cluster/StoreEDiffInCache.java
+143
View File
@@ -0,0 +1,143 @@
package edu.lu.uni.serval.fixminer.cluster;
import edu.lu.uni.serval.FixPattern.utils.EDiffHelper;
import edu.lu.uni.serval.FixPattern.utils.PoolBuilder;
import edu.lu.uni.serval.gumtree.regroup.HierarchicalActionSet;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import java.io.*;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.Stream;
/**
* Created by anilkoyuncu on 03/04/2018.
*/
public class StoreEDiffInCache {
private static Logger log = LoggerFactory.getLogger(StoreEDiffInCache.class);
public static void main(String inputPath,String portInner,String dbDir,String chunkName,String operation) throws Exception {
String parameters = String.format("\nInput path %s \nportInner %s \nchunkName %s \ndbDir %s \noperation %s",inputPath,portInner,chunkName,dbDir,operation);
log.info(parameters);
CallShell cs = new CallShell();
String cmd = "bash "+dbDir + "/" + "startServer.sh" +" %s %s %s";
cmd = String.format(cmd, dbDir,chunkName,Integer.valueOf(portInner));
cs.runShell(cmd, portInner);
File folder = new File(inputPath);
File[] subFolders = folder.listFiles();
Stream<File> stream = Arrays.stream(subFolders);
List<File> pjs = stream
.filter(x -> !x.getName().startsWith("."))
.collect(Collectors.toList());
List<String> workList = new ArrayList<String>();
File[] dumps;
for (File pj : pjs) {
String pjName = pj.getName();
File[] files = pj.listFiles();
Stream<File> fileStream = Arrays.stream(files);
List<File> fs;
if (operation.equals("ALLOP")){
fs= fileStream
.filter(x -> x.getName().startsWith("UPD") ||
x.getName().startsWith("INS") ||
x.getName().startsWith("DEL") ||
x.getName().startsWith("MOV"))
.collect(Collectors.toList());
File[] files1 = fs.get(0).listFiles();
File[] files2 = fs.get(1).listFiles();
File[] files3 = fs.get(2).listFiles();
File[] files4 = fs.get(3).listFiles();
dumps = Stream.of(files1, files2, files3,files4).flatMap(Stream::of).toArray(File[]::new);
}else{
fs = fileStream
.filter(x -> x.getName().startsWith(operation))
.collect(Collectors.toList());
dumps = fs.get(0).listFiles();
}
for (File f : dumps) {
String name = f.getName();
String key = pjName + "/"+ operation+"/" + name;
String result = key +","+f.getPath();
workList.add(result);
}
}
log.info(String.valueOf(workList.size()));
JedisPool innerPool = new JedisPool(PoolBuilder.getPoolConfig(), "127.0.0.1",Integer.valueOf(portInner),20000000);
workList.stream().parallel()
.forEach(m -> storeCore(innerPool, m.split(",")[1],m.split(",")[0]));
log.info(parameters);
String stopServer = "bash "+dbDir + "/" + "stopServer.sh" +" %s";
String stopServer2 = String.format(stopServer,Integer.valueOf(portInner));
cs.runShell(stopServer2, portInner);
}
public static void storeCore(JedisPool innerPool,String path,String key){
try {
HierarchicalActionSet actionSet = null;
try {
FileInputStream fi = new FileInputStream(new File(path));
ObjectInputStream oi = new ObjectInputStream(fi);
actionSet = (HierarchicalActionSet) oi.readObject();
oi.close();
fi.close();
} catch (FileNotFoundException e) {
log.error("File not found");
e.printStackTrace();
} catch (IOException e) {
log.error("Error initializing stream");
e.printStackTrace();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
try (Jedis inner = innerPool.getResource()) {
inner.set(key, EDiffHelper.toString(actionSet));
}
} catch (FileNotFoundException e) {
log.error("File not found");
e.printStackTrace();
} catch (IOException e) {
log.error("Error initializing stream");
e.printStackTrace();
}
}
}
src/main/java/edu/lu/uni/serval/fixminer/cluster/TreeActor.java
+83
View File
@@ -0,0 +1,83 @@
package edu.lu.uni.serval.fixminer.cluster;
import akka.actor.ActorRef;
import akka.actor.Props;
import akka.actor.UntypedActor;
import akka.japi.Creator;
import akka.routing.RoundRobinPool;
import edu.lu.uni.serval.fixminer.cluster.akka.TreeMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.JedisPool;
import java.util.List;
public class TreeActor extends UntypedActor {
private static Logger logger = LoggerFactory.getLogger(TreeActor.class);
private ActorRef mineRouter;
private final int numberOfWorkers;
private int counter = 0;
public TreeActor(int numberOfWorkers) {
mineRouter = this.getContext().actorOf(new RoundRobinPool(numberOfWorkers)
.props(TreeWorker.props()), "tree-router");
this.numberOfWorkers = numberOfWorkers;
}
public static Props props(final int numberOfWorkers) {
return Props.create(new Creator<TreeActor>() {
private static final long serialVersionUID = 9207427376110704705L;
@Override
public TreeActor create() throws Exception {
return new TreeActor(numberOfWorkers);
}
});
}
@SuppressWarnings("deprecation")
@Override
public void onReceive(Object message) throws Exception {
if (message instanceof TreeMessage) {
List<String> pairs = ((TreeMessage) message).getName();
JedisPool innerPool = ((TreeMessage) message).getInnerPool();
JedisPool outerPool = ((TreeMessage) message).getOuterPool();
int size = pairs.size();
int average = size / numberOfWorkers;
int reminder = size % numberOfWorkers;
int counter = 0;
for (int i = 0; i < numberOfWorkers; i ++) {
int fromIndex = i * average + counter;
if (counter < reminder) counter ++;
int toIndex = (i + 1) * average + counter;
List<String> pairsOfWorkers = pairs.subList(fromIndex, toIndex);
final TreeMessage workMsg = new TreeMessage(i + 1, pairsOfWorkers,innerPool,outerPool);
mineRouter.tell(workMsg, getSelf());
logger.info("Assign a task to worker #" + (i + 1) + "...");
}
} else if ("STOP".equals(message.toString())) {
counter ++;
logger.info(counter + " workers finailized their work...");
if (counter >= numberOfWorkers) {
logger.info("All workers finailized their work...");
this.getContext().stop(mineRouter);
this.getContext().stop(getSelf());
this.getContext().system().shutdown();
}
} else {
unhandled(message);
}
}
}
src/main/java/edu/lu/uni/serval/fixminer/cluster/TreeLoaderClusterL1.java
+123
View File
@@ -0,0 +1,123 @@
package edu.lu.uni.serval.fixminer.cluster;
import edu.lu.uni.serval.FixPattern.utils.PoolBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.ScanParams;
import redis.clients.jedis.ScanResult;
import java.io.File;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.Stream;
/**
* Created by anilkoyuncu on 19/03/2018.
*/
public class TreeLoaderClusterL1 {
private static Logger log = LoggerFactory.getLogger(TreeLoaderClusterL1.class);
public static void main(String portInner,String port,String inputPath,String level1DB,String level1Path,String innerTypePrefix) throws Exception {
String parameters = String.format("\nInput path %s \nportInner %s \nport %s",inputPath,portInner,port);
log.info(parameters);
String cmd = "bash "+inputPath + "/" + "startServer.sh" +" %s %s %s";
cmd = String.format(cmd, inputPath,level1DB,Integer.valueOf(port));
CallShell cs = new CallShell();
cs.runShell(cmd, port);
JedisPool outerPool = new JedisPool(PoolBuilder.getPoolConfig(), "127.0.0.1",Integer.valueOf(port),20000000);
File chunks = new File(level1Path);
File[] listFiles = chunks.listFiles();
Stream<File> stream = Arrays.stream(listFiles);
List<File> dbs = stream
.filter(x -> x.getName().endsWith(".rdb"))
.filter(x-> x.getName().startsWith(innerTypePrefix))
.collect(Collectors.toList());
for (File db : dbs) {
String cmdInner = "bash "+inputPath + "/" + "startServer.sh" +" %s %s %s";
cmdInner = String.format(cmdInner, inputPath,db.getName(),Integer.valueOf(portInner));
cs.runShell(cmdInner, port);
JedisPool innerPool = new JedisPool(PoolBuilder.getPoolConfig(), "127.0.0.1",Integer.valueOf(portInner),20000000);
Jedis inner = null;
try {
inner = innerPool.getResource();
inner.select(1);
ScanParams sc = new ScanParams();
//150000000
sc.count(150000000);
sc.match("match_[0-9]*");
ScanResult<String> scan; scan = inner.scan("0", sc);
int size = scan.getResult().size();
log.info("Scanning " + String.valueOf(size));
List<String> result = scan.getResult();
for (String key : result) {
String value = inner.get(key);
Jedis outer = null;
try {
outer = outerPool.getResource();
outer.set(key,value);
} catch (Exception e) {
log.error(e.toString() + " {}", (key));
}finally {
if (outer != null) {
outer.close();
}
}
}
} catch (Exception e) {
log.error(e.toString() + " {}", (db.getName()));
}finally {
if (inner != null) {
inner.close();
}
innerPool.close();
}
String stopServer = "bash "+inputPath + "/" + "stopServer.sh" +" %s";
stopServer = String.format(stopServer,Integer.valueOf(portInner));
cs.runShell(stopServer, port);
}
String stopServer1 = "bash "+inputPath + "/" + "stopServer.sh" +" %s";
stopServer1 = String.format(stopServer1,Integer.valueOf(port));
cs.runShell(stopServer1, port);
}
}
src/main/java/edu/lu/uni/serval/fixminer/cluster/TreeWorker.java
+224
View File
@@ -0,0 +1,224 @@
package edu.lu.uni.serval.fixminer.cluster;
import akka.actor.Props;
import akka.actor.UntypedActor;
import akka.japi.Creator;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.fixminer.cluster.akka.TreeMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.JedisPool;
import java.util.List;
import java.util.concurrent.*;
public class TreeWorker extends UntypedActor {
private static Logger log = LoggerFactory.getLogger(TreeWorker.class);
// private JedisPool innerPool;
// private JedisPool outerPool;
//
// public TreeWorker(String innerPort,String outerPort) {
//// this.innerPool = innerPool;
//// this.outerPool = outerPool;
// this.outerPool = new JedisPool(poolConfig, "127.0.0.1",Integer.valueOf(outerPort),20000000);
// this.innerPool = new JedisPool(poolConfig, "127.0.0.1",Integer.valueOf(innerPort),20000000);
//
//
// }
//
// public static Props props(final String innerPort,final String outerPort) {
// return Props.create(new Creator<TreeWorker>() {
//
// private static final long serialVersionUID = -7615153844097275009L;
//
// @Override
// public TreeWorker create() throws Exception {
// return new TreeWorker(innerPort,outerPort);
// }
//
// });
// }
public TreeWorker() {
}
public static Props props() {
return Props.create(new Creator<TreeWorker>() {
private static final long serialVersionUID = -7615153844097275009L;
@Override
public TreeWorker create() throws Exception {
return new TreeWorker();
}
});
}
@Override
public void onReceive(Object message) throws Exception {
if(message instanceof TreeMessage) {
// if (message instanceof edu.lu.uni.serval.MultipleThreadsParser.WorkMessage) {
TreeMessage msg = (TreeMessage) message;
List<String> files = msg.getName();
JedisPool innerPool = msg.getInnerPool();
JedisPool outerPool = msg.getOuterPool();
int id = msg.getId();
// int counter = new Object() {
int counter = 0;
//
for (String name : files)
{
// files.stream().
// parallel().
// peek(x -> counter++).
// forEach(m ->
// {
// Compare compare = new Compare();
// compare.coreCompare(m, innerPool, outerPool);
// }
// );
// }
// }.counter;
//
final ExecutorService executor = Executors.newFixedThreadPool(20);
// // schedule the work
final Future<?> future = executor.submit(new RunnableCompare(name, innerPool, outerPool, new Compare()));
try {
// wait for task to complete
future.get(Configuration.SECONDS_TO_WAIT, TimeUnit.SECONDS);
Compare compare = new Compare();
compare.coreCompare(name, innerPool, outerPool);
counter++;
// nullDiffEntry += parser.nullMatchedDiffEntry;
// nullMappingGumTreeResults += parser.nullMappingGumTreeResult;
// pureDeletion += parser.pureDeletions;
// largeHunk += parser.largeHunk;
// nullSourceCode += parser.nullSourceCode;
// testInfos += parser.testInfos;
// testingInfo.append(parser.testingInfo);
// builder.append(parser.unfixedViolations);
//
// String editScript = parser.getAstEditScripts();
// if ("".equals(editScript)) {
//// if (parser.resultType == 1) {
//// nullGumTreeResults += countAlarms(positionFile, "");
//// } else if (parser.resultType == 2) {
//// noSourceCodeChanges += countAlarms(positionFile, "");
//// } else if (parser.resultType == 3) {
//// noStatementChanges += countAlarms(positionFile, "");
//// }
// } else {
// editScripts.append(editScript);
// patchesSourceCode.append(parser.getPatchesSourceCode());
// sizes.append(parser.getSizes());
// tokens.append(parser.getTokensOfSourceCode());
//
// counter ++;
// if (counter % 100 == 0) {
// FileHelper.outputToFile(editScriptsFilePath + "edistScripts_" + id + ".list", editScripts, true);
// FileHelper.outputToFile(patchesSourceCodeFilePath + "patches_" + id + ".list", patchesSourceCode, true);
// FileHelper.outputToFile(editScriptSizesFilePath + "sizes_" + id + ".list", sizes, true);
// FileHelper.outputToFile(buggyTokensFilePath + "tokens_" + id + ".list", tokens, true);
// editScripts.setLength(0);
// patchesSourceCode.setLength(0);
// sizes.setLength(0);
// tokens.setLength(0);
// log.info("Worker #" + id +" finialized parsing " + counter + " files...");
// FileHelper.outputToFile("OUTPUT/testingInfo_" + id + ".list", testingInfo, true);
// testingInfo.setLength(0);
// }
// }
} catch (TimeoutException e) {
future.cancel(true);
////// timeouts += countAlarms(positionFile, "#Timeout:");
// System.err.println("#Timeout: " + name);
} catch (InterruptedException e) {
////// timeouts += countAlarms(positionFile, "#TimeInterrupted:");
//// System.err.println("#TimeInterrupted: " + revFile.getName());
e.printStackTrace();
} catch (ExecutionException e) {
////// timeouts += countAlarms(positionFile, "#TimeAborted:");
//// System.err.println("#TimeAborted: " + revFile.getName());
e.printStackTrace();
} finally {
executor.shutdownNow();
}
}
log.info("bitti");
log.info("Worker #" + id +"finialized parsing " + counter + " files...");
log.info("Worker #" + id + " finialized the work...");
this.getSender().tell("STOP", getSelf());
// String stopServer = "bash "+dbDir + "/" + "stopServer.sh" +" %s";
// stopServer = String.format(stopServer,Integer.valueOf(innerPort));
// loadRedis(stopServer,serverWait);
//
// if (sizes.length() > 0) {
// FileHelper.outputToFile(editScriptsFilePath + "editScripts_" + id + ".list", editScripts, true);
// FileHelper.outputToFile(patchesSourceCodeFilePath + "patches_" + id + ".list", patchesSourceCode, true);
// FileHelper.outputToFile(editScriptSizesFilePath + "sizes_" + id + ".list", sizes, true);
// FileHelper.outputToFile(buggyTokensFilePath + "tokens_" + id + ".list", tokens, true);
// editScripts.setLength(0);
// patchesSourceCode.setLength(0);
// sizes.setLength(0);
// tokens.setLength(0);
//
// FileHelper.outputToFile("OUTPUT/testingInfo_" + id + ".list", testingInfo, true);
// testingInfo.setLength(0);
// }
// String statistic = "\nNullGumTreeResults: " + nullGumTreeResults + "\nNoSourceCodeChanges: " + noSourceCodeChanges +
// "\nNoStatementChanges: " + noStatementChanges + "\nNullDiffEntry: " + nullDiffEntry + "\nNullMatchedGumTreeResults: " + nullMappingGumTreeResults +
// "\nPureDeletion: " + pureDeletion + "\nLargeHunk: " + largeHunk + "\nNullSourceCode: " + nullSourceCode +
// "\nTestingInfo: " + testInfos + "\nTimeout: " + timeouts;
// FileHelper.outputToFile("OUTPUT/statistic_" + id + ".list", statistic, false);
// FileHelper.outputToFile("OUTPUT/UnfixedV_" + id + ".list", builder, false);
//
// log.info("Worker #" + id +"finialized parsing " + counter + " files...");
// log.info("Worker #" + id + " finialized the work...");
// this.getSender().tell("STOP", getSelf());
// } else {
}else{
unhandled(message);
}
}
// static final JedisPoolConfig poolConfig = buildPoolConfig();
//
//
// private static JedisPoolConfig buildPoolConfig() {
// final JedisPoolConfig poolConfig = new JedisPoolConfig();
// poolConfig.setMaxTotal(128);
// poolConfig.setMaxIdle(128);
// poolConfig.setMinIdle(16);
// poolConfig.setTestOnBorrow(true);
// poolConfig.setTestOnReturn(true);
// poolConfig.setTestWhileIdle(true);
// poolConfig.setMinEvictableIdleTimeMillis(Duration.ofMinutes(60).toMillis());
// poolConfig.setTimeBetweenEvictionRunsMillis(Duration.ofHours(30).toMillis());
// poolConfig.setNumTestsPerEvictionRun(3);
// poolConfig.setBlockWhenExhausted(true);
//
// return poolConfig;
// }
}
src/main/java/edu/lu/uni/serval/fixminer/cluster/WorkMessage.java
+51
View File
@@ -0,0 +1,51 @@
package edu.lu.uni.serval.fixminer.cluster;
import java.util.List;
public class WorkMessage {
private int id;
private List<String> msgFiles;
public String getDbDir() {
return dbDir;
}
public String getServerWait() {
return serverWait;
}
private String dbDir;
private String serverWait;
public String getInputPath() {
return inputPath;
}
private String inputPath;
public String getInnerPort() {
return innerPort;
}
private String innerPort;
public WorkMessage(int id, List<String> msgFiles,String innerPort,String inputPath,String dbDir,String serverWait) {
super();
this.id = id;
this.msgFiles = msgFiles;
this.innerPort = innerPort;
this.inputPath = inputPath;
this.dbDir = dbDir;
this.serverWait = serverWait;
}
public int getId() {
return id;
}
public List<String> getMsgFiles() {
return msgFiles;
}
}
src/main/java/edu/lu/uni/serval/fixminer/cluster/akka/AkkaTreeParser.java
+75
View File
@@ -0,0 +1,75 @@
package edu.lu.uni.serval.fixminer.cluster.akka;
import akka.actor.ActorRef;
import akka.actor.ActorSystem;
import edu.lu.uni.serval.fixminer.cluster.TreeActor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.ScanParams;
import redis.clients.jedis.ScanResult;
import java.util.List;
/**
* Created by anilkoyuncu on 12/09/2018.
*/
public class AkkaTreeParser {
private static Logger log = LoggerFactory.getLogger(AkkaTreeParser.class);
public static void akkaCompare(JedisPool innerPool, JedisPool outerPool, String numOfWorkers, String cursor){
final List<String> listOfPairs = getMessages(innerPool,cursor); //"/Users/anilkoyuncu/bugStudy/code/python/GumTreeInput/Apache/CAMEL/"
ActorSystem system = null;
ActorRef parsingActor = null;
final TreeMessage msg = new TreeMessage(0,listOfPairs, innerPool,outerPool);
try {
log.info("Akka begins...");
system = ActorSystem.create("Compare-EnhancedDiff-System");
parsingActor = system.actorOf(TreeActor.props(Integer.valueOf(numOfWorkers)), "mine-fix-pattern-actor");
parsingActor.tell(msg, ActorRef.noSender());
} catch (Exception e) {
system.shutdown();
e.printStackTrace();
}
}
public static List<String> getMessages(JedisPool innerPool, String cursor){
ScanResult<String> scan;
try (Jedis inner = innerPool.getResource()) {
while (!inner.ping().equals("PONG")){
log.info("wait");
}
ScanParams sc = new ScanParams();
//150000000
log.info("Scanning ");
sc.count(Integer.valueOf(cursor));
sc.match("pair_[0-9]*");
scan = inner.scan("0", sc);
int size = scan.getResult().size();
log.info("Scanned " + String.valueOf(size));
}
List<String> result = scan.getResult();
log.info("Getting results");
return result;
}
}
src/main/java/edu/lu/uni/serval/fixminer/cluster/akka/TreeMessage.java
+54
View File
@@ -0,0 +1,54 @@
package edu.lu.uni.serval.fixminer.cluster.akka;
import redis.clients.jedis.JedisPool;
import java.util.List;
/**
* Created by anilkoyuncu on 12/09/2018.
*/
public class TreeMessage {
private int id;
private List<String> name;
private JedisPool innerPool;
private JedisPool outerPool;
public TreeMessage(int id, List<String> name, JedisPool innerPool, JedisPool outerPool) {
this.id = id;
this.name = name;
this.innerPool = innerPool;
this.outerPool = outerPool;
}
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
public List<String> getName() {
return name;
}
public void setName(List<String> name) {
this.name = name;
}
public JedisPool getInnerPool() {
return innerPool;
}
public void setInnerPool(JedisPool innerPool) {
this.innerPool = innerPool;
}
public JedisPool getOuterPool() {
return outerPool;
}
public void setOuterPool(JedisPool outerPool) {
this.outerPool = outerPool;
}
}
src/main/java/edu/lu/uni/serval/gumtree/regroup/ActionFilter.java
-254
View File
@@ -1,254 +0,0 @@
package edu.lu.uni.serval.gumtree.regroup;
import java.util.ArrayList;
import java.util.List;
public class ActionFilter {
private List<String> methodNames = new ArrayList<>();
private List<String> variableNames = new ArrayList<>();
/**
* Filter out the modify actions of changing method names, method parameters, variable names and field names in declaration part.
*
* @param actionSets
* @return
*/
public List<HierarchicalActionSet> filterOutUselessActions(List<HierarchicalActionSet> actionSets) {
// Filter out modifications of variable names and method names.
List<HierarchicalActionSet> uselessActions = findoutUselessActions(actionSets);
actionSets.removeAll(uselessActions);
uselessActions.clear();
// Filter out non-UPD modifications, and modifications of variable names and method names.
uselessActions = findoutUselessActionSets(actionSets, true);
actionSets.removeAll(uselessActions);
uselessActions.clear();
return actionSets;
}
private List<HierarchicalActionSet> findoutUselessActionSets(List<HierarchicalActionSet> actionSets, boolean isRoot) {
List<HierarchicalActionSet> uselessActions = new ArrayList<>();
FindActionSet: {
for (HierarchicalActionSet actionSet : actionSets) {
if (!isRoot) {
String actionStr = actionSet.getActionString();
if (actionStr.startsWith("UPD MethodInvocation") || actionStr.startsWith("INS MethodInvocation") || actionStr.startsWith("DEL MethodInvocation")) {
String label = actionSet.getAction().getNode().getLabel();
for (String methodName : methodNames) {
if (actionSet.getActionString().startsWith("UPD MethodInvocation@@" + methodName + "(")
|| actionSet.getActionString().startsWith("INS MethodInvocation@@" + methodName + "(")
|| actionSet.getActionString().startsWith("DEL MethodInvocation@@" + methodName + "(")
|| label.contains("." + methodName + "(")) {
addToUselessActions(actionSet, uselessActions);
break FindActionSet;
}
}
} else if (actionStr.startsWith("UPD SimpleName") || actionStr.startsWith("INS SimpleName") || actionStr.startsWith("DEL SimpleName")) {
String label = actionSet.getAction().getNode().getLabel();
for (String variableName : variableNames) {
if (label.equals(variableName) || label.equals("Name:" + variableName)) {
addToUselessActions(actionSet, uselessActions);
break FindActionSet;
}
}
} else if (actionStr.startsWith("UPD StringLiteral") || actionStr.startsWith("INS StringLiteral")
|| actionStr.startsWith("DEL StringLiteral") || actionStr.startsWith("MOV StringLiteral")
|| actionStr.startsWith("UPD CharacterLiteral") || actionStr.startsWith("INS CharacterLiteral")
|| actionStr.startsWith("DEL CharacterLiteral") || actionStr.startsWith("MOV CharacterLiteral")) {
addToUselessActions(actionSet, uselessActions);
break FindActionSet;
}
List<HierarchicalActionSet> uselessActionSets = findoutUselessActionSets(actionSet.getSubActions(), false);
if (uselessActionSets.size() > 0) {
uselessActions.addAll(uselessActionSets);
break;
}
} else {
if (actionSet.getAstNodeType().endsWith("Statement") || "FieldDeclaration".equals(actionSet.getAstNodeType())) {
uselessActions.addAll(findoutUselessActionSets(actionSet.getSubActions(), false));
} else {
uselessActions.add(actionSet);
}
}
}
}
return uselessActions;
}
private void addToUselessActions(HierarchicalActionSet actionSet, List<HierarchicalActionSet> uselessActions) {
while (actionSet.getParent() != null) {
actionSet = actionSet.getParent();
}
if (!uselessActions.contains(actionSet)) {
uselessActions.add(actionSet);
}
}
/**
* Identify the the modify actions of changing method names, method parameters, variable names and field names in declaration part.
*
* @param actionSets
* @return
*/
private List<HierarchicalActionSet> findoutUselessActions(List<HierarchicalActionSet> actionSets) {
List<HierarchicalActionSet> uselessActions = new ArrayList<>();
for (HierarchicalActionSet actionSet : actionSets) {
String actionType = actionSet.getAstNodeType();
if (actionType.equals("MethodDeclaration")) {
addToUselessActions(actionSet, uselessActions);// INS, DEL: useful?, UPD, except the modifier actions
if (!actionSet.getActionString().startsWith("MOV ")) {
String label = actionSet.getNode().getLabel();
String methodName = label.substring(label.indexOf("MethodName:"));
methodName = methodName.substring(11, methodName.indexOf(","));
methodNames.add(methodName); // "MethodName:***"
// UPD, DEL, INS parameters.
List<HierarchicalActionSet> subActionSets = actionSet.getSubActions();
for (HierarchicalActionSet subActionSet : subActionSets) {
if (subActionSet.getAstNodeType().equals("SingleVariableDeclaration")) {
List<HierarchicalActionSet> subActionSets2 = subActionSet.getSubActions(); // <Type, identifier>
if (subActionSets2.size() == 0) {
String actSetStr = subActionSet.getActionString();
int index1 = actSetStr.indexOf("@@");
int index2 = 0;
if (actSetStr.startsWith("DEL")) {
index2 = actSetStr.indexOf("@AT@");
} else {
index2 = actSetStr.indexOf("@TO@");;
}
actSetStr = actSetStr.substring(index1, index2).trim();
String variableName = actSetStr.substring(actSetStr.lastIndexOf(" "));
variableNames.add(variableName); // "SimpleName:" + variableName TODO: effect range
} else {
HierarchicalActionSet actSet = subActionSets2.get(subActionSets2.size() - 1);
String actStr = actSet.getActionString();
if (actStr.startsWith("UPD SimpleName") || actStr.startsWith("INS SimpleName") || actStr.startsWith("DEL SimpleName")) {
String variableName = actSet.getNode().getLabel();
variableNames.add(variableName); // "SimpleName:" + variableName TODO: effect range
}
}
}
}
}
} else if (actionType.equals("FieldDeclaration") || actionType.equals("VariableDeclarationStatement")) {
// UPD VariableDeclarationFragment
if (!actionSet.getActionString().startsWith("MOV ")) {
List<HierarchicalActionSet> subActionSets = actionSet.getSubActions();
if (subActionSets.size() > 0) {
for (HierarchicalActionSet subActionSet : subActionSets) { // VariableDeclarationFragments
if (identifyUpdateVDF(subActionSet)) {
addToUselessActions(actionSet, uselessActions);
}
}
}
}
} else if (actionType.equals("TryStatement")) {
if (actionSet.getActionString().startsWith("UPD ")) {
List<HierarchicalActionSet> subActionSets = actionSet.getSubActions();
if (subActionSets.size() > 0) {
for (HierarchicalActionSet subActionSet : subActionSets) {
if (subActionSet.getActionString().startsWith("UPD VariableDeclarationExpression")) {
List<HierarchicalActionSet> subActionSets2 = subActionSet.getSubActions(); // VariableDeclarationFragments
for (HierarchicalActionSet subActionSet2 : subActionSets2) {
if (identifyUpdateVDF(subActionSet2)) {
addToUselessActions(actionSet, uselessActions);
}
}
} else {
break;
}
}
}
}
} else if (actionType.equals("EnhancedForStatement")) { // SingleVariableDeclaration
if (!actionSet.getActionString().startsWith("MOV ")) {
List<HierarchicalActionSet> subActionSets = actionSet.getSubActions();
if (subActionSets.size() > 0) {
HierarchicalActionSet subActionSet = subActionSets.get(0);
if (subActionSet.getActionString().startsWith("UPD SingleVariableDeclaration")) {
List<HierarchicalActionSet> subActionSets2 = subActionSet.getSubActions();
for (HierarchicalActionSet subActionSet2 : subActionSets2) { // Type or Identifier
if (subActionSet2.getActionString().startsWith("UPD SimpleName")) {
String variableName = subActionSet2.getNode().getLabel();
variableNames.add(variableName); // "SimpleName:" + variableName TODO: effect range
addToUselessActions(actionSet, uselessActions);
}
}
}
}
}
} else if (actionType.equals("SingleVariableDeclaration")) {
if (!actionSet.getActionString().startsWith("MOV ")) {
List<HierarchicalActionSet> subActionSets2 = actionSet.getSubActions(); // <Type, identifier>
if (subActionSets2.size() == 0) {
String actSetStr = actionSet.getActionString();
int index1 = actSetStr.indexOf("@@");
int index2 = 0;
if (actSetStr.startsWith("DEL")) {
index2 = actSetStr.indexOf("@AT@");
} else {
index2 = actSetStr.indexOf("@TO@");;
}
actSetStr = actSetStr.substring(index1, index2).trim();
String variableName = actSetStr.substring(actSetStr.lastIndexOf(" "));
variableNames.add(variableName); // "SimpleName:" + variableName TODO: effect range
addToUselessActions(actionSet, uselessActions);
} else {
HierarchicalActionSet actSet = subActionSets2.get(subActionSets2.size() - 1);
String actStr = actSet.getActionString();
if (actStr.startsWith("UPD SimpleName") || actStr.startsWith("INS SimpleName") || actStr.startsWith("DEL SimpleName")) {
String variableName = actSet.getNode().getLabel();
variableNames.add(variableName); // "SimpleName:" + variableName TODO: effect range
addToUselessActions(actionSet, uselessActions);
}
}
}
} else {
if (actionSet.getParent() != null) {
while (actionSet.getParent() != null) {
actionSet = actionSet.getParent();
}
if (uselessActions.contains(actionSet)) {
return uselessActions;
} else {
uselessActions.addAll(findoutUselessActions(actionSet.getSubActions()));
}
}
}
}
return uselessActions;
}
/**
* Identify the AST node of this ActionSet is VariableDeclarationFragment or not.
* And, whether the action is happened on the Variable name or not.
*
* @param actionSet
*/
private boolean identifyUpdateVDF(HierarchicalActionSet actionSet) {
String actStr = actionSet.getActionString();
if (actStr.startsWith("UPD VariableDeclarationFragment")
|| actStr.startsWith("INS VariableDeclarationFragment")
|| actStr.startsWith("DEL VariableDeclarationFragment")) {
List<HierarchicalActionSet> subActionSets = actionSet.getSubActions();
if (subActionSets == null || subActionSets.size() == 0) {
// modification of Dimension
return true;
}
HierarchicalActionSet actSet = subActionSets.get(0);
String actSetStr = actSet.getActionString();
if (actSetStr.startsWith("UPD SimpleName") || actSetStr.startsWith("INS SimpleName") || actSetStr.startsWith("DEL SimpleName")) {
String variableName = actSet.getNode().getLabel();
variableNames.add(variableName); // "SimpleName:" + variableName TODO: effect range
return true;
}
}
return false;
}
}
src/main/java/edu/lu/uni/serval/gumtree/regroup/HierarchicalActionSet.java
+20 -77
View File
@@ -1,18 +1,19 @@
package edu.lu.uni.serval.gumtree.regroup;
import java.util.ArrayList;
import java.util.List;
import com.github.gumtreediff.actions.model.Action;
import com.github.gumtreediff.tree.ITree;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.List;
/**
* Hierarchical-level results of GumTree results
*
* @author kui.liu
*
*/
public class HierarchicalActionSet implements Comparable<HierarchicalActionSet> {
public class HierarchicalActionSet implements Comparable<HierarchicalActionSet>,Serializable {
private String astNodeType;
private Action action;
@@ -28,11 +29,16 @@ public class HierarchicalActionSet implements Comparable<HierarchicalActionSet>
private List<HierarchicalActionSet> subActions = new ArrayList<>();
private ITree node;
private SimpleTree abstractSimpleTree = null; // semi-source code tree. and AST node type tree
private SimpleTree abstractIdentifierTree = null; // abstract identifier tree
private SimpleTree simpleTree = null; // source code tree and AST node type tree
private SimpleTree originalTree = null; // source code tree.
// source code tree.
public int getBugEndPosition() {
return bugEndPosition;
}
public int getFixEndPosition() {
return fixEndPosition;
}
private int bugEndPosition;
private int fixEndPosition;
@@ -146,49 +152,12 @@ public class HierarchicalActionSet implements Comparable<HierarchicalActionSet>
this.subActions = subActions;
}
public SimpleTree getAbstractSimpleTree() {
return abstractSimpleTree;
}
public void setAbstractSimpleTree(SimpleTree simpleTree) {
this.abstractSimpleTree = simpleTree;
}
public SimpleTree getAbstractIdentifierTree() {
return abstractIdentifierTree;
}
public void setAbstractIdentifierTree(SimpleTree abstractIdentifierTree) {
this.abstractIdentifierTree = abstractIdentifierTree;
}
public SimpleTree getSimpleTree() {
return simpleTree;
}
public void setSimpleTree(SimpleTree rawTokenTree) {
this.simpleTree = rawTokenTree;
}
public SimpleTree getOriginalTree() {
return originalTree;
}
public void setOriginalTree(SimpleTree originalTree) {
this.originalTree = originalTree;
}
public int getBugEndPosition() {
return bugEndPosition;
}
public void setBugEndPosition(int bugEndPosition) {
this.bugEndPosition = bugEndPosition;
}
public int getFixEndPosition() {
return fixEndPosition;
}
public void setFixEndPosition(int fixEndPosition) {
this.fixEndPosition = fixEndPosition;
@@ -196,10 +165,7 @@ public class HierarchicalActionSet implements Comparable<HierarchicalActionSet>
@Override
public int compareTo(HierarchicalActionSet o) {
// int result = this.startPosition.compareTo(o.startPosition);
// if (result == 0) {
// result = this.length >= o.length ? -1 : 1;
// }
return this.startPosition.compareTo(o.startPosition);//this.action.compareTo(o.action);
}
@@ -214,7 +180,7 @@ public class HierarchicalActionSet implements Comparable<HierarchicalActionSet>
actionSet.toString();
List<String> strList1 = actionSet.strList;
for (String str1 : strList1) {
strList.add("----" + str1);
strList.add("---" + str1);
}
}
} else {
@@ -224,7 +190,7 @@ public class HierarchicalActionSet implements Comparable<HierarchicalActionSet>
actionSet.toString();
List<String> strList1 = actionSet.strList;
for (String str1 : strList1) {
strList.add("----" + str1);
strList.add("---" + str1);
}
}
}
@@ -237,28 +203,5 @@ public class HierarchicalActionSet implements Comparable<HierarchicalActionSet>
return str;
}
public String toASTNodeLevelAction() {
if (strList.size() == 0) {
toString();
}
String astNodeStr = "";
for (String str : strList) {
astNodeStr += str.substring(0, str.indexOf("@@")) + "\n";
}
return astNodeStr;
}
public String toRawCodeLevelAction() {
if (strList.size() == 0) {
toString();
}
String astNodeStr = "";
for (String str : strList) {
str = str.substring(0, str.indexOf(" @AT@")) + "\n";
int index1 = str.indexOf(" ") + 1;
int index2 = str.indexOf("@@") + 2;
astNodeStr += str.substring(0, index1) + str.substring(index2);
}
return astNodeStr;
}
}
src/main/java/edu/lu/uni/serval/gumtree/regroup/HierarchicalRegrouper.java
+10 -19
View File
@@ -1,20 +1,13 @@
package edu.lu.uni.serval.gumtree.regroup;
import com.github.gumtreediff.actions.model.*;
import com.github.gumtreediff.tree.ITree;
import edu.lu.uni.serval.FixPattern.utils.ASTNodeMap;
import edu.lu.uni.serval.utils.ListSorter;
import java.util.ArrayList;
import java.util.List;
import com.github.gumtreediff.actions.model.Action;
import com.github.gumtreediff.actions.model.Addition;
import com.github.gumtreediff.actions.model.Delete;
import com.github.gumtreediff.actions.model.Insert;
import com.github.gumtreediff.actions.model.Move;
import com.github.gumtreediff.actions.model.Update;
import com.github.gumtreediff.tree.ITree;
import edu.lu.uni.serval.FixPattern.utils.ASTNodeMap;
import edu.lu.uni.serval.FixPattern.utils.Checker;
import edu.lu.uni.serval.utils.ListSorter;
/**
* Regroup GumTree results to a hierarchical construction.
*
@@ -22,7 +15,7 @@ import edu.lu.uni.serval.utils.ListSorter;
*
*/
public class HierarchicalRegrouper {
public List<HierarchicalActionSet> regroupGumTreeResults(List<Action> actions) {
/*
* First, sort actions by their positions.
@@ -70,6 +63,7 @@ public class HierarchicalRegrouper {
}
}
}
return reActionSets;
}
@@ -200,15 +194,12 @@ public class HierarchicalRegrouper {
if (type != 83 && type != 77 && type != 78 && type != 79
&& type != 5 && type != 39 && type != 43 && type != 74 && type != 75
&& type != 76 && type != 84 && type != 87 && type != 88 && type != 42) {
// ArrayType, PrimitiveType, SimpleType, ParameterizedType,
// ArrayType, PrimitiveType, SimpleType, ParameterizedType,
// QualifiedType, WildcardType, UnionType, IntersectionType, NameQualifiedType, SimpleName
return null;
}
// } else if (parent.getType() == 31) { // method declaration
// int type = action.getNode().getType();
// if (Checker.isStatement(type)) {// statements
// return null;
// }
}
for (Action act : actions) {
src/main/java/edu/lu/uni/serval/gumtree/regroup/HunkActionFilter.java
-764
View File
@@ -1,764 +0,0 @@
package edu.lu.uni.serval.gumtree.regroup;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import org.eclipse.jdt.core.dom.CompilationUnit;
import com.github.gumtreediff.actions.model.Move;
import com.github.gumtreediff.actions.model.Update;
import com.github.gumtreediff.tree.ITree;
import edu.lu.uni.serval.FixPattern.utils.Checker;
import edu.lu.uni.serval.FixPatternParser.CUCreator;
import edu.lu.uni.serval.FixPatternParser.violations.Violation;
import edu.lu.uni.serval.diffentry.DiffEntryHunk;
public class HunkActionFilter {
/**
* Filter out the modify actions, which are not in the DiffEntry hunks, without considering the same parent node.
*
* @param hunks
* @param actionSets
* @return
*/
public List<HierarchicalActionSet> filterActionsByDiffEntryHunk(List<DiffEntryHunk> hunks,
List<HierarchicalActionSet> actionSets, File revFile, File prevFile) {
List<HierarchicalActionSet> uselessActions = new ArrayList<>();
CUCreator cuCreator = new CUCreator();
CompilationUnit prevUnit = cuCreator.createCompilationUnit(prevFile);
CompilationUnit revUnit = cuCreator.createCompilationUnit(revFile);
if (prevUnit == null || revUnit == null) {
return uselessActions;
}
for (HierarchicalActionSet actionSet : actionSets) {
// position of buggy statements
int startPosition = 0;
int endPosition = 0;
int startLine = 0;
int endLine = 0;
// position of fixed statements
int startPosition2 = 0;
int endPosition2 = 0;
int startLine2 = 0;
int endLine2 = 0;
String actionStr = actionSet.getActionString();
if (actionStr.startsWith("INS")) {
startPosition2 = actionSet.getStartPosition();
endPosition2 = startPosition2 + actionSet.getLength();
List<Move> firstAndLastMov = getFirstAndLastMoveAction(actionSet);
if (firstAndLastMov != null) {
startPosition = firstAndLastMov.get(0).getNode().getPos();
ITree lastTree = firstAndLastMov.get(1).getNode();
endPosition = lastTree.getPos() + lastTree.getLength();
}
} else {
startPosition = actionSet.getStartPosition(); // range of actions
endPosition = startPosition + actionSet.getLength();
if (actionStr.startsWith("UPD")) {
Update update = (Update) actionSet.getAction();
ITree newNode = update.getNewNode();
startPosition2 = newNode.getPos();
endPosition2 = startPosition2 + newNode.getLength();
}
}
startLine = startPosition == 0 ? 0 : prevUnit.getLineNumber(startPosition);
endLine = endPosition == 0 ? 0 : prevUnit.getLineNumber(endPosition);
startLine2 = startPosition2 == 0 ? 0 : revUnit.getLineNumber(startPosition2);
endLine2 = endPosition2 == 0 ? 0 : revUnit.getLineNumber(endPosition2);
for (DiffEntryHunk hunk : hunks) {
int bugStartLine = hunk.getBugLineStartNum();
int bugRange = hunk.getBugRange();
int fixStartLine = hunk.getFixLineStartNum();
int fixRange = hunk.getFixRange();
if (actionStr.startsWith("INS")) {
if (fixStartLine + fixRange < startLine2) {
continue;
}
if (endLine2 < fixStartLine ) {
uselessActions.add(actionSet);
}
} else {
if (bugStartLine + bugRange < startLine) {
continue;
}
if (endLine < bugStartLine ) {
uselessActions.add(actionSet);
}
break;
}
}
actionSet.setBugStartLineNum(startLine);
actionSet.setBugEndLineNum(endLine);
actionSet.setFixStartLineNum(startLine2);
actionSet.setFixEndLineNum(endLine2);
}
actionSets.removeAll(uselessActions);
uselessActions.clear();
return actionSets;
}
/**
* Filter out the modify actions, which are not in the DiffEntry hunks, with considering the same parent node.
*
* @param hunks
* @param actionSets
* @return
*/
public List<HunkFixPattern> filterActionsByDiffEntryHunk2(List<DiffEntryHunk> hunks,
List<HierarchicalActionSet> actionSets, File revFile, File prevFile) {
List<HunkFixPattern> allHunkFixPatterns = new ArrayList<>();
CUCreator cuCreator = new CUCreator();
CompilationUnit prevUnit = cuCreator.createCompilationUnit(prevFile);
CompilationUnit revUnit = cuCreator.createCompilationUnit(revFile);
if (prevUnit == null || revUnit == null) {
return allHunkFixPatterns;
}
int i = 0;
int size = actionSets.size();
for (DiffEntryHunk hunk : hunks) {
int hunkBugStartLine = hunk.getBugLineStartNum();
int hunkBugRange = hunk.getBugRange();
int hunkFixStartLine = hunk.getFixLineStartNum();
int hunkFixRange = hunk.getFixRange();
for (; i < size; i ++) {
HierarchicalActionSet actionSet = actionSets.get(i);
int actionBugStartLine = actionSet.getBugStartLineNum();
if (actionBugStartLine == 0) {
actionBugStartLine = setLineNumbers(actionSet, prevUnit, revUnit);
}
int actionBugEndLine = actionSet.getBugEndLineNum();
int actionFixStartLine = actionSet.getFixStartLineNum();
int actionFixEndLine = actionSet.getFixEndLineNum();
String actionStr = actionSet.getActionString();
ITree previousParent = null;
List<HierarchicalActionSet> hunkActionSets = new ArrayList<>();
if (actionStr.startsWith("INS")) {
if (hunkFixStartLine + hunkFixRange < actionFixStartLine) {
addHunkActionSets(hunkActionSets, allHunkFixPatterns, hunk);// save the previous non-null hunkFixPattern.
break;
}
if (actionFixEndLine >= hunkFixStartLine ) {
ITree parent = addToHunkActionSets(actionSet, hunkActionSets, allHunkFixPatterns, previousParent, hunk);
if (parent != null) {
if (parent != previousParent) {
hunkActionSets = new ArrayList<>();
}
hunkActionSets.add(actionSet);
} else if (hunkActionSets.size() > 0) {
hunkActionSets = new ArrayList<>();
}
previousParent = parent;
}
} else { // UPD, DEL, MOV
if (hunkBugStartLine + hunkBugRange < actionBugStartLine) {
addHunkActionSets(hunkActionSets, allHunkFixPatterns, hunk);// save the previous non-null hunkFixPattern.
break;
}
if (actionBugEndLine >= hunkBugStartLine ) {
ITree parent = addToHunkActionSets(actionSet, hunkActionSets, allHunkFixPatterns, previousParent, hunk);
if (parent != null) { // same parent
if (parent != previousParent) {
hunkActionSets = new ArrayList<>();
}
hunkActionSets.add(actionSet);
} else if (hunkActionSets.size() > 0) {
hunkActionSets = new ArrayList<>();
}
previousParent = parent;
}
}
addHunkActionSets(hunkActionSets, allHunkFixPatterns, hunk);
}
}
return allHunkFixPatterns;
}
private int getEndPosition(List<ITree> children) {
int endPosition = 0;
for (int i = 0, size = children.size(); i < size; i ++) {
ITree child = children.get(i);
int type = child.getType();
if (Checker.isStatement2(type)) {
if ( i > 0) {
child = children.get(i - 1);
endPosition = child.getPos() + child.getLength();
} else {
endPosition = child.getPos() - 1;
}
break;
}
}
return endPosition;
}
private void addHunkActionSets(List<HierarchicalActionSet> hunkActionSets, List<HunkFixPattern> allHunkFixPatterns, DiffEntryHunk hunk) {
if (hunkActionSets.size() > 0) {
HunkFixPattern hunkFixPattern = new HunkFixPattern(hunk, hunkActionSets);
allHunkFixPatterns.add(hunkFixPattern);
}
}
private ITree addToHunkActionSets(HierarchicalActionSet actionSet, List<HierarchicalActionSet> hunkActionSets,
List<HunkFixPattern> allHunkFixPatterns, ITree previousParent, DiffEntryHunk hunk) {
String astNodeType = actionSet.getAstNodeType();
if ("FieldDeclaration".equals(astNodeType)) {
addHunkActionSets(hunkActionSets, allHunkFixPatterns, hunk);
hunkActionSets = new ArrayList<>();
hunkActionSets.add(actionSet);
HunkFixPattern hunkFixPattern = new HunkFixPattern(hunk, hunkActionSets);
allHunkFixPatterns.add(hunkFixPattern);
return null;
} else {
ITree currentParent = actionSet.getNode().getParent();
if (previousParent == null) {
previousParent = currentParent;
} else {
if (!previousParent.equals(currentParent)) {
HunkFixPattern hunkFixPattern = new HunkFixPattern(hunk, hunkActionSets);
allHunkFixPatterns.add(hunkFixPattern);
previousParent = currentParent;
}
}
return previousParent;
}
}
private List<Move> getFirstAndLastMoveAction(HierarchicalActionSet gumTreeResult) {
List<Move> firstAndLastMoveActions = new ArrayList<>();
List<HierarchicalActionSet> actions = new ArrayList<>();
actions.addAll(gumTreeResult.getSubActions());
if (actions.size() == 0) {
return null;
}
Move firstMoveAction = null;
Move lastMoveAction = null;
while (actions.size() > 0) {
List<HierarchicalActionSet> subActions = new ArrayList<>();
for (HierarchicalActionSet action : actions) {
subActions.addAll(action.getSubActions());
if (action.toString().startsWith("MOV")) {
if (firstMoveAction == null) {
firstMoveAction = (Move) action.getAction();
lastMoveAction = (Move) action.getAction();
} else {
int startPosition = action.getStartPosition();
int length = action.getLength();
int startPositionFirst = firstMoveAction.getPosition();
int startPositionLast = lastMoveAction.getPosition();
int lengthLast = lastMoveAction.getNode().getLength();
if (startPosition < startPositionFirst || (startPosition == startPositionFirst && length > firstMoveAction.getLength())) {
firstMoveAction = (Move) action.getAction();
}
if ((startPosition + length) > (startPositionLast + lengthLast)) {
lastMoveAction = (Move) action.getAction();
}
}
}
}
actions.clear();
actions.addAll(subActions);
}
if (firstMoveAction == null) {
return null;
}
firstAndLastMoveActions.add(firstMoveAction);
firstAndLastMoveActions.add(lastMoveAction);
return firstAndLastMoveActions;
}
/**
* Filter out the modify actions, which are not in the DiffEntry hunks, without considering the same parent node.
*
* @param violations
* @param actionSets
* @param revFile
* @param prevFile
* @return
*/
public List<Violation> filterActionsByModifiedRange(List<Violation> violations,
List<HierarchicalActionSet> actionSets, File revFile, File prevFile) {
List<Violation> selectedViolations = new ArrayList<>();
CUCreator cuCreator = new CUCreator();
CompilationUnit prevUnit = cuCreator.createCompilationUnit(prevFile);
CompilationUnit revUnit = cuCreator.createCompilationUnit(revFile);
if (prevUnit == null || revUnit == null) {
return selectedViolations;
}
for (Violation violation : violations) {
int startLine = violation.getStartLineNum();
int endLine = violation.getEndLineNum();
int bugStartLine = violation.getBugStartLineNum();
int bugEndLine = violation.getBugEndLineNum();
int fixStartLine = violation.getFixStartLineNum();
int fixEndLine = violation.getFixEndLineNum();
for (HierarchicalActionSet actionSet : actionSets) {
int actionBugStartLine = actionSet.getBugStartLineNum();
if (actionBugStartLine == 0) {
actionBugStartLine = setLineNumbers(actionSet, prevUnit, revUnit);
}
int actionBugEndLine = actionSet.getBugEndLineNum();
int actionFixStartLine = actionSet.getFixStartLineNum();
int actionFixEndLine = actionSet.getFixEndLineNum();
String actionStr = actionSet.getActionString();
if (actionStr.startsWith("INS")) {
if (fixStartLine <= actionFixStartLine && actionFixEndLine <= fixEndLine) {
if (actionBugStartLine != 0) {
if (startLine <= actionBugEndLine && endLine >= actionBugStartLine) {
violation.getActionSets().add(actionSet);
}
} else {
violation.getActionSets().add(actionSet);
}
}
} else {
// if (bugEndLine < actionBugStartLine) {
// break;
// }
if (bugStartLine <= actionBugStartLine && actionBugEndLine <= bugEndLine) {
if (startLine <= actionBugEndLine && endLine >= actionBugStartLine) {
violation.getActionSets().add(actionSet);
}
}
}
}
if (violation.getActionSets().size() > 0) {
selectedViolations.add(violation);
}
}
return selectedViolations;
}
/**
* Filter out the modify actions, which are not in the DiffEntry hunks, with considering the same parent node.
*
* @param violations
* @param actionSets
* @param revFile
* @param prevFile
* @return
*/
public List<Violation> filterActionsByModifiedRange2(List<Violation> violations,
List<HierarchicalActionSet> actionSets, File revFile, File prevFile) {
List<Violation> selectedViolations = new ArrayList<>();
CUCreator cuCreator = new CUCreator();
CompilationUnit prevUnit = cuCreator.createCompilationUnit(prevFile);
CompilationUnit revUnit = cuCreator.createCompilationUnit(revFile);
if (prevUnit == null || revUnit == null) {
for (Violation violation : violations) {
this.unfixedViolations += "#NullMatchedGumTreeResult:" + revFile.getName() + ":" +violation.getStartLineNum() + ":" +
violation.getEndLineNum() + ":" + violation.getViolationType() + "\n";
}
return selectedViolations;
}
for (Violation violation : violations) {
int violationStartLine = violation.getStartLineNum();
int violationEndLine = violation.getEndLineNum();
int bugHunkStartLine = violation.getBugStartLineNum();
if (!specialVioaltionTypes(violation, actionSets, prevUnit, revUnit)) {
if (bugHunkStartLine == 0) {// Null source code matched for this violation.
// String type = getType(violation);
} else if (bugHunkStartLine == -1) {//
// specialVioaltionTypes(violation, actionSets, prevUnit, revUnit);
} else {
int bugHunkEndLine = violation.getBugEndLineNum();
int fixHunkStartLine = violation.getFixStartLineNum();
int fixHunkEndLine = violation.getFixEndLineNum();
int bugFixStartLineNum = violation.getBugFixStartLineNum();
int bugFixEndLineNum = violation.getBugFixEndLineNum();
for (HierarchicalActionSet actionSet : actionSets) {
int actionBugStartLine = actionSet.getBugStartLineNum();
if (actionBugStartLine == 0) {
actionBugStartLine = setLineNumbers(actionSet, prevUnit, revUnit);
}
int actionBugEndLine = actionSet.getBugEndLineNum();
int actionFixStartLine = actionSet.getFixStartLineNum();
int actionFixEndLine = actionSet.getFixEndLineNum();
String actionStr = actionSet.getActionString();
if (actionStr.startsWith("INS")) {
if (fixHunkStartLine <= actionFixStartLine && actionFixEndLine <= fixHunkEndLine) {
if (actionBugStartLine != 0) {
if (violationStartLine <= actionBugEndLine && violationEndLine >= actionBugStartLine) {
violation.getActionSets().add(actionSet);
continue;
}
}
// INS with MOV actions that are not identified in previous IF predicate, and pure INS actions
if (bugFixStartLineNum >= actionFixEndLine && actionFixStartLine <= bugFixEndLineNum && Math.abs(bugFixStartLineNum - actionFixStartLine) <= 3
&& Math.abs(bugFixEndLineNum - actionFixEndLine) <= 3) {
violation.getActionSets().add(actionSet);
}
}
} else {//if (!actionStr.startsWith("MOV")){// ignore move actions.
if (bugHunkStartLine <= actionBugStartLine && violationEndLine <= bugHunkEndLine) {
if (violationStartLine <= actionBugEndLine && violationEndLine >= actionBugStartLine) {
violation.getActionSets().add(actionSet);
}
}
}
}
}
}
if (violation.getActionSets().size() > 0) {
selectedViolations.add(violation);
} else {
this.unfixedViolations += "#NullMatchedGumTreeResult:" + revFile.getName() + ":" +violation.getStartLineNum() + ":" +
violation.getEndLineNum() + ":" + violation.getViolationType() + "\n";
}
}
return selectedViolations;
}
public String unfixedViolations = "";
private boolean specialVioaltionTypes(Violation violation, List<HierarchicalActionSet> actionSets, CompilationUnit prevUnit, CompilationUnit revUnit) {
String type = violation.getViolationType();
if ("NM_METHOD_NAMING_CONVENTION".equals(type) || "SE_NO_SUITABLE_CONSTRUCTOR_FOR_EXTERNALIZATION".equals(type)) {
int startLine = violation.getStartLineNum();
for (HierarchicalActionSet actionSet : actionSets) {
int actionBugStartLine = actionSet.getBugStartLineNum();
if (actionBugStartLine == 0) {
actionBugStartLine = setLineNumbers(actionSet, prevUnit, revUnit);
}
if (actionSet.getActionString().startsWith("UPD MethodDeclaration@@")) {
if (Math.abs(startLine - actionBugStartLine) <= 2) {
violation.getActionSets().add(actionSet);
break;
}
}
}
} else if ("EQ_DOESNT_OVERRIDE_EQUALS".equals(type)) {
for (HierarchicalActionSet actionSet : actionSets) {
int actionBugStartLine = actionSet.getBugStartLineNum();
if (actionBugStartLine == 0) {
actionBugStartLine = setLineNumbers(actionSet, prevUnit, revUnit);
}
if (actionSet.getActionString().startsWith("INS MethodDeclaration@@MethodName:equals")) {
violation.getActionSets().add(actionSet);
break;
}
}
} else if ("HE_EQUALS_USE_HASHCODE".equals(type) || "HE_INHERITS_EQUALS_USE_HASHCODE".equals(type)) {
for (HierarchicalActionSet actionSet : actionSets) {
int actionBugStartLine = actionSet.getBugStartLineNum();
if (actionBugStartLine == 0) {
actionBugStartLine = setLineNumbers(actionSet, prevUnit, revUnit);
}
if (actionSet.getActionString().startsWith("INS MethodDeclaration@@MethodName:hashCode")) {
violation.getActionSets().add(actionSet);
break;
}
}
} else if ("SE_NO_SUITABLE_CONSTRUCTOR".equals(type) || "RI_REDUNDANT_INTERFACES".equals(type)) {
int startLine = violation.getStartLineNum();
for (HierarchicalActionSet actionSet : actionSets) {
int actionBugStartLine = actionSet.getBugStartLineNum();
if (actionBugStartLine == 0) {
actionBugStartLine = setLineNumbers(actionSet, prevUnit, revUnit);
}
if (actionSet.getActionString().startsWith("UPD TypeDeclaration@@")) {
if (Math.abs(startLine - actionBugStartLine) <= 2) {
violation.getActionSets().add(actionSet);
break;
}
}
}
} else if ("CN_IDIOM".equals(type)) { // 202 23
//add clone method. or update clone method
for (HierarchicalActionSet actionSet : actionSets) {
int actionBugStartLine = actionSet.getBugStartLineNum();
if (actionBugStartLine == 0) {
actionBugStartLine = setLineNumbers(actionSet, prevUnit, revUnit);
}
if (actionSet.getActionString().startsWith("INS MethodDeclaration@@MethodName:clone")) {
// || actionSet.getActionString().startsWith("UPD MethodDeclaration@@clone")) {
violation.getActionSets().add(actionSet);
break;
}
}
} else if ("SE_NO_SERIALVERSIONID".equals(type) || "SE_COMPARATOR_SHOULD_BE_SERIALIZABLE".equals(type)) {// 12 1960
// change superclass or interface, add field or remove @SuppressWarnings("serial"), some are inner class
for (HierarchicalActionSet actionSet : actionSets) {
int actionBugStartLine = actionSet.getBugStartLineNum();
if (actionBugStartLine == 0) {
actionBugStartLine = setLineNumbers(actionSet, prevUnit, revUnit);
}
if (actionSet.getActionString().startsWith("INS FieldDeclaration@") && actionSet.getNode().getLabel().contains("serialVersionUID")) {
violation.getActionSets().add(actionSet);
break;
}
if (actionSet.getActionString().startsWith("UPD TypeDeclaration@")) {
int startLine = violation.getStartLineNum();
if (Math.abs(startLine - actionBugStartLine) <= 2) {
violation.getActionSets().add(actionSet);
break;
}
}
}
} else {
return false;
}
return true;
}
private String getType(Violation violation) {
String type = violation.getViolationType();
switch (type) {
case "CI_CONFUSED_INHERITANCE":// field
// update fieldDeclaration
break;
case "CO_ABSTRACT_SELF":
// java file is an interface, and delete compareTo().
break;
case "EQ_ABSTRACT_SELF":
// java file is an interface, and delete equals().
break;
case "SE_NO_SERIALVERSIONID":
// add a field: serialVersionUID
break;
case "EQ_COMPARETO_USE_OBJECT_EQUALS":
//Update or Delete compareTo(), Add equals()
break;
case "EQ_DOESNT_OVERRIDE_EQUALS":
//override equals()
break;
case "HE_SIGNATURE_DECLARES_HASHING_OF_UNHASHABLE_CLASS":
//remove equals()
break;
case "ME_MUTABLE_ENUM_FIELD":
// under enum, field add final keyword
break;
case "MF_CLASS_MASKS_FIELD":
// change super class or delete the field with a same name in super class.
break;
case "MS_SHOULD_BE_FINAL":
// add final to field
break;
case "STCAL_STATIC_SIMPLE_DATE_FORMAT_INSTANCE":
//remove public static final DateFormat DATE_FORMAT.... or SimpleDateFormat
break;
case "UUF_UNUSED_FIELD":
//remove unused fields. not sure
break;
case "UUF_UNUSED_PUBLIC_OR_PROTECTED_FIELD":
//remove unused fields. not sure
case "UWF_NULL_FIELD":
//update field, remove field
break;
case "UWF_UNWRITTEN_FIELD":
//field
break;
case "UWF_UNWRITTEN_PUBLIC_OR_PROTECTED_FIELD":
//remove field
break;
case "VO_VOLATILE_REFERENCE_TO_ARRAY":
//field
break;
default:
break;
}
return null;
}
/**
* Filter out the modify actions, which are not in the DiffEntry hunks, without considering DiffEntry hunks.
*
* @param violations
* @param actionSets
* @param revFile
* @param prevFile
* @return
*/
public List<Violation> filterActionsByModifiedRange3(List<Violation> violations,
List<HierarchicalActionSet> actionSets, File revFile, File prevFile) {
List<Violation> selectedViolations = new ArrayList<>();
CUCreator cuCreator = new CUCreator();
CompilationUnit prevUnit = cuCreator.createCompilationUnit(prevFile);
CompilationUnit revUnit = cuCreator.createCompilationUnit(revFile);
if (prevUnit == null || revUnit == null) {
return selectedViolations;
}
for (Violation violation : violations) {
int startLine = violation.getStartLineNum();
int endLine = violation.getEndLineNum();
// ITree parent = null;
// List<HierarchicalActionSet> actionSetsWithSameParent = new ArrayList<>(); //TODO
for (HierarchicalActionSet actionSet : actionSets) {
int actionBugStartLine = actionSet.getBugStartLineNum();
if (actionBugStartLine == 0) {
actionBugStartLine = setLineNumbers(actionSet, prevUnit, revUnit);
}
int actionBugEndLine = actionSet.getBugEndLineNum();
int actionFixStartLine = actionSet.getFixStartLineNum();
int actionFixEndLine = actionSet.getFixEndLineNum();
String actionStr = actionSet.getActionString();
if (actionStr.startsWith("INS")) { // FIXME It is impossible to locate the INS action by the buggy line range.
if (startLine <= actionFixStartLine && actionFixEndLine <= endLine) {
if (actionBugStartLine != 0) {
if (startLine <= actionBugEndLine && endLine >= actionBugStartLine) {
violation.getActionSets().add(actionSet);
}
} else {
violation.getActionSets().add(actionSet);
}
}
} else {
// if (endLine < actionBugStartLine) {
// break;
// }
if (startLine <= actionBugStartLine && actionBugEndLine <= endLine) {
if (startLine <= actionBugEndLine && endLine >= actionBugStartLine) {
violation.getActionSets().add(actionSet);
}
}
}
}
if (violation.getActionSets().size() > 0) {
selectedViolations.add(violation);
}
}
return selectedViolations;
}
private int setLineNumbers(HierarchicalActionSet actionSet, CompilationUnit prevUnit, CompilationUnit revUnit) {
int actionBugStartLine;
int actionBugEndLine;
int actionFixStartLine;
int actionFixEndLine;
// position of buggy statements
int bugStartPosition = 0;
int bugEndPosition = 0;
// position of fixed statements
int fixStartPosition = 0;
int fixEndPosition = 0;
String actionStr = actionSet.getActionString();
if (actionStr.startsWith("INS")) {
fixStartPosition = actionSet.getStartPosition();
fixEndPosition = fixStartPosition + actionSet.getLength();
List<Move> firstAndLastMov = getFirstAndLastMoveAction(actionSet);
if (firstAndLastMov != null) {
bugStartPosition = firstAndLastMov.get(0).getNode().getPos();
ITree lastTree = firstAndLastMov.get(1).getNode();
bugEndPosition = lastTree.getPos() + lastTree.getLength();
}
} else {
bugStartPosition = actionSet.getStartPosition(); // range of actions
bugEndPosition = bugStartPosition + actionSet.getLength();
if (actionStr.startsWith("UPD")) {
Update update = (Update) actionSet.getAction();
ITree newNode = update.getNewNode();
fixStartPosition = newNode.getPos();
fixEndPosition = fixStartPosition + newNode.getLength();
String astNodeType = actionSet.getAstNodeType();
// if (Checker.withBlockStatement(newNode.getType())) {
//// List<ITree> children = update.getNode().getChildren();
//// bugEndPosition = getEndPosition(children);
//// List<ITree> newChildren = newNode.getChildren();
//// fixEndPosition = getEndPosition(newChildren);
// } else
if ("TypeDeclaration".equals(astNodeType)) {
bugEndPosition = getClassBodyStartPosition(update.getNode());
fixEndPosition = getClassBodyStartPosition(newNode);
} else if ("MethodDeclaration".equals(astNodeType)) {
List<ITree> children = update.getNode().getChildren();
bugEndPosition = getEndPosition(children);
List<ITree> newChildren = newNode.getChildren();
fixEndPosition = getEndPosition(newChildren);
}
if (fixEndPosition == 0) {
fixEndPosition = fixStartPosition + newNode.getLength();
}
} else if (actionStr.startsWith("DEL")) {
ITree buggyTree = actionSet.getNode();
int type = buggyTree.getType();
if (type == 55) { // TypeDeclaration
bugEndPosition = getClassBodyStartPosition(buggyTree);
} else if (type == 31 || Checker.withBlockStatement(type)) {//MethodDeclaration && Block-Statements
List<ITree> children = buggyTree.getChildren();
bugEndPosition = getEndPosition(children);
}
}
if (bugEndPosition == 0) {
bugEndPosition = bugStartPosition + actionSet.getLength();
}
}
actionBugStartLine = bugStartPosition == 0 ? 0 : prevUnit.getLineNumber(bugStartPosition);
actionBugEndLine = bugEndPosition == 0 ? 0 : prevUnit.getLineNumber(bugEndPosition);
actionFixStartLine = fixStartPosition == 0 ? 0 : revUnit.getLineNumber(fixStartPosition);
actionFixEndLine = fixEndPosition == 0 ? 0 : revUnit.getLineNumber(fixEndPosition);
actionSet.setBugStartLineNum(actionBugStartLine);
actionSet.setBugEndLineNum(actionBugEndLine);
actionSet.setFixStartLineNum(actionFixStartLine);
actionSet.setFixEndLineNum(actionFixEndLine);
actionSet.setBugEndPosition(bugEndPosition);
actionSet.setFixEndPosition(fixEndPosition);
return actionBugStartLine;
}
private int getClassBodyStartPosition(ITree tree) {
List<ITree> children = tree.getChildren();
for (int i = 0, size = children.size(); i < size; i ++) {
ITree child = children.get(i);
int type = child.getType();
// Modifier, NormalAnnotation, MarkerAnnotation, SingleMemberAnnotation
if (type != 83 && type != 77 && type != 78 && type != 79
&& type != 5 && type != 39 && type != 43 && type != 74 && type != 75
&& type != 76 && type != 84 && type != 87 && type != 88 && type != 42) {
// ArrayType, PrimitiveType, SimpleType, ParameterizedType,
// QualifiedType, WildcardType, UnionType, IntersectionType, NameQualifiedType, SimpleName
if (i > 0) {
child = children.get(i - 1);
return child.getPos() + child.getLength() + 1;
} else {
return child.getPos() - 1;
}
}
}
return 0;
}
}
src/main/java/edu/lu/uni/serval/gumtree/regroup/HunkFixPattern.java
-27
View File
@@ -1,27 +0,0 @@
package edu.lu.uni.serval.gumtree.regroup;
import java.util.List;
import edu.lu.uni.serval.diffentry.DiffEntryHunk;
public class HunkFixPattern {
private DiffEntryHunk hunk;
private List<HierarchicalActionSet> hunkActionSets;
public HunkFixPattern(DiffEntryHunk hunk, List<HierarchicalActionSet> hunkActionSets) {
super();
this.hunk = hunk;
this.hunkActionSets = hunkActionSets;
}
public DiffEntryHunk getHunk() {
return hunk;
}
public List<HierarchicalActionSet> getHunkActionSets() {
return hunkActionSets;
}
}
src/main/java/edu/lu/uni/serval/gumtree/regroup/SimpleTree.java
-68
View File
@@ -1,68 +0,0 @@
package edu.lu.uni.serval.gumtree.regroup;
import java.util.ArrayList;
import java.util.List;
public class SimpleTree {
private String nodeType;
private String label;
private SimpleTree parent;
private List<SimpleTree> children = new ArrayList<>();
public String getNodeType() {
return nodeType;
}
public void setNodeType(String nodeType) {
this.nodeType = nodeType;
}
public String getLabel() {
return label;
}
public void setLabel(String label) {
this.label = label;
}
public SimpleTree getParent() {
return parent;
}
public void setParent(SimpleTree parent) {
this.parent = parent;
}
public List<SimpleTree> getChildren() {
return children;
}
public void setChildren(List<SimpleTree> children) {
this.children = children;
}
private List<String> strList = new ArrayList<>();
@Override
public String toString() {
String str = this.nodeType + "@@" + this.label;
if (strList.size() == 0) {
strList.add(str);
for (SimpleTree child : children) {
child.toString();
List<String> strList1 = child.strList;
for (String str1 : strList1) {
strList.add("------" + str1);
}
}
}
str = "";
for (String str1 : strList) {
str += str1 + "\n";
}
return str;
}
}
src/main/java/edu/lu/uni/serval/gumtree/regroup/SimplifyTree.java
-845
View File
@@ -1,845 +0,0 @@
package edu.lu.uni.serval.gumtree.regroup;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import com.github.gumtreediff.actions.model.Action;
import com.github.gumtreediff.tree.ITree;
import edu.lu.uni.serval.FixPattern.utils.ASTNodeMap;
import edu.lu.uni.serval.FixPattern.utils.Checker;
import edu.lu.uni.serval.utils.ListSorter;
/**
* Simplify the ITree of source code into a simple tree.
*
* @author kui.liu
*
*/
public class SimplifyTree {
private static final String ABSTRACT_TYPE = "T";
private static final String ABSTRACT_NAME = "N";
private static final String ABSTRACT_METHOD = "m";
private static final String ABSTRACT_VARIABLE = "v";
private Map<String, String> abstractTypeIdentifiers = new HashMap<>();
private Map<String, String> abstractMethodIdentifiers = new HashMap<>();
private Map<String, String> abstractNameIdentifiers = new HashMap<>();
private Map<String, String> abstractVariableIdentifiers = new HashMap<>();
/**
* Convert ITree to a source code simple tree, an abstract identifier simple tree, and a semi-source code simple tree.
*
* @param actionSet
*/
public void abstractTree(HierarchicalActionSet actionSet) {
SimpleTree sourceCodeSimpleTree = null; // source code tree and AST node type tree
SimpleTree abstractIdentifierTree = null; // abstract identifier tree
SimpleTree abstractSimpleTree = null; // semi-source code tree. and AST node type tree
SimpleTree simpleTree = null; // source code tree with canonical variable names.
if (actionSet.getActionString().startsWith("INS")) {
List<Action> allMoveActions = getAllMoveActions(actionSet);
if (allMoveActions != null) {
List<Action> actions = new ArrayList<>();
for (Action action : allMoveActions) {
boolean hasParent = false;
ITree parent = action.getNode().getParent();
for (Action act : allMoveActions) {
if (act == action) continue;
ITree actNode = act.getNode();
if (actNode.equals(parent)) {
hasParent = true;
break;
}
}
if (!hasParent) {
actions.add(action);
}
}
// sourceCodeSimpleTree = sourceCodeTree(actions);
simpleTree = canonicalizeSourceCodeTree(actions, null);
}
} else {
ITree tree = actionSet.getNode();
String astNodeType = actionSet.getAstNodeType();
if (Checker.containsBodyBlock(astNodeType)) {
// delete the body block.
List<ITree> children = tree.getChildren();
List<ITree> newChildren = new ArrayList<>();
for (ITree child : children) {
if (!child.getLabel().endsWith("Body")) {
newChildren.add(child);
}
}
tree.setChildren(newChildren);
}
// sourceCodeSimpleTree = originalSourceCodeTree(tree, null);
// abstractIdentifierTree = abstractIdentifierTree(actionSet, tree, null);
// abstractSimpleTree = semiSourceCodeTree(actionSet, tree, null);
simpleTree = canonicalizeSourceCodeTree(tree, null);
}
actionSet.setAbstractSimpleTree(abstractSimpleTree);
actionSet.setAbstractIdentifierTree(abstractIdentifierTree);
actionSet.setSimpleTree(simpleTree);
actionSet.setOriginalTree(sourceCodeSimpleTree);
}
/**
* Convert ITree to a source code simple tree, an abstract identifier simple tree, and a semi-source code simple tree.
*
* @param actionSet
*/
public void abstractTree(HierarchicalActionSet actionSet, int bugEndPosition) {
SimpleTree sourceCodeSimpleTree = null; // source code tree and AST node type tree
SimpleTree abstractIdentifierTree = null; // abstract identifier tree
SimpleTree abstractSimpleTree = null; // semi-source code tree. and AST node type tree
SimpleTree simpleTree = null; // source code tree with canonical variable names.
if (actionSet.getActionString().startsWith("INS")) {
List<Action> allMoveActions = getAllMoveActions(actionSet);
if (allMoveActions != null) {
List<Action> actions = new ArrayList<>();
for (Action action : allMoveActions) {
boolean hasParent = false;
ITree parent = action.getNode().getParent();
for (Action act : allMoveActions) {
if (act == action) continue;
ITree actNode = act.getNode();
if (actNode.equals(parent)) {
hasParent = true;
break;
}
}
if (!hasParent) {
actions.add(action);
}
}
// sourceCodeSimpleTree = sourceCodeTree(actions);
simpleTree = canonicalizeSourceCodeTree(actions, null);
}
} else {
ITree tree = actionSet.getNode();
// String astNodeType = actionSet.getAstNodeType();
// if (Checker.containsBodyBlock(astNodeType)) {
// // delete the body block.
// List<ITree> children = tree.getChildren();
// List<ITree> newChildren = new ArrayList<>();
// for (ITree child : children) {
// if (!child.getLabel().endsWith("Body")) {
// newChildren.add(child);
// }
// }
// tree.setChildren(newChildren);
// }
// sourceCodeSimpleTree = originalSourceCodeTree(tree, null);
// abstractIdentifierTree = abstractIdentifierTree(actionSet, tree, null);
// abstractSimpleTree = semiSourceCodeTree(actionSet, tree, null);
// if (actionSet.getActionString().startsWith("UPD")) {
// simpleTree = canonicalizeSourceCodeTree(tree, null, bugEndPosition);
// } else {
// simpleTree = canonicalizeSourceCodeTree(tree, null);
// }
simpleTree = canonicalizeSourceCodeTree(tree, null, bugEndPosition);
}
actionSet.setAbstractSimpleTree(abstractSimpleTree);
actionSet.setAbstractIdentifierTree(abstractIdentifierTree);
actionSet.setSimpleTree(simpleTree);
actionSet.setOriginalTree(sourceCodeSimpleTree);
}
private SimpleTree canonicalizeSourceCodeTree(List<Action> actions, SimpleTree parent) {
SimpleTree simpleTree = new SimpleTree();
simpleTree.setLabel("Block");
simpleTree.setNodeType("Block");
simpleTree.setParent(parent);
List<SimpleTree> children = new ArrayList<>();
for (Action action : actions) {
ITree node = action.getNode();
children.add(canonicalizeSourceCodeTree(node, simpleTree));
}
simpleTree.setChildren(children);
return simpleTree;
}
public SimpleTree canonicalizeSourceCodeTree(ITree tree, SimpleTree parent) {
SimpleTree simpleTree = new SimpleTree();
String label = tree.getLabel();
String astNode = ASTNodeMap.map.get(tree.getType());
List<ITree> children = tree.getChildren();
if (children.size() > 0) {
simpleTree.setNodeType(astNode);
if (astNode.endsWith("Type")) {
simpleTree.setLabel(canonicalizeTypeStr(label).replaceAll(" ", ""));
} else {
if ((astNode.equals("SimpleName") || astNode.equals("MethodInvocation"))) {
if (label.startsWith("MethodName:")) {
simpleTree.setNodeType("MethodName");
label = label.substring(11);
int argusIndex = label.indexOf(":[");
if (argusIndex > 0) {
label = label.substring(0, argusIndex);
}
} else if (label.startsWith("ClassName:")) {
simpleTree.setNodeType("ClassName");
label = label.substring(10);
}
simpleTree.setLabel(label);
} else {
simpleTree.setLabel(astNode);
}
List<SimpleTree> subTrees = new ArrayList<>();
for (ITree child : children) {
subTrees.add(canonicalizeSourceCodeTree(child, simpleTree));
}
simpleTree.setChildren(subTrees);
}
} else {
if (astNode.endsWith("Name")) {
// variableName, methodName, QualifiedName
if (label.startsWith("MethodName:")) { // <MethodName, name>
simpleTree.setNodeType("MethodName");
label = label.substring(11);
int argusIndex = label.indexOf(":[");
if (argusIndex > 0) {
label = label.substring(0, argusIndex);
}
simpleTree.setLabel(label);
} else if (label.startsWith("ClassName:")) {
simpleTree.setNodeType("ClassName");
label = label.substring(10);
simpleTree.setLabel(label);
} else if (label.startsWith("Name:")) {
label = label.substring(5);
char firstChar = label.charAt(0);
if (Character.isUpperCase(firstChar)) {
simpleTree.setNodeType("Name");
simpleTree.setLabel(label);
} else {// variableName: <VariableName, canonicalName>
simpleTree.setNodeType("VariableName");
simpleTree.setLabel(canonicalVariableName(label, tree));
}
} else {// variableName: <VariableName, canonicalName>
simpleTree.setNodeType("VariableName");
simpleTree.setLabel(canonicalVariableName(label, tree));
}
} else {
simpleTree.setNodeType(astNode);
if (astNode.endsWith("Type")) {
simpleTree.setLabel(canonicalizeTypeStr(label).replaceAll(" ", ""));
} else if (astNode.startsWith("Type")) {
simpleTree.setLabel(canonicalizeTypeStr(label).replaceAll(" ", ""));
} else if ((astNode.equals("SimpleName") || astNode.equals("MethodInvocation")) && label.startsWith("MethodName:")) {
simpleTree.setNodeType("MethodName");
label = label.substring(11);
int argusIndex = label.indexOf(":[");
if (argusIndex > 0) {
label = label.substring(0, argusIndex);
}
simpleTree.setLabel(label);
} else {
simpleTree.setLabel(label.replaceAll(" ", ""));
}
}
}
simpleTree.setParent(parent);
return simpleTree;
}
public SimpleTree canonicalizeSourceCodeTree(ITree tree, SimpleTree parent, int bugEndPosition) {
SimpleTree simpleTree = new SimpleTree();
String label = tree.getLabel();
String astNode = ASTNodeMap.map.get(tree.getType());
List<ITree> children = tree.getChildren();
if (children.size() > 0) {
simpleTree.setNodeType(astNode);
if (astNode.endsWith("Type")) {
simpleTree.setLabel(canonicalizeTypeStr(label).replaceAll(" ", ""));
} else {
if ((astNode.equals("SimpleName") || astNode.equals("MethodInvocation"))) {
if (label.startsWith("MethodName:")) {
simpleTree.setNodeType("MethodName");
label = label.substring(11);
int argusIndex = label.indexOf(":[");
if (argusIndex > 0) {
label = label.substring(0, argusIndex);
}
} else if (label.startsWith("ClassName:")) {
simpleTree.setNodeType("ClassName");
label = label.substring(10);
}
simpleTree.setLabel(label);
} else {
simpleTree.setLabel(astNode);
}
List<SimpleTree> subTrees = new ArrayList<>();
for (ITree child : children) {
if (child.getPos() > bugEndPosition) continue;
subTrees.add(canonicalizeSourceCodeTree(child, simpleTree));
}
simpleTree.setChildren(subTrees);
}
} else {
if (astNode.endsWith("Name")) {
// variableName, methodName, QualifiedName
if (label.startsWith("MethodName:")) { // <MethodName, name>
simpleTree.setNodeType("MethodName");
label = label.substring(11);
int argusIndex = label.indexOf(":[");
if (argusIndex > 0) {
label = label.substring(0, argusIndex);
}
simpleTree.setLabel(label);
} else if (label.startsWith("ClassName:")) {
simpleTree.setNodeType("ClassName");
label = label.substring(10);
simpleTree.setLabel(label);
} else if (label.startsWith("Name:")) {
label = label.substring(5);
char firstChar = label.charAt(0);
if (Character.isUpperCase(firstChar)) {
simpleTree.setNodeType("Name");
simpleTree.setLabel(label);
} else {// variableName: <VariableName, canonicalName>
simpleTree.setNodeType("VariableName");
simpleTree.setLabel(canonicalVariableName(label, tree));
}
} else {// variableName: <VariableName, canonicalName>
simpleTree.setNodeType("VariableName");
simpleTree.setLabel(canonicalVariableName(label, tree));
}
} else {
simpleTree.setNodeType(astNode);
if (astNode.endsWith("Type")) {
simpleTree.setLabel(canonicalizeTypeStr(label).replaceAll(" ", ""));
} else if (astNode.startsWith("Type")) {
simpleTree.setLabel(canonicalizeTypeStr(label).replaceAll(" ", ""));
} else if ((astNode.equals("SimpleName") || astNode.equals("MethodInvocation")) && label.startsWith("MethodName:")) {
simpleTree.setNodeType("MethodName");
label = label.substring(11);
int argusIndex = label.indexOf(":[");
if (argusIndex > 0) {
label = label.substring(0, argusIndex);
}
simpleTree.setLabel(label);
} else {
simpleTree.setLabel(label.replaceAll(" ", ""));
}
}
}
simpleTree.setParent(parent);
return simpleTree;
}
public String canonicalVariableName(String label, ITree tree) {
ITree parent = tree.getParent();
if (parent == null) {
return label;
} else {
String matchStr = null;
int parentType = parent.getType();
if (parentType == 44) { // SingleVariableDeclaration
matchStr = matchSingleVariableDeclaration(parent, label);
} else if (parentType == 23 || parentType == 58 || parentType == 60) {
//FieldDeclaration, VariableDeclarationExpression, VariableDeclarationStatement
matchStr = matchVariableDeclarationExpression(parent, label);
} else if (parentType == 31) { // MethodDeclaration
List<ITree> children = parent.getChildren();
int index = children.indexOf(tree);
for (int i = index - 1; i >=0; i --) {
ITree child = children.get(i);
int childType = child.getType();
if (childType == 60) { // VariableDeclarationStatement
matchStr = matchVariableDeclarationExpression(child, label);
} else if (childType == 44) { // SingleVariableDeclaration
matchStr = matchSingleVariableDeclaration(child, label);
}
if (matchStr != null) break;
}
} else if (parentType ==70 || parentType == 24 ||parentType == 12 || parentType == 54) {
// EnhancedForStatement, ForStatement, CatchClause, TryStatement
matchStr = matchStatements(parentType, parent, label);
} else if (parentType == 55) { // TypeDeclaration: Class Declaration
List<ITree> children = parent.getChildren();
int index = children.indexOf(tree);
for (int i = index - 1; i >=0; i --) {
ITree child = children.get(i);
if (child.getType() == 23) { // FieldDeclaration
matchStr = matchVariableDeclarationExpression(child, label);
if (matchStr != null) break;
}
}
} else if (parentType == 8) { // Block body
List<ITree> children = parent.getChildren();
int index = children.indexOf(tree);
for (int i = index - 1; i >=0; i --) {
ITree child = children.get(i);
if (child.getType() == 60) { // VariableDeclarationStatement
matchStr = matchVariableDeclarationExpression(child, label);
if (matchStr != null) break;
}
}
}
if (matchStr != null) {
return matchStr;
} else {
return canonicalVariableName(label, parent);
}
}
}
private String matchStatements(int typeInt, ITree tree, String label) {
String matchStr = null;
if (typeInt == 70) { // EnhancedForStatement
matchStr = matchSingleVariableDeclaration(tree.getChild(0), label);
} else if (typeInt == 24) { // ForStatement
List<ITree> children = tree.getChildren();
for (ITree child : children) {
if (child.getType() == 58) {
matchStr = matchVariableDeclarationExpression(child, label);
if (matchStr != null) break;
} else {
break;
}
}
} else if (typeInt == 12) { // CatchClause
matchStr = matchSingleVariableDeclaration(tree.getChild(0), label);
} else if (typeInt == 54) { // TryStatement
List<ITree> children = tree.getChildren();
for (ITree child : children) {
if (child.getType() == 58) { //VariableDeclarationExpression
matchStr = matchVariableDeclarationExpression(tree, label);
if (matchStr != null) break;
} else {
break;
}
}
}
return null;
}
private String matchVariableDeclarationExpression(ITree variable, String label) {
List<ITree> children = variable.getChildren();
ITree type = null;
for (int i = 0, size = children.size(); i < size; i ++) {
ITree child = children.get(i);
if (child.getType() == 59) {// VariableDeclarationFragment
if (type == null) {
type = children.get(i - 1);
}
ITree simpleName = child.getChild(0);
if (simpleName.getLabel().equals(label)) {
String typeStr = canonicalizeTypeStr(type.getLabel());
label = typeStr.toLowerCase() + "Var";
return label;
}
}
}
return null;
}
private String matchSingleVariableDeclaration(ITree singleVariable, String label) {
List<ITree> children = singleVariable.getChildren();
for (int i = 0, size = children.size(); i < size; i ++) {
ITree child = children.get(i);
if (child.getType() == 42) { // SimpleName
if (child.getLabel().equals(label)) {
ITree type = children.get(i - 1);
String typeStr = canonicalizeTypeStr(type.getLabel());
label = typeStr.toLowerCase() + "Var";
return label;
}
break;
}
}
return null;
}
private String canonicalizeTypeStr(String label) {
String typeStr = label;
int index1 = typeStr.indexOf("<");
if (index1 != -1) {
typeStr = typeStr.substring(0, index1);
}
index1 = typeStr.lastIndexOf(".");
if (index1 != -1) {
typeStr = typeStr.substring(index1 + 1);
}
return typeStr;
}
// public static String addPrefixByType(Type type) {
// String newName = "";
// if (type instanceof PrimitiveType) {
// // byte,short,char,int,long,float,double,boolean,void
// newName = type.toString().toLowerCase();
// } else if (type instanceof ArrayType) {
// // Type [ ]
// ArrayType at = (ArrayType) type;
// type = at.getElementType();
// if (type instanceof SimpleType || type instanceof PrimitiveType) {
// newName = getNewName(type);
// } else {
// newName = addPrefixByType(type);
// }
// } else if (type instanceof SimpleType) {
// // TypeName
// if (type.toString().equals("Integer")) {
// newName = "int";
// } else {
// newName = getNewName(type);
// }
// } else if (type instanceof QualifiedType) {
// // Type.SimpleName
// newName = ((QualifiedType) type).getName().toString().toLowerCase();
// } else if (type instanceof ParameterizedType) {
// // Type < Type { , Type } > 泛型
// ParameterizedType t = (ParameterizedType) type;
// newName = getNewName(t.getType());
// } else if (type instanceof WildcardType) {
// newName = "object";
// }
// return newName;
// }
//
// private static String getNewName(Type type) {
// String newName = "";
// String typeName = type.toString();
// int dot = typeName.lastIndexOf(".");
// if (dot > 0) {
// newName = typeName.substring(dot + 1).toString().toLowerCase();
// } else {
// newName = typeName.toString().toLowerCase();
// }
// return newName;
// }
/**
* Convert the Move actions of an INS action into a simple tree with AST nodes and leaf labels.
*
* @param actions
* @return
*/
private SimpleTree sourceCodeTree(List<Action> actions) {
if (actions.size() > 0) {
SimpleTree simpleTree = new SimpleTree();
simpleTree.setNodeType("Block");
simpleTree.setLabel("Block");
simpleTree.setParent(null);
List<SimpleTree> subTrees = new ArrayList<>();
for (Action action : actions) {
ITree node = action.getNode();
subTrees.add(sourceCodeTree(node, simpleTree));
}
simpleTree.setChildren(subTrees);
return simpleTree;
}
return null;
}
/**
* Convert a Move action into a simple tree with AST nodes and leaf labels.
*
* @param tree
* @param parent
* @return
*/
private SimpleTree sourceCodeTree(ITree tree, SimpleTree parent) {
SimpleTree simpleTree = new SimpleTree();
String astNode = ASTNodeMap.map.get(tree.getType());
do {
if (astNode.endsWith("Statement") || astNode.equals("FieldDeclaration")) break;
tree = tree.getParent();
astNode = ASTNodeMap.map.get(tree.getType());// FIXME if the ASTNode is a method declaration or class declaration?
} while (!astNode.endsWith("Statement") && !astNode.equals("FieldDeclaration"));
String label = tree.getLabel();
List<ITree> children = tree.getChildren();
if (children.size() > 0) {
List<SimpleTree> subTrees = new ArrayList<>();
for (ITree child : children) {
subTrees.add(sourceCodeTree(child, simpleTree));
}
simpleTree.setChildren(subTrees);
simpleTree.setLabel(astNode);
} else {
simpleTree.setLabel(label);
}
simpleTree.setNodeType(astNode);
simpleTree.setParent(parent);
return simpleTree;
}
/**
* Convert an UPD/DEL/MOV action into a simple tree with AST nodes and leaf labels.
*
* @param tree
* @param parent
* @return
*/
private SimpleTree originalSourceCodeTree(ITree tree, SimpleTree parent) {
SimpleTree simpleTree = new SimpleTree();
String label = tree.getLabel();
String astNode = ASTNodeMap.map.get(tree.getType());
simpleTree.setNodeType(astNode);
List<ITree> children = tree.getChildren();
if (children.size() > 0) {
List<SimpleTree> subTrees = new ArrayList<>();
for (ITree child : children) {
subTrees.add(originalSourceCodeTree(child, simpleTree));
}
simpleTree.setChildren(subTrees);
simpleTree.setLabel(astNode);
} else {
simpleTree.setLabel(label);
}
simpleTree.setParent(parent);
return simpleTree;
}
/**
* Convert an UPD/DEL/MOV action into a simple tree with abstract identifiers of AST nodes and abstract identifiers of leaf labels.
*
* @param actionSet
* @param tree
* @param parent
* @return
*/
private SimpleTree abstractIdentifierTree(ITree tree, SimpleTree parent) {
SimpleTree simpleTree = new SimpleTree();
String label = tree.getLabel();
String astNode = ASTNodeMap.map.get(tree.getType());
simpleTree.setNodeType(astNode);
List<ITree> children = tree.getChildren();
if (children.size() > 0) {
if (astNode.endsWith("Type")) {
simpleTree.setNodeType("Type");
simpleTree.setLabel(getAbstractLabel(abstractTypeIdentifiers, label, ABSTRACT_TYPE)); // abstract Type identifier
} else {
List<SimpleTree> subTrees = new ArrayList<>();
for (ITree child : children) {
subTrees.add(abstractIdentifierTree(child, simpleTree));
}
simpleTree.setChildren(subTrees);
simpleTree.setLabel(astNode);
}
} else {
if (astNode.endsWith("Type")) {
simpleTree.setNodeType("Type");
if (astNode.equals("WildcardType")) {
simpleTree.setLabel("?");
} else {
simpleTree.setLabel(getAbstractLabel(abstractTypeIdentifiers, label, ABSTRACT_TYPE)); // abstract Type identifier
}
} else if (astNode.endsWith("Name")) {
// variableName, methodName, QualifiedName
if (label.startsWith("MethodName:")) { // <Method, name>
label = label.substring(11);
simpleTree.setNodeType("Method");
simpleTree.setLabel(getAbstractLabel(abstractMethodIdentifiers, label, ABSTRACT_METHOD)); // abstract method identifier
} else if (label.startsWith("Name:")) {
label = label.substring(5);
String firstChar = label.substring(0, 1);
if (firstChar.equals(firstChar.toUpperCase())) {
simpleTree.setNodeType("Name");
simpleTree.setLabel(getAbstractLabel(abstractNameIdentifiers, label, ABSTRACT_NAME)); // abstract Name identifier
} else {// variableName: <Variable, var>
simpleTree.setNodeType("Variable");
simpleTree.setLabel(getAbstractLabel(abstractVariableIdentifiers, label, ABSTRACT_VARIABLE));// abstract Variable identifier
}
} else {// variableName: <Variable, var>
simpleTree.setNodeType("Variable");
simpleTree.setLabel(getAbstractLabel(abstractVariableIdentifiers, label, ABSTRACT_VARIABLE));// abstract Variable identifier
}
} else if (astNode.equals("BooleanLiteral") || astNode.equals("CharacterLiteral") || astNode.equals("NullLiteral")
|| astNode.equals("NumberLiteral") || astNode.equals("StringLiteral") || astNode.equals("ThisExpression")
|| astNode.equals("Modifier") || astNode.equals("Operator")) {
simpleTree.setNodeType(astNode);
simpleTree.setLabel(label);
}
}
simpleTree.setParent(parent);
return simpleTree;
}
/**
* Convert an UPD/DEL/MOV action into a semi-source code simple tree by abstracting the non-buggy code.
*
* @param actionSet
* @param tree
* @param parent
* @return
*/
private SimpleTree semiSourceCodeTree(HierarchicalActionSet actionSet, ITree tree, SimpleTree parent) {
SimpleTree simpleTree = new SimpleTree();
simpleTree.setParent(parent);
// deep first
abstractBuggyTreeDeepFirst(actionSet, tree, simpleTree);
return simpleTree;
}
private void abstractBuggyTreeDeepFirst(HierarchicalActionSet actionSet, ITree tree, SimpleTree simpleTree) {
List<ITree> children = tree.getChildren();
HierarchicalActionSet modifyAction = findHierarchicalActionSet(tree.getPos(), tree.getLength(), actionSet);
String label = tree.getLabel();
String astNode = ASTNodeMap.map.get(tree.getType());
if (Checker.isExpressionType(astNode)) {
if (modifyAction == null || !modifyAction.getActionString().contains("@@" + label)) {
simpleTree.setNodeType("Expression");
simpleTree.setLabel("EXP"); // astNode
}
} else {
if (astNode.endsWith("Type")) { // <Type, ?> TODO: sub Type
simpleTree.setNodeType("Type");
// simpleTree.setLabel("?");
if (astNode.equals("WildcardType")) {
simpleTree.setLabel("?");
} else { // ArrayType, PrimitiveType, SimpleType, ParameterizedType, QualifiedType, WildcardType, UnionType,NameQualifiedType, IntersectionType
simpleTree.setLabel(astNode + "@@" + label);
}
} else if (astNode.endsWith("Name")) { // variableName, methodName, QualifiedName
if (label.startsWith("MethodName:")) { // <Method, name>
label = label.substring(11);
simpleTree.setNodeType("Method");
simpleTree.setLabel(label);
} else if (label.startsWith("Name:")) {
label = label.substring(5);
String firstChar = label.substring(0, 1);
if (firstChar.equals(firstChar.toUpperCase())) {
simpleTree.setNodeType("Name");
simpleTree.setLabel(label); // <Name, name>
} else {// variableName: <Variable, var>
simpleTree.setNodeType("Variable");
simpleTree.setLabel(getAbstractLabel(abstractVariableIdentifiers, label, ABSTRACT_VARIABLE));
}
} else {// variableName: <Variable, var>
simpleTree.setNodeType("Variable");
simpleTree.setLabel(getAbstractLabel(abstractVariableIdentifiers, label, ABSTRACT_VARIABLE));
}
} else if (astNode.equals("BooleanLiteral") ||astNode.equals("CharacterLiteral") || astNode.equals("ThisExpression")
|| astNode.equals("NullLiteral") || astNode.equals("NumberLiteral") || astNode.equals("StringLiteral")
|| astNode.equals("Modifier") || astNode.equals("Operator")) {
simpleTree.setNodeType(astNode);
simpleTree.setLabel(label);
} else {
simpleTree.setNodeType(astNode);
simpleTree.setLabel(astNode);
}
}
List<SimpleTree> simpleChildren = new ArrayList<>();
if (children != null && !astNode.endsWith("Type")) {
for (ITree child : children) {
simpleChildren.add(semiSourceCodeTree(actionSet, child, simpleTree));
}
}
simpleTree.setChildren(simpleChildren);
}
private List<Action> getAllMoveActions(HierarchicalActionSet actionSet) {
String astNodeType = actionSet.getAstNodeType();
if (Checker.containsBodyBlock(astNodeType)) {
List<Action> allMoveActions = getAllMoveActions2(actionSet);
if (allMoveActions != null && allMoveActions.size() > 0) {
ListSorter<Action> sorter = new ListSorter<Action>(allMoveActions);
List<Action> moveActions = sorter.sortAscending();
if (moveActions != null) {
allMoveActions = moveActions;
}
return allMoveActions;
} else {// FIXME: pure INS actions.
return null;
}
} else {// FIXME: pure INS actions.
return null;
}
/**
* Variables, non-new and used in the inserted statements, could be selected to localize buggy code
*/
}
private List<Action> getAllMoveActions2(HierarchicalActionSet actionSet) {
List<Action> allMoveActions = new ArrayList<>();
List<HierarchicalActionSet> actions = new ArrayList<>();
actions.addAll(actionSet.getSubActions());
if (actions.size() == 0) {
return null;
}
while (actions.size() > 0) {
List<HierarchicalActionSet> subActions = new ArrayList<>();
for (HierarchicalActionSet action : actions) {
subActions.addAll(action.getSubActions());
if (action.toString().startsWith("MOV")) {
allMoveActions.add(action.getAction());
}
}
actions.clear();
actions.addAll(subActions);
}
return allMoveActions;
}
private String getAbstractLabel(Map<String, String> map, String label, String nameType) {
if (map.containsKey(label)) {
return map.get(label);
} else {
String name = nameType + map.size();
map.put(label, name);
return name;
}
}
private HierarchicalActionSet findHierarchicalActionSet(int position, int length, HierarchicalActionSet actionSet) {
if (actionSet.getStartPosition() == position && actionSet.getLength() == length && !actionSet.getActionString().startsWith("INS")) {
return actionSet;
} else {
for (HierarchicalActionSet subActionSet : actionSet.getSubActions()) {
HierarchicalActionSet actSet = findHierarchicalActionSet(position, length, subActionSet);
if (actSet != null) {
return actSet;
}
}
}
return null;
}
public Map<String, String> getAbstractTypeIdentifiers() {
return abstractTypeIdentifiers;
}
public Map<String, String> getAbstractMethodIdentifiers() {
return abstractMethodIdentifiers;
}
public Map<String, String> getAbstractNameIdentifiers() {
return abstractNameIdentifiers;
}
public Map<String, String> getAbstractVariableIdentifiers() {
return abstractVariableIdentifiers;
}
}
src/main/java/edu/lu/uni/serval/gumtree/regroup/Traveler.java
-140
View File
@@ -1,140 +0,0 @@
package edu.lu.uni.serval.gumtree.regroup;
import java.util.ArrayList;
import java.util.List;
import com.github.gumtreediff.tree.ITree;
import edu.lu.uni.serval.FixPattern.utils.ASTNodeMap;
/**
* A traveler to travel a tree-constructed object.
*
* @author kui.liu
*
*/
public class Traveler {
public List<List<String>> list = new ArrayList<>();
/**
* Get all action string by traveling HierarchicalActionSet in a deep-first way.
*
* @param actionSet
* @param astNodeTypeActionQueue
*/
public void travelActionSetDeepFirstToASTNodeQueue(HierarchicalActionSet actionSet, List<String> astNodeTypeActionQueue) {
if (actionSet == null) {
System.err.println("Null Action set!");
} else {
if (astNodeTypeActionQueue == null) {
astNodeTypeActionQueue = new ArrayList<>();
}
String actionStr = actionSet.getActionString();
actionStr = actionStr.substring(0, actionStr.indexOf("@@"));
astNodeTypeActionQueue.add(actionStr); // RawToken: TODO
if (actionStr.startsWith("DEL")) {
list.add(astNodeTypeActionQueue); // FIXME BUG: Change AST node type 1 to AST node type 2. Solve method: a list is one pattern.
} else {
List<HierarchicalActionSet> subActionSet = actionSet.getSubActions();
int size = subActionSet.size();
if (size > 0) {
for (HierarchicalActionSet subAction : subActionSet) {
List<String> astNodeTypeActionQueue_ = new ArrayList<>();
astNodeTypeActionQueue_.addAll(astNodeTypeActionQueue);
travelActionSetDeepFirstToASTNodeQueue(subAction, astNodeTypeActionQueue_);
}
} else {
list.add(astNodeTypeActionQueue);
}
}
}
}
/**
* Get all AST node types of a root tree by traveling the root tree in a deep-first way.
*
* @param root
* @return
*/
public static List<String> travelTreeDeepFirstToASTNodeQueue(ITree root) {
if (root == null) {
System.err.println("Null tree!");
return null;
}
List<String> astNodeTypeQueue = new ArrayList<>();
astNodeTypeQueue.add(ASTNodeMap.map.get(root.getType())); // RawToken: root.getLabel();
List<ITree> childrenTreeList = root.getChildren();
if (childrenTreeList != null && childrenTreeList.size() > 0) {
for (ITree childTree : childrenTreeList) {
astNodeTypeQueue.addAll(travelTreeDeepFirstToASTNodeQueue(childTree));
}
}
return astNodeTypeQueue;
}
/**
* Get all AST node types of a root tree by traveling the root tree in a breadth-first way.
*
* @param root
* @return
*/
public static List<String> travelTreeBreadthFirstToASTNodeQueue(ITree root) {
if (root == null) {
System.err.println("Null tree.");
return null;
}
List<String> astNodeTypeQueue = new ArrayList<>();
astNodeTypeQueue.add(ASTNodeMap.map.get(root.getType())); // RawToken: root.getLabel();
List<ITree> treeList = new ArrayList<>();
treeList.add(root);
while (!treeList.isEmpty()) {
List<ITree> childrenTreeList = new ArrayList<>();
for (ITree tree : treeList) {
astNodeTypeQueue.addAll(travelTreeBreadthFirstToASTNodeQueue(tree));
childrenTreeList.addAll(tree.getChildren());
}
treeList.clear();
treeList.addAll(childrenTreeList);
}
return astNodeTypeQueue;
}
/**
* Convert a root ITree into a SimpleTree by traveling the root tree in a deep-first way.
*
* SimpleTree node label is root.toShortString().
*
* @param root
* @param parent
* @return
*/
public static SimpleTree travelITreeDeepFirstToSimpleTree(ITree root, SimpleTree parent) {
if (root == null) {
System.err.println("Null tree!");
return null;
}
SimpleTree simpleTree = new SimpleTree();
simpleTree.setLabel(root.toShortString());
simpleTree.setParent(parent);
List<SimpleTree> children = new ArrayList<>();
List<ITree> childrenTreeList = root.getChildren();
if (childrenTreeList != null && childrenTreeList.size() > 0) {
for (ITree childTree : childrenTreeList) {
children.add(travelITreeDeepFirstToSimpleTree(childTree, simpleTree));
}
}
simpleTree.setChildren(children);
return simpleTree;
}
}
src/main/java/edu/lu/uni/serval/livestudy/Alarm.java
-33
View File
@@ -1,33 +0,0 @@
package edu.lu.uni.serval.livestudy;
public class Alarm {
private String alarmType;
private String fileName;
private int startLine;
private int endLine;
public String getAlarmType() {
return alarmType;
}
public String getFileName() {
return fileName;
}
public int getStartLine() {
return startLine;
}
public int getEndLine() {
return endLine;
}
public Alarm(String alarmType, String fileName, int startLine, int endLine) {
super();
this.alarmType = alarmType;
this.fileName = fileName;
this.startLine = startLine;
this.endLine = endLine;
}
}
src/main/java/edu/lu/uni/serval/livestudy/ParseViolations.java
-212
View File
@@ -1,212 +0,0 @@
package edu.lu.uni.serval.livestudy;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Scanner;
import edu.lu.uni.serval.FixPatternParser.Tokenizer;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.gumtree.regroup.SimpleTree;
import edu.lu.uni.serval.utils.FileHelper;
public class ParseViolations {
private static final String OUTPUT_PATH = Configuration.ROOT_PATH + "LiveStudy/BugsInfo/";
private static int i = 0;
private static int j = 0;
public static void main(String[] args) {
// output path
// FileHelper.deleteDirectory(OUTPUT_PATH);
List<String> projects = new ArrayList<>();
File[] files = new File(OUTPUT_PATH).listFiles();
for (File file : files) {
if (file.isDirectory()) {
projects.add(file.getName());
}
}
String projectsPath = Configuration.ROOT_PATH + "LiveStudy/projects/";
String bugsListPath = Configuration.ROOT_PATH + "LiveStudy/BugsList/";
List<File> bugsListFiles = FileHelper.getAllFiles(bugsListPath, ".list");
for (File bugsListFile : bugsListFiles) {
String fileName = FileHelper.getFileNameWithoutExtension(bugsListFile);
if (projects.contains(fileName)) continue;
ParseViolations parser = new ParseViolations();
Map<String, Violation> violations = new HashMap<>(); // <ProjectName, Violations>, Violation: <projectName, List<Alarm>>.
violations = parser.readViolations(bugsListFile);
parser.parseViolationToTokens(violations, projectsPath);
}
System.out.println(i);
System.out.println(j);
}
/**
* Get the source code tokens of all violation instances by visiting each java project.
*
* @param violations
* @param projectsPath
*/
public void parseViolationToTokens(Map<String, Violation> violations, String projectsPath) {
for (Map.Entry<String, Violation> entry : violations.entrySet()) {
String projectName = entry.getKey();
List<Alarm> alarms = entry.getValue().getAlarms();
List<File> javaFiles = FileHelper.getAllFiles(projectsPath + projectName, ".java");
//Each violation: commons-math : DLS_DEAD_LOCAL_STORE : org/apache/commons/math4/dfp/Dfp.java : 2049 : 2049
/**
* @ProjectName
* @ViolationType
* @FileName
* @LineNumber
* @SourceCode
* @Tokens
* @NumberOfTokens
*/
// sizes file, tokens file, and bugs-info file
StringBuilder sizesBuilder = new StringBuilder();
StringBuilder tokensBuilder = new StringBuilder();
StringBuilder bugsInfoBuilder = new StringBuilder();
for (Alarm alarm : alarms) {
String fileName = alarm.getFileName();
// if (fileName.toLowerCase(Locale.ENGLISH).contains("test")) continue;
File sourceCodeFile = locateSourceCodeFile(javaFiles, fileName);
if (sourceCodeFile == null) {
j ++;
continue;
}
Parser parser = new Parser(alarm, sourceCodeFile);
parser.parse();
SimpleTree simpleTree = parser.getSimpleTree();
if (simpleTree != null) {
int finalStartLine = parser.getFinalStartLine();
int finalEndLine = parser.getFinalEndLine();
String violationType = alarm.getAlarmType();
String tokens = Tokenizer.getTokensDeepFirst(simpleTree);
String[] tokensArray = tokens.split(" ");
int length = tokensArray.length;
StringBuilder sourceCode = readSourceCode(sourceCodeFile, finalStartLine, finalEndLine);
sizesBuilder.append(length).append("\n");
tokensBuilder.append(tokens).append("\n");
bugsInfoBuilder.append("###BugInstance###\n##Info:");
bugsInfoBuilder.append(violationType).append(":");
bugsInfoBuilder.append(projectName).append(":");
bugsInfoBuilder.append(fileName).append(":");
bugsInfoBuilder.append(finalStartLine).append(":");
bugsInfoBuilder.append(finalEndLine).append("\n");
bugsInfoBuilder.append(sourceCode).append("\n");
i ++;
} else {
j ++;
}
}
FileHelper.outputToFile(OUTPUT_PATH + "sizes.list", sizesBuilder, true);
FileHelper.outputToFile(OUTPUT_PATH + "tokens.list", tokensBuilder, true);
FileHelper.outputToFile(OUTPUT_PATH + "bugsInfo.list", bugsInfoBuilder, true);
}
}
private StringBuilder readSourceCode(File javaFile, int startLine, int endLine) {
StringBuilder sourceCode = new StringBuilder("##Source_Code:\n");
FileInputStream fis = null;
Scanner scanner = null;
try {
fis = new FileInputStream(javaFile);
scanner = new Scanner(fis);
int counter = 0;
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
counter ++;
if (startLine <= counter && counter <= endLine) {
sourceCode.append(line + "\n");
}
if (counter == endLine) break;
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
try {
scanner.close();
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sourceCode;
}
private File locateSourceCodeFile(List<File> javaFiles, String fileName) {
for (File javaFile : javaFiles) {
if (javaFile.getPath().endsWith(fileName)) return javaFile;
}
return null;
}
public Map<String, Violation> readViolations(String violationFile) {
return readViolations(new File(violationFile));
}
public Map<String, Violation> readViolations(File violationFile) {
Map<String, Violation> violations = new HashMap<>();
FileInputStream fis = null;
Scanner scanner = null;
try {
fis = new FileInputStream(violationFile);
scanner = new Scanner(fis);
while (scanner.hasNextLine()) {
//commons-math : DLS_DEAD_LOCAL_STORE : org/apache/commons/math4/dfp/Dfp.java : 2049 : 2049
String line = scanner.nextLine();
String[] elements = line.split(" : ");
String projectName = elements[0];
String alarmType = elements[1];
String fileName = elements[2];
int startLine = Integer.parseInt(elements[3]);
int endLine = Integer.parseInt(elements[4]);
Alarm alarm = new Alarm(alarmType, fileName, startLine, endLine);
addAlarmToViolations(projectName, alarm, violations);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
try {
scanner.close();
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return violations;
}
private void addAlarmToViolations(String projectName, Alarm alarm, Map<String, Violation> violations) {
Violation violation = null;
if (violations.containsKey(projectName)) {
violation = violations.get(projectName);
} else {
violation = new Violation(projectName);
violations.put(projectName, violation);
}
List<Alarm> alarms = violation.getAlarms();
alarms.add(alarm);
}
}
src/main/java/edu/lu/uni/serval/livestudy/Parser.java
-84
View File
@@ -1,84 +0,0 @@
package edu.lu.uni.serval.livestudy;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import com.github.gumtreediff.tree.ITree;
import edu.lu.uni.serval.gumtree.regroup.SimpleTree;
import edu.lu.uni.serval.gumtree.regroup.SimplifyTree;
import edu.lu.uni.serval.violation.code.parser.ViolationSourceCodeTree;
public class Parser {
// input
private Alarm alarm;
private File sourceCodeFile;
// output
private int finalStartLine = 0;
private int finalEndLine = 0;
private SimpleTree simpleTree = null;
public Parser(Alarm alarm, File sourceCodeFile) {
this.alarm = alarm;
this.sourceCodeFile = sourceCodeFile;
}
public int getFinalStartLine() {
return finalStartLine;
}
public int getFinalEndLine() {
return finalEndLine;
}
public SimpleTree getSimpleTree() {
return simpleTree;
}
public void parse() {
int startLine = this.alarm.getStartLine();
int endLine = this.alarm.getEndLine();
String violationType = this.alarm.getAlarmType();
ViolationSourceCodeTree treeParser = new ViolationSourceCodeTree(this.sourceCodeFile, startLine, endLine);
if ("EQ_DOESNT_OVERRIDE_EQUALS".equals(violationType)|| "HE_EQUALS_USE_HASHCODE".equals(violationType)
|| "HE_INHERITS_EQUALS_USE_HASHCODE".equals(violationType)|| "SE_NO_SUITABLE_CONSTRUCTOR".equals(violationType)
|| "RI_REDUNDANT_INTERFACES".equals(violationType) || "CN_IDIOM".equals(violationType)
|| "SE_NO_SERIALVERSIONID".equals(violationType) || "SE_COMPARATOR_SHOULD_BE_SERIALIZABLE".equals(violationType)) {
// Class name level, tokens
// classStartP <= vS <= vE <= classEndP
ITree classNameTree = treeParser.getClassNameTokens();
if (classNameTree != null) {
this.simpleTree = new SimplifyTree().canonicalizeSourceCodeTree(classNameTree, null);
} else {
System.err.println("#Null_Violation_Hunk: " + this.alarm.getAlarmType() + ":" + this.sourceCodeFile.getPath() + ":" + startLine + ":" + endLine);
}
} else {
treeParser.extract();
List<ITree> matchedTrees = treeParser.getViolationSourceCodeTrees();
if (matchedTrees.size() == 0) {
System.err.println("#Null_Violation_Hunk: " + this.alarm.getAlarmType() + ":" + this.sourceCodeFile.getPath() + ":" + startLine + ":" + endLine);
} else {
this.simpleTree = new SimpleTree();
this.simpleTree.setLabel("Block");
this.simpleTree.setNodeType("Block");
List<SimpleTree> children = new ArrayList<>();
for (ITree matchedTree : matchedTrees) {
SimpleTree simpleT = new SimplifyTree().canonicalizeSourceCodeTree(matchedTree, this.simpleTree);
children.add(simpleT);
}
this.simpleTree.setChildren(children);
}
}
if (this.simpleTree != null) {
this.finalStartLine = treeParser.getViolationFinalStartLine();
this.finalEndLine = treeParser.getViolationFinalEndLine();
}
}
}
src/main/java/edu/lu/uni/serval/livestudy/README.md
-6
View File
@@ -1,6 +0,0 @@
## Parse the violations of Live Study ##
1. Localize bugs with FindBugs.
2. Parse results of FindBugs. (bug-type project.)
3. Parse source code of violations. (ParseViolation.java)
src/main/java/edu/lu/uni/serval/livestudy/Violation.java
-25
View File
@@ -1,25 +0,0 @@
package edu.lu.uni.serval.livestudy;
import java.util.ArrayList;
import java.util.List;
public class Violation {
private String project;
private List<Alarm> alarms;
public String getProject() {
return project;
}
public List<Alarm> getAlarms() {
return alarms;
}
public Violation(String project) {
super();
this.project = project;
this.alarms = new ArrayList<>();
}
}
src/main/java/edu/lu/uni/serval/parameters/ParseAlarms.java
-145
View File
@@ -1,145 +0,0 @@
package edu.lu.uni.serval.parameters;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.List;
import com.github.gumtreediff.tree.ITree;
import edu.lu.uni.serval.FixPatternParser.Tokenizer;
import edu.lu.uni.serval.FixPatternParser.violations.Violation;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.gumtree.regroup.SimpleTree;
import edu.lu.uni.serval.gumtree.regroup.SimplifyTree;
import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.violation.code.parser.ViolationSourceCodeTree;
/**
* Prepare data for tuning parameters of deep learning models.
*
* @author kui.liu
*
*/
public class ParseAlarms {
public static void main(String[] args) {
ParseAlarms parser = new ParseAlarms();
String outputPath = Configuration.ROOT_PATH + "TuneParameters/fixedAlarmTokens.list";
String outputPath2 = Configuration.ROOT_PATH + "TuneParameters/EmptyStatement/fixedAlarmTokens.list";
FileHelper.deleteFile(outputPath);
FileHelper.deleteFile(outputPath2);
int subIndex = 5;
String fixedAlarmFilesPath = Configuration.GUM_TREE_INPUT + "prevFiles/";
String positionsFilePath = Configuration.GUM_TREE_INPUT + "positions/";
parser.dataPreparation(fixedAlarmFilesPath, positionsFilePath, subIndex, outputPath, outputPath2);
outputPath = Configuration.ROOT_PATH + "TuneParameters/unfixedAlarmTokens.list";
outputPath2 = Configuration.ROOT_PATH + "TuneParameters/EmptyStatement/unfixedAlarmTokens.list";
FileHelper.deleteFile(outputPath);
FileHelper.deleteFile(outputPath2);
subIndex = 8;
String unfixedAlarmFilesPath = Configuration.GUM_TREE_INPUT + "unfixAlarms/";
String unfixedPositionsFilePath = Configuration.GUM_TREE_INPUT + "un_positions/";
parser.dataPreparation(unfixedAlarmFilesPath, unfixedPositionsFilePath, subIndex, outputPath, outputPath2);
}
public void dataPreparation(String sourceCodeFilePath, String positionFilePath, int subIndex, String outputPath, String outputPath2) {
StringBuilder tokensBuilder = new StringBuilder();
List<File> javaFiles = FileHelper.getAllFilesInCurrentDiectory(sourceCodeFilePath, ".java");
int counter = 0;
int a = 0;
StringBuilder emptyStatements = new StringBuilder();
StringBuilder sizes = new StringBuilder();
for (File javaFile : javaFiles) {
String fileName = javaFile.getName().replace(".java", ".txt");
fileName = fileName.substring(subIndex);
// if (fileName.endsWith("apache-commons-configuration_8c42aa_8b26e6src#java#org#apache#commons#configuration#plist#XMLPropertyListConfiguration.txt")) {
// System.out.println();
// }
List<Violation> violations = readViolations(positionFilePath + fileName);
for (Violation violation : violations) {
int startLine = violation.getStartLineNum();
int endLine = violation.getEndLineNum();
String alarmType = violation.getViolationType();
if (endLine > startLine + 5) continue;
ViolationSourceCodeTree alarmTree = new ViolationSourceCodeTree(javaFile, startLine, endLine);
alarmTree.extract(alarmType);
List<ITree> matchedTrees = alarmTree.getViolationSourceCodeTrees();
if (matchedTrees.size() == 0) {
emptyStatements.append(alarmType + "," + fileName + "," + startLine + "," + endLine + "\n");
a ++;
continue;
}
SimpleTree simpleTree = new SimpleTree();
simpleTree.setLabel("Block");
simpleTree.setNodeType("Block");
List<SimpleTree> children = new ArrayList<>();
for (ITree matchedTree : matchedTrees) {
SimpleTree simpleT = new SimplifyTree().canonicalizeSourceCodeTree(matchedTree, null);
children.add(simpleT);
}
simpleTree.setChildren(children);
String tokens = Tokenizer.getTokensDeepFirst(simpleTree);
String[] tokensArray = tokens.split(" ");
int length = tokensArray.length;
sizes.append(length + "\n");
tokensBuilder.append(alarmType + ":" + fileName + ":" + alarmTree.getViolationFinalStartLine() + ":" + alarmTree.getViolationFinalEndLine() + ":" + tokens + "\n");
counter ++;
if (counter % 10000 == 0) {
FileHelper.outputToFile(outputPath, tokensBuilder, true);
tokensBuilder.setLength(0);
}
}
}
System.out.println("Volidated Instances: " + counter);
System.out.println("Empty Instances: " + a);
FileHelper.outputToFile(outputPath2, emptyStatements, false);
FileHelper.outputToFile(outputPath.replace(".list", "Sizes.csv"), sizes, false);
FileHelper.outputToFile(outputPath, tokensBuilder, true);
}
private List<Violation> readViolations(String file) {
List<Violation> violations = new ArrayList<>();
String fileContent = FileHelper.readFile(file);
BufferedReader reader = null;
reader = new BufferedReader(new StringReader(fileContent));
String line = null;
try {
while ((line = reader.readLine()) != null) {
String[] positionStr = line.split(":");
int startLine = Integer.parseInt(positionStr[1]);
int endLine = Integer.parseInt(positionStr[2]);
if (startLine == -1 || endLine == -1) {
continue;
}
String alarmType = positionStr[0];
Violation violation = new Violation(startLine, endLine, alarmType);
violations.add(violation);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return violations;
}
}
src/main/java/edu/lu/uni/serval/statistics/.Rapp.history
View File
src/main/java/edu/lu/uni/serval/statistics/Categories.java
-75
View File
@@ -1,75 +0,0 @@
package edu.lu.uni.serval.statistics;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
public class Categories {
public static void main(String[] args) throws IOException {
String content = FileHelper.readFile(Configuration.ROOT_PATH + "RQ1/Quantity-per-Fixed_1.0V-Type.csv");//"RQ1/Quantity-per-V-Type.csv");
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = reader.readLine();
int counter = 0;
// int size = 400;
Map<String, Integer> map = new HashMap<>();
map.put("Dodgy code", 0);
map.put("Experimental", 0);
map.put("Internationalization", 0);
map.put("Multithreaded correctness", 0);
map.put("Malicious code vulnerability", 0);
map.put("Performance", 0);
map.put("Correctness", 0);
map.put("Security", 0);
map.put("Bad practice", 0);
map.put("Other", 0);
// List<Integer> list = new ArrayList<>();
Map<String, List<Integer>> map2 = new HashMap<>();
map2.put("Dodgy code", new ArrayList<Integer>());
map2.put("Experimental", new ArrayList<Integer>());
map2.put("Internationalization", new ArrayList<Integer>());
map2.put("Multithreaded correctness", new ArrayList<Integer>());
map2.put("Malicious code vulnerability", new ArrayList<Integer>());
map2.put("Performance", new ArrayList<Integer>());
map2.put("Correctness", new ArrayList<Integer>());
map2.put("Security", new ArrayList<Integer>());
map2.put("Bad practice", new ArrayList<Integer>());
map2.put("Other", new ArrayList<Integer>());
while ((line = reader.readLine()) != null) {
String type = line.substring(line.lastIndexOf(",") + 1);
counter ++;
if (map.containsKey(type)) {
map.put(type, map.get(type) + 1);
} else {
map.put(type, 1);
}
if (counter % 10 == 0) {
for (Map.Entry<String, List<Integer>> entry : map2.entrySet()) {
String type2 = entry.getKey();
Integer integ = map.get(type2);
if (integ == null) {
map2.get(type2).add(0);
} else {
map2.get(type2).add(integ);
}
}
// System.out.println();
}
}
reader.close();
for (Map.Entry<String, Integer> entry : map.entrySet()) {
System.out.println(entry.getKey() + "," + entry.getValue());
}
for (Map.Entry<String, List<Integer>> entry : map2.entrySet()) {
System.out.println(entry.getKey() + "," + entry.getValue());
}
}
}
src/main/java/edu/lu/uni/serval/statistics/FixedProjects.java
-26
View File
@@ -1,26 +0,0 @@
package edu.lu.uni.serval.statistics;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.StringReader;
import edu.lu.uni.serval.utils.FileHelper;
public class FixedProjects {
public static void main(String[] args) throws IOException {
String content = FileHelper.readFile("../FPM_Violations/RQ1/Quantity-per-FixedProj.csv");
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = reader.readLine();
StringBuilder builder = new StringBuilder();
while ((line = reader.readLine()) != null) {
String[] elements = line.split(",");
// String projectName = elements[0];
builder.append(elements[0] + "\n");
}
reader.close();
FileHelper.outputToFile("../FPM_Violations/RQ1/fixedProjects.list", builder, false);
}
}
src/main/java/edu/lu/uni/serval/statistics/FixedVilations.java
-47
View File
@@ -1,47 +0,0 @@
package edu.lu.uni.serval.statistics;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.Scanner;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
public class FixedVilations {
public static void main(String[] args) throws IOException {
// projectName,ViolationType,Number.
FileInputStream fis = new FileInputStream(Configuration.ROOT_PATH + "RQ1/fixed-alarms-v1.0.list");
Scanner scanner = new Scanner(fis);
Map<String, Integer> map = new HashMap<>();
int counter = 0;
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
String[] elements = line.split(":");
// String violationType = elements[0];
// String projectName = elements[1];
String key = elements[1] + "," + elements[0];
if (map.containsKey(key)) {
map.put(key, map.get(key)+ 1);
} else {
map.put(key, 1);
}
counter ++;
}
scanner.close();
fis.close();
System.out.println(counter);
StringBuilder builder = new StringBuilder();
for (Map.Entry<String, Integer> entry : map.entrySet()) {
builder.append(entry.getKey()).append(",").append(entry.getValue()).append("\n");
counter -= entry.getValue();
}
FileHelper.outputToFile(Configuration.ROOT_PATH + "RQ1/fixedViolations-v-1.0.csv", builder, false);
System.out.println(counter);
}
}
src/main/java/edu/lu/uni/serval/statistics/HunkSizeComputer.java
-106
View File
@@ -1,106 +0,0 @@
package edu.lu.uni.serval.statistics;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.utils.ListSorter;
public class HunkSizeComputer {
public static void main(String[] args) throws IOException {
String violationsFilePath = Configuration.ROOT_PATH + "unFixedInstances/";
sizes(violationsFilePath, ".list");
}
public static void sizes(String filePath, String fileType) throws NumberFormatException, IOException {
List<File> unfixedAlarmFiles = FileHelper.getAllFilesInCurrentDiectory(filePath, fileType);
System.out.println(unfixedAlarmFiles.size());
StringBuilder builder = new StringBuilder();
int counter = 0;
int size = 0;
int i = 0;
List<Integer> sizes = new ArrayList<>();
Map<Integer, Integer> map = new HashMap<>();// <size, amount>
for (File file : unfixedAlarmFiles) {
String content = FileHelper.readFile(file);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
while ((line = reader.readLine()) != null) {
String[] elements = line.split(",");
int startLine = Integer.parseInt(elements[4]);
int endLine = Integer.parseInt(elements[5]);
i ++;
if (startLine == -1 || endLine == -1) continue;
int range = endLine - startLine + 1;
builder.append(range + "\n");
if (range != 1) {
counter ++;
}
size ++;
sizes.add(range);
if (map.containsKey(range)) {
map.put(range, map.get(range) + 1);
} else {
map.put(range, 1);
}
}
reader.close();
}
System.out.println(i);
String content = FileHelper.readFile("Dataset/fixed-alarms-v1.0.list");
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
while ((line = reader.readLine()) != null) {
int arrowIndex = line.indexOf("=>");
String buggyInfo = line.substring(0, arrowIndex);
String[] buggyElements = buggyInfo.split(":");
int startLine = Integer.parseInt(buggyElements[4]);
int endLine = Integer.parseInt(buggyElements[5]);
if (startLine == -1 || endLine == -1) continue;
int range = endLine - startLine + 1;
if (range != 1) {
counter ++;
}
builder.append(range + "\n");
size ++;
sizes.add(range);
if (map.containsKey(range)) {
map.put(range, map.get(range) + 1);
} else {
map.put(range, 1);
}
}
reader.close();
FileHelper.outputToFile("Dataset/sizes.csv", builder, false);
System.out.println(size);
System.out.println(counter);
ListSorter<Integer> sorter = new ListSorter<>(sizes);
sizes = sorter.sortAscending();
System.out.println(sizes.get((int) (sizes.size() * 0.7)));
System.out.println(sizes.get((int) (sizes.size() * 0.8)));
System.out.println(sizes.get((int) (sizes.size() * 0.9)));
System.out.println(sizes.get((int) (sizes.size() * 0.95)));
int sum = 0;
for (i = 1; i <= 100; i ++) {
sum += map.get(i);
System.out.println(i + "," + String.format("%.2f", ((double) sum / size * 100)));
}
}
}
src/main/java/edu/lu/uni/serval/statistics/NumberOfFilesAndCodeLines.java
-74
View File
@@ -1,74 +0,0 @@
package edu.lu.uni.serval.statistics;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.List;
import java.util.Locale;
import java.util.Scanner;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
/**
* Statistics of files and code lines of projects in Live Study.
*
* @author kui.liu
*
*/
public class NumberOfFilesAndCodeLines {
public static void main(String[] args) {
String projectsPath = Configuration.ROOT_PATH + "LiveStudy/BugsInfo/";
File[] projects = new File(projectsPath).listFiles();
for (File project : projects) {
if (project.isDirectory()) {
if (project.getName().equals("poi"))
new NumberOfFilesAndCodeLines().statistic(project.getName());
}
}
}
public void statistic(String project) {
List<File> javaFiles = FileHelper.getAllFiles(Configuration.ROOT_PATH + "LiveStudy/projects/" + project, ".java");
int numberOfFiles = 0;
int LOC = 0;
for (File javaFile : javaFiles) {
if (!javaFile.getPath().toLowerCase(Locale.ENGLISH).contains("test")) {
numberOfFiles ++;
LOC += readLinesOfCode(javaFile);
}
// numberOfFiles ++;
// LOC += readLinesOfCode(javaFile);
}
System.out.println(project + " : Files = " + numberOfFiles + ", LOC = " + LOC);
}
private int readLinesOfCode(File javaFile) {
int LOC = 0;
Scanner scanner = null;
FileInputStream fis = null;
try {
fis = new FileInputStream(javaFile);
scanner = new Scanner(fis);
while (scanner.hasNextLine()) {
scanner.nextLine();
LOC ++;
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
try {
scanner.close();
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return LOC;
}
}
src/main/java/edu/lu/uni/serval/statistics/ReadSpearmanResults.java
-104
View File
@@ -1,104 +0,0 @@
package edu.lu.uni.serval.statistics;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.StringReader;
import edu.lu.uni.serval.utils.FileHelper;
public class ReadSpearmanResults {
public static void main(String[] args) throws IOException {
String fileName = "../FPM_Violations/RQ1/SpearmanResults.list";
String content = FileHelper.readFile(fileName);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
String str1 = "1";
String str2 = "2";
String str3 = "3";
String str4 = "4";
String str5 = "5";
String str6 = "6";
String str7 = "7";
String str8 = "8";
String str9 = "9";
String str10 = "10";
int counter = 0;
while ((line = reader.readLine()) != null) {
// String[] elements = line.split("");
counter ++;
if ((counter / 10 == 2 || counter / 10 == 4) && counter % 10 == 1) {
str1 += "@@@@1";
str2 += "@@@@2";
str3 += "@@@@3";
str4 += "@@@@4";
str5 += "@@@@5";
str6 += "@@@@6";
str7 += "@@@@7";
str8 += "@@@@8";
str9 += "@@@@9";
str10 += "@@@@10";
}
int n = counter % 10;
switch (n) {
case 1:
str1 += "@" + line;
break;
case 2:
str2 += "@" + line;
break;
case 3:
str3 += "@" + line;
break;
case 4:
str4 += "@" + line;
break;
case 5:
str5 += "@" + line;
break;
case 6:
str6 += "@" + line;
break;
case 7:
str7 += "@" + line;
break;
case 8:
str8 += "@" + line;
break;
case 9:
str9 += "@" + line;
break;
case 0:
str10 += "@" + line;
break;
}
}
reader.close();
str1 = str1.replace(".", ",");
str2 = str2.replace(".", ",");
str3 = str3.replace(".", ",");
str4 = str4.replace(".", ",");
str5 = str5.replace(".", ",");
str6 = str6.replace(".", ",");
str7 = str7.replace(".", ",");
str8 = str8.replace(".", ",");
str9 = str9.replace(".", ",");
str10 = str10.replace(".", ",");
System.out.println(str1);
System.out.println(str2);
System.out.println(str3);
System.out.println(str4);
System.out.println(str5);
System.out.println(str6);
System.out.println(str7);
System.out.println(str8);
System.out.println(str9);
System.out.println(str10);
}
}
src/main/java/edu/lu/uni/serval/statistics/Spearman.R
-362
View File
@@ -1,362 +0,0 @@
a <- read.csv("~/Public/git/FPM_Violations/RQ1/TenFolds-Q/Ten-fold-all-fixed-violation-type1-70.csv", header=T)
d<-cor.test( ~ X0 + X0.T,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = FALSE, sep = "@")
d<-cor.test( ~ X1 + X1.T,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X2 + X2.T,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X3 + X3.T,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X4 + X4.T,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X5 + X5.T,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X6 + X6.T,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X7 + X7.T,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X8 + X8.T,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X9 + X9.T,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X0.R + X0.T.R,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X1.R + X1.T.R,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X2.R + X2.T.R,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X3.R + X3.T.R,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X4.R + X4.T.R,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X5.R + X5.T.R,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X6.R + X6.T.R,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X7.R + X7.T.R,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X8.R + X8.T.R,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X9.R + X9.T.R,
data=a, method = "spearman",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X0 + X0.T,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X1 + X1.T,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X2 + X2.T,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X3 + X3.T,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X4 + X4.T,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X5 + X5.T,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X6 + X6.T,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X7 + X7.T,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X8 + X8.T,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X9 + X9.T,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X0.R + X0.T.R,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X1.R + X1.T.R,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X2.R + X2.T.R,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X3.R + X3.T.R,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X4.R + X4.T.R,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X5.R + X5.T.R,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X6.R + X6.T.R,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X7.R + X7.T.R,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X8.R + X8.T.R,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X9.R + X9.T.R,
data=a, method = "pearson",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X0 + X0.T,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X1 + X1.T,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X2 + X2.T,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X3 + X3.T,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X4 + X4.T,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X5 + X5.T,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X6 + X6.T,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X7 + X7.T,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X8 + X8.T,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X9 + X9.T,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X0.R + X0.T.R,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X1.R + X1.T.R,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X2.R + X2.T.R,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X3.R + X3.T.R,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X4.R + X4.T.R,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X5.R + X5.T.R,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X6.R + X6.T.R,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X7.R + X7.T.R,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X8.R + X8.T.R,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
d<-cor.test( ~ X9.R + X9.T.R,
data=a, method = "kendall",
continuity = FALSE,
conf.level = 0.95)
write(c(d$estimate, d$p.value), file = "~/Public/git/FPM_Violations/RQ1/SpearmanResults.list",append = TRUE, sep = "@")
a=1
src/main/java/edu/lu/uni/serval/statistics/SpearmanCodeGenerator.java
-81
View File
@@ -1,81 +0,0 @@
package edu.lu.uni.serval.statistics;
public class SpearmanCodeGenerator {
public static void main(String[] args) {
generateSpearmanCode();
}
public static void generateSpearmanCode() {
for (int i = 0; i < 10; i ++) {
String s = "d<-cor.test( ~ X" + i + " + X" + i + ".T, \n";
s += " data=a,";
s += " method = \"spearman\",\n";
s += " continuity = FALSE,\n";
s += " conf.level = 0.95)\n";
if (i == 0) {
s += "write(c(d$estimate, d$p.value), file = \"~/Public/git/FPM_Violations/RQ1/SpearmanResults.list\"," +
"append = FALSE, sep = \"@\")\n";
} else {
s += "write(c(d$estimate, d$p.value), file = \"~/Public/git/FPM_Violations/RQ1/SpearmanResults.list\"," +
"append = TRUE, sep = \"@\")\n";
}
System.out.println(s);
}
for (int i = 0; i < 10; i ++) {
String s = "d<-cor.test( ~ X" + i + ".R + X" + i + ".T.R, \n";
s += " data=a,";
s += " method = \"spearman\",\n";
s += " continuity = FALSE,\n";
s += " conf.level = 0.95)\n";
s += "write(c(d$estimate, d$p.value), file = \"~/Public/git/FPM_Violations/RQ1/SpearmanResults.list\"," +
"append = TRUE, sep = \"@\")\n";
System.out.println(s);
}
for (int i = 0; i < 10; i ++) {
String s = "d<-cor.test( ~ X" + i + " + X" + i + ".T, \n";
s += " data=a,";
s += " method = \"pearson\",\n";
s += " continuity = FALSE,\n";
s += " conf.level = 0.95)\n";
s += "write(c(d$estimate, d$p.value), file = \"~/Public/git/FPM_Violations/RQ1/SpearmanResults.list\"," +
"append = TRUE, sep = \"@\")\n";
System.out.println(s);
}
for (int i = 0; i < 10; i ++) {
String s = "d<-cor.test( ~ X" + i + ".R + X" + i + ".T.R, \n";
s += " data=a,";
s += " method = \"pearson\",\n";
s += " continuity = FALSE,\n";
s += " conf.level = 0.95)\n";
s += "write(c(d$estimate, d$p.value), file = \"~/Public/git/FPM_Violations/RQ1/SpearmanResults.list\"," +
"append = TRUE, sep = \"@\")\n";
System.out.println(s);
}
for (int i = 0; i < 10; i ++) {
String s = "d<-cor.test( ~ X" + i + " + X" + i + ".T, \n";
s += " data=a,";
s += " method = \"kendall\",\n";
s += " continuity = FALSE,\n";
s += " conf.level = 0.95)\n";
s += "write(c(d$estimate, d$p.value), file = \"~/Public/git/FPM_Violations/RQ1/SpearmanResults.list\"," +
"append = TRUE, sep = \"@\")\n";
System.out.println(s);
}
for (int i = 0; i < 10; i ++) {
String s = "d<-cor.test( ~ X" + i + ".R + X" + i + ".T.R, \n";
s += " data=a,";
s += " method = \"kendall\",\n";
s += " continuity = FALSE,\n";
s += " conf.level = 0.95)\n";
s += "write(c(d$estimate, d$p.value), file = \"~/Public/git/FPM_Violations/RQ1/SpearmanResults.list\"," +
"append = TRUE, sep = \"@\")\n";
System.out.println(s);
}
}
}
src/main/java/edu/lu/uni/serval/statistics/Statistic.java
-947
View File
@@ -1,947 +0,0 @@
package edu.lu.uni.serval.statistics;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Scanner;
import edu.lu.uni.serval.config.Configuration;
import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.utils.ListSorter;
import edu.lu.uni.serval.utils.MapSorter;
public class Statistic {
private static Map<String, Integer> map1 = new HashMap<String, Integer>();
private static Map<String, Integer> map2 = new HashMap<>();
public static void main(String[] args) throws IOException {
/*
* DM_DEFAULT_ENCODING
* NP_NONNULL_RETURN_VIOLATION
* NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE
* NP_PARAMETER_MUST_BE_NONNULL_BUT_MARKED_AS_NULLABLE
* ODR_OPEN_DATABASE_RESOURCE
* PZLA_PREFER_ZERO_LENGTH_ARRAYS
* RI_REDUNDANT_INTERFACES
* RV_RETURN_VALUE_IGNORED_BAD_PRACTICE
* RV_RETURN_VALUE_IGNORED_NO_SIDE_EFFECT
* SE_NO_SERIALVERSIONID
* SF_SWITCH_NO_DEFAULT
* SIC_INNER_SHOULD_BE_STATIC
* SQL_PREPARED_STATEMENT_GENERATED_FROM_NONCONSTANT_STRING
* UC_USELESS_CONDITION
* UC_USELESS_OBJECT
* UCF_USELESS_CONTROL_FLOW
* WMI_WRONG_MAP_ITERATOR
*/
// /*
// * Quantities' distribution of all violation types.
// */
// quantityOfEachViolationType();
// /*
// * Widespread of each violation type.
// */
// widespreadOfEachViolationType();
// /*
// * Statistics of categories of all violation types.
// */
// statisticWithCategories();
//
// /*
// * Quantity of each violation type in each project.
// */
// reloadData();
//
// /*
// * Quantities' distribution of all fixed violation types.
// * Widespread of each fixed violation type.
// * Statistics of categories of all fixed violation types.
// * Quantity of each fixed violation type in each project.
// */
// statisticOfFixedViolations();
// fixedVSunfixed();
/**
* Do statistics from two files:
*/
statistics("../FPM_Violations/RQ1/all-leafnodes-per-project-vtype.csv", "", map1, 16918530, 730);
// statistics("../FPM_Violations/RQ1/distinct-fixed-summary-per-project-vtype.csv", "Fixed");
statistics("../FPM_Violations/RQ1/fixedViolations-v-1.0.csv", "Fixed_1.0", map2, 88927, 548);
// fixedVSunfixed();
StringBuilder builder = new StringBuilder();
for (Map.Entry<String, Integer> entry : map1.entrySet()) {
String key = entry.getKey();
builder.append(key + "," + entry.getValue() + "," + (map2.containsKey(key) ? map2.get(key) : 0) + "\n");
}
FileHelper.outputToFile(Configuration.ROOT_PATH + "RQ2/quantity-ratios.csv", builder, false);
// statisticsOfFixedViolations();
//rsync -avP gaia-cluster:/work/users/kliu/FixPattern/FPM_Violations/UnfixedViolations/BC_UNCONFIRMED_CAST/Sizes.list Sizes/Sizes1.list
// String s = "rsync -avP gaia-cluster:/work/users/kliu/FixPattern/FPM_Violations/UnfixedViolations_RQ3/";
// File[] files = new File(Configuration.ROOT_PATH + "RQ3_1/UnfixedInstances/").listFiles();
// int i = 0;
// for (File file : files) {
// if (file.getName().endsWith(".list")) {
// i ++;
// System.out.println(s + FileHelper.getFileNameWithoutExtension(file) + "/Sizes.list Sizes/Sizes" + i + ".list");
// }
// }
// tokenSizes();
}
public static void tokenSizes() {
List<Integer> sizes = new ArrayList<>();
List<File> files = FileHelper.getAllFilesInCurrentDiectory(Configuration.ROOT_PATH + "RQ3_1/Sizes/", ".list");
StringBuilder builder = new StringBuilder();
for (File file : files) {
try {
String content = FileHelper.readFile(file);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
while ((line = reader.readLine()) != null) {
sizes.add(Integer.parseInt(line));
builder.append("1,").append(line).append("\n");
}
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
System.out.println(sizes.size());
FileHelper.outputToFile(Configuration.ROOT_PATH + "RQ3_1/Sizes.csv", builder, false);
}
public static void statisticsOfFixedViolations() {
String statistic = "../FPM_Violations/OUTPUT3/";
List<File> files = FileHelper.getAllFiles(statistic, ".list");
int positions = 0;
int numV = 0;
int testAlarms = 0;
int nullGumTreeResults = 0;
int nullMappingGumTreeResults = 0;
int pureDeletion = 0;
int timeout = 0;
int noSourceCodeChagnes = 0;
int largeHunk = 0;
int nullSourceCode = 0;
int noStatementChanges = 0;
int nullDiffentry = 0;
int TestingInfo = 0;
int i = 0;
Map<String, Integer> types1 = new HashMap<>();
for (File file : files) {
if (file.getName().startsWith("statistic")) {
String content = FileHelper.readFile(file);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
i ++;
try {
while ((line = reader.readLine()) != null) {
if (line.startsWith("Positions")) {
positions += Integer.parseInt(line.substring(line.lastIndexOf(":") + 1).trim());
} else if (line.startsWith("NumViolations")) {
numV += Integer.parseInt(line.substring(line.lastIndexOf(":") + 1).trim());
} else
if (line.startsWith("TestViolations")) {
testAlarms += Integer.parseInt(line.substring(line.lastIndexOf(":") + 1).trim());
} else if (line.startsWith("NullGumTreeResults")) {
nullGumTreeResults += Integer.parseInt(line.substring(line.lastIndexOf(":") + 1).trim());
} else if (line.startsWith("NoSourceCodeChanges")) {
noSourceCodeChagnes += Integer.parseInt(line.substring(line.lastIndexOf(":") + 1).trim());
} else if (line.startsWith("NoStatementChanges")) {
noStatementChanges += Integer.parseInt(line.substring(line.lastIndexOf(":") + 1).trim());
} else if (line.startsWith("NullDiffEntry")) {
nullDiffentry += Integer.parseInt(line.substring(line.lastIndexOf(":") + 1).trim());
} else if (line.startsWith("NullMatchedGumTreeResults")) {
nullMappingGumTreeResults += Integer.parseInt(line.substring(line.lastIndexOf(":") + 1).trim());
} else if (line.startsWith("PureDeletion")) {
pureDeletion += Integer.parseInt(line.substring(line.lastIndexOf(":") + 1).trim());
} else if (line.startsWith("LargeHunk")) {
largeHunk += Integer.parseInt(line.substring(line.lastIndexOf(":") + 1).trim());
} else if (line.startsWith("NullSourceCode")) {
nullSourceCode += Integer.parseInt(line.substring(line.lastIndexOf(":") + 1).trim());
} else if (line.startsWith("Timeout")) {
timeout += Integer.parseInt(line.substring(line.lastIndexOf(":") + 1).trim());
} else if (line.startsWith("TestingInfo")) {
TestingInfo +=Integer.parseInt(line.substring(line.lastIndexOf(":") + 1).trim());
}
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
if (file.getName().startsWith("UnfixedV")) {
String content = FileHelper.readFile(file);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
try {
while ((line = reader.readLine()) != null) {
// types1
if (line.startsWith("## OAR [")) break;
String type = line.substring(0, line.indexOf(":"));
if (types1.containsKey(type)) {
types1.put(type, types1.get(type) + 1);
} else {
types1.put(type, 1);
}
if (line.startsWith("#NullSourceCode:")) {
System.out.println(line);
}
}
} catch(IOException e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
Map<String, Integer> types = new HashMap<>();
// FileInputStream fis = new FileInputStream("../FPM_Violations/OAR.FPM.4222208.stderr");
// Scanner scanner = new Scanner(fis);
// while (scanner.hasNextLine()) {
// String line = scanner.nextLine();
// if (line.startsWith("## OAR [")) break;
// String type = line.substring(0, line.indexOf(":"));
// if (types.containsKey(type)) {
// types.put(type, types.get(type) + 1);
// } else {
// types.put(type, 1);
// }
// }
// scanner.close();
// fis.close();
// int sum = 0;
// int sum2 = 0;
// for (Map.Entry<String, Integer> entry : types.entrySet()) {
// System.out.println(entry.getKey() + ": " + entry.getValue());
// if (!entry.getKey().startsWith("#PureDeletion")) {
// if (!entry.getKey().startsWith("#Timeout") && !entry.getKey().startsWith("#TestViolation"))
// sum += entry.getValue();
// else sum2 += entry.getValue();
// }
// }
int sum3 = 0;
int sum4 = 0;
int sum5 = 0;
for (Map.Entry<String, Integer> entry : types1.entrySet()) {
System.out.println(entry.getKey() + ": " + entry.getValue());
if (!entry.getKey().startsWith("#PureDeletion")) {
if (!entry.getKey().startsWith("#Timeout") && !entry.getKey().startsWith("#TestViolation"))
sum3 += entry.getValue();
else sum4 += entry.getValue();
sum5+= entry.getValue();
}
}
System.out.println(sum5);
System.out.println(i);
System.out.println("\n\nStatistics:\nPositions: " + positions);
System.out.println("NumViolations: " + numV);
System.out.println("\nTestViolation: " + testAlarms + " :: " + types.get("#TestViolation") + " :: " + types1.get("#TestViolation"));
System.out.println("NullGumTreeResults: " + nullGumTreeResults + " :: " + types.get("#NullGumTreeResults") + " :: " + types1.get("#NullGumTreeResults"));
System.out.println("NoSourceCodeChange: " + noSourceCodeChagnes + " :: " + types.get("#NoSourceCodeChange") + " :: " + types1.get("#NoSourceCodeChange"));
System.out.println("NoStatementChange: " + noStatementChanges + " :: " + types.get("#NoStatementChange") + " :: " + types1.get("#NoStatementChange"));
System.out.println("NullDiffEntry: " + nullDiffentry + " :: " + types.get("#NullDiffEntry") + " :: " + types1.get("#NullDiffEntry"));
System.out.println("NullMatchedGumTreeResult: " + nullMappingGumTreeResults + " :: " + types.get("#NullMatchedGumTreeResult") + " :: " + types1.get("#NullMatchedGumTreeResult"));
System.out.println("PureDeletion: " + pureDeletion + " :: " + types.get("#PureDeletion") + " :: " + types1.get("#PureDeletion"));
System.out.println("LargeHunk: " + largeHunk + " :: " + types.get("#LargeHunk") + " :: " + types1.get("#LargeHunk"));
System.out.println("NullSourceCode: " + nullSourceCode + " :: " + types.get("#NullSourceCode") + " :: " + types1.get("#NullSourceCode"));
System.out.println("Timeout: " + timeout + " :: " + types.get("#Timeout") + " :: " + types1.get("#Timeout"));
System.out.println("TestingInfo: " + TestingInfo + " :: " + types.get("#TestingInfo") + " :: " + types1.get("#TestingInfo"));
// System.out.println("A: " + (positions + testAlarms + timeout));
// System.out.println("B: " + (numV + testAlarms + timeout));
System.out.println(testAlarms + nullGumTreeResults + noSourceCodeChagnes + noStatementChanges +
nullDiffentry + nullMappingGumTreeResults + nullSourceCode + timeout + TestingInfo);
// System.out.println(nullGumTreeResults + noSourceCodeChagnes + noStatementChanges +
// nullDiffentry + nullMappingGumTreeResults + nullSourceCode + TestingInfo);
// System.out.println(testAlarms + timeout);
// System.out.println(sum);
// System.out.println(sum2);
System.out.println(sum3);
System.out.println(sum4);
System.out.println(types1);
/*32690 56237 ,, 88927 31782
* Statistics:
TestViolation: 4682 :: null :: null
NullGumTreeResults: 0 :: null :: null
NoSourceCodeChange: 7010 :: null :: 6984
NoStatementChange: 163 :: null :: 163
NullDiffEntry: 6943 :: null :: 6943
NullMatchedGumTreeResult: 25747 :: null :: 25747
PureDeletion: 10815 :: null :: 10815
LargeHunk: 6318 :: null :: 6318
NullSourceCode: 1025 :: null :: 1025
Timeout: 0 :: null :: null
TestingInfo: 0 :: null :: null
A: 4682
B: 4682
51888
47206
4682
47180
0
*/
}
public static void statistics(String fileName, String type, Map<String, Integer> map, int totalQ, int totalS) throws IOException {
FileInputStream fis = new FileInputStream(fileName);
Scanner scanner = new Scanner(fis);
Map<String, Integer> violationTypesMap = new HashMap<>();
Map<String, Integer> projectsMap = new HashMap<>();
Map<String, Integer> perVperProjMap = new HashMap<>();
Map<String, List<String>> perVProjs = new HashMap<>();
Map<String, List<String>> widespreadViolationsMap = new HashMap<>();
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
String[] elements = line.split(",");
String projectName = elements[0];
String violationType = elements[1];
int quantity = Integer.parseInt(elements[2]);
addToMap(projectsMap, projectName, quantity);
addToMap(violationTypesMap, violationType, quantity);
addToMap(perVperProjMap, projectName + "," + violationType, quantity);
if (widespreadViolationsMap.containsKey(violationType)) {
List<String> projectList = widespreadViolationsMap.get(violationType);
if (!projectList.contains(projectName)) {
projectList.add(projectName);
}
} else {
List<String> projectList = new ArrayList<>();
projectList.add(projectName);
widespreadViolationsMap.put(violationType, projectList);
}
if (perVProjs.containsKey(violationType)) {
perVProjs.get(violationType).add(projectName);
} else {
List<String> projs = new ArrayList<>();
projs.add(projectName);
perVProjs.put(violationType, projs);
}
}
scanner.close();
fis.close();
// Category
MapSorter<String, Integer> sorter = new MapSorter<String, Integer>();
violationTypesMap = sorter.sortByValueDescending(violationTypesMap);
projectsMap = sorter.sortByValueDescending(projectsMap);
Map<String, String> categories = new HashMap<>();
Map<String, List<String>> categoryVList = new HashMap<>();
String content = FileHelper.readFile("../FPM_Violations/RQ1/ViolationCategory.list");
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
while ((line = reader.readLine()) != null) {
String[] elements = line.split("@@");
categories.put(elements[1], elements[0]); // Violation type, category
}
reader.close();
// Sort Violation types by widespread.
Map<String, Integer> widespreadOfAllViolations = new HashMap<>();
for (Map.Entry<String, List<String>> entry : widespreadViolationsMap.entrySet()) {
widespreadOfAllViolations.put(entry.getKey(), entry.getValue().size());
}
widespreadOfAllViolations = sorter.sortByValueDescending(widespreadOfAllViolations);
StringBuilder wbuilder = new StringBuilder("Type,Identifier,Quantity,Category\n");
int identifier1 = 0;
for (Map.Entry<String, Integer> entry : widespreadOfAllViolations.entrySet()) {
identifier1 ++;
String violationType = entry.getKey();
String category = categories.get(violationType);
if (category == null || category.equals("null")) {
category = "Other";
}
wbuilder.append(violationType + "," + identifier1 + "," + entry.getValue() + "," + category + "\n");
if (categoryVList.containsKey(category)) {
categoryVList.get(category).add(violationType);
} else {
List<String> list = new ArrayList<>();
list.add(violationType);
categoryVList.put(category, list);
}
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Widespread-per-" + type + "V-Type.csv", wbuilder, false);
// output statistics
List<String> sortedViolationTypes = new ArrayList<>();
Map<String, Integer> quantityOfCategory = new HashMap<>();
Map<String, List<String>> violationTypesOfCategory = new HashMap<>();
Map<String, List<String>> projectsOfCategory = new HashMap<>();
StringBuilder violationsBuilder = new StringBuilder("Type,Identifier,Quantity,Ratio,Widespread,Ratio2, Categories\n");
int identifier = 0;
for (Map.Entry<String, Integer> entry : violationTypesMap.entrySet()) {
String violationType = entry.getKey();
int quantity = entry.getValue();
identifier ++;
String category = categories.get(violationType);
if (category == null || category.equals("null")) {
category = "Other";
}
int spread = widespreadViolationsMap.get(violationType).size();
violationsBuilder.append(violationType + "," + identifier + "," + quantity + "," + ((double) quantity / totalQ * 100) + ","
+ spread + "," + ((double) spread / totalS * 100) + "," + category + "\n");
if (quantityOfCategory.containsKey(category)) {
quantityOfCategory.put(category, quantityOfCategory.get(category) + quantity);
} else {
quantityOfCategory.put(category, quantity);
}
if (violationTypesOfCategory.containsKey(category)) {
List<String> violationTypes = violationTypesOfCategory.get(category);
if (!violationTypes.contains(violationType)) {
violationTypes.add(violationType);
}
} else {
List<String> violationTypes = new ArrayList<>();
violationTypes.add(violationType);
violationTypesOfCategory.put(category, violationTypes);
}
if (projectsOfCategory.containsKey(category)) {
List<String> projs = projectsOfCategory.get(category);
List<String> projs2 = perVProjs.get(violationType);
for (String proj : projs2) {
if (!projs.contains(proj)) {
projs.add(proj);
}
}
} else {
projectsOfCategory.put(category, perVProjs.get(violationType));
}
sortedViolationTypes.add(violationType);
map.put(violationType, quantity);
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Quantity-per-" + type + "V-Type.csv", violationsBuilder, false);
StringBuilder pBuilder = new StringBuilder("Project,Quantity\n");
List<String> projectNames = new ArrayList<>();
for (Map.Entry<String, Integer> entry : projectsMap.entrySet()) {
String project = entry.getKey();
pBuilder.append(project + "," + entry.getValue() + "\n");
projectNames.add(project);
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Quantity-per-" + type + "Proj.csv", pBuilder, false);
StringBuilder categoryBuilder = new StringBuilder("Category,Quantity,Types,Projects\n");
for (Map.Entry<String, Integer> entry : quantityOfCategory.entrySet()) {
String key = entry.getKey();
categoryBuilder.append(key + "," + entry.getValue() + "," + violationTypesOfCategory.get(key).size() + "," + projectsOfCategory.get(key).size() + "\n");
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Quantity-per-" + type + "Category.csv", categoryBuilder, false);
StringBuilder builder = new StringBuilder("Projects");
String a = "";
String b = "";
for (int i = 0; i < sortedViolationTypes.size(); i ++) {
builder.append("," + sortedViolationTypes.get(i));
if (i < 50) {
a += "a$" + sortedViolationTypes.get(i) + ",";
b += "'" + (i + 1) + "',";
}
}
builder.append("\n");
System.out.println(a);
System.out.println(b);
// for (int i = 0; i < sortedViolationTypes.size(); i ++) {
// builder.append("," + i);
// }
// builder.append("\n");
StringBuilder bui = new StringBuilder();
for (int i = 0; i < projectNames.size(); i++) {
String projectName = projectNames.get(i);
builder.append(projectName);
for (int j = 0; j < sortedViolationTypes.size(); j ++) {
String violationType = sortedViolationTypes.get(j);
String key = projectName + "," + violationType;
Integer value = perVperProjMap.get(key);
if (value == null) {
value = 0;
} else {
bui.append(projectName).append(",").append(violationType).append(",").append(value).append(",").append(categories.get(violationType) == null ? "Other" : categories.get(violationType)).append("\n");
}
builder.append("," + value);
}
builder.append("\n");
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Distribution-per-project-per-" + type + "type.csv", builder, false);
FileHelper.outputToFile("../FPM_Violations/RQ1/Distribution-per-project-per-" + type + "type___.csv", bui, false);
StringBuilder ssbuilder = new StringBuilder("Type, Identifier, Quantity, Category\n");
Map<String, Integer> perTypePerProj = new HashMap<>();
Map<String, List<String>> categoryProjects = new HashMap<>();
List<String> others = new ArrayList<>();
for (int j = 0; j < sortedViolationTypes.size(); j ++) {
String violationType = sortedViolationTypes.get(j);
List<Integer> projs = new ArrayList<>();
for (int i = 0; i < projectNames.size(); i++) {
String projectName = projectNames.get(i);
String key = projectName + "," + violationType;
Integer value = perVperProjMap.get(key);
if (value != null) {
String category = categories.get(violationType);
if (category == null || category.equals("null")) {
category = "Other";
if (!others.contains(violationType)) {
others.add(violationType);
}
}
ssbuilder.append(violationType + "," + (j + 1) + "," + value + "," + category + "\n");
projs.add(value);
if (categoryProjects.containsKey(category)) {
List<String> projects = categoryProjects.get(category);
if (!projects.contains(projectName)) {
projects.add(projectName);
}
} else {
List<String> projects = new ArrayList<>();
projects.add(projectName);
categoryProjects.put(category, projects);
}
}
}
ListSorter<Integer> sorter2 = new ListSorter<Integer>(projs);
projs = sorter2.sortAscending();
int index = projs.size() % 2 == 0 ? projs.size() / 2 - 1 : projs.size() / 2;
perTypePerProj.put(violationType, projs.get(index));
}
System.out.println(others);
for (Map.Entry<String, List<String>> entry : categoryVList.entrySet()) {
System.out.println(entry.getKey() + ":" + entry.getValue().size() + ":" + categoryProjects.get(entry.getKey()).size());
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Distribution-per-project-per-" + type + "type2.csv", ssbuilder, false);
StringBuilder ssbuilder2 = new StringBuilder("Type, Identifier, Quantity, Category\n");
perTypePerProj = sorter.sortByValueDescending(perTypePerProj);
int j = 0;
for (Map.Entry<String, Integer> entry : perTypePerProj.entrySet()) {
String violationType = entry.getKey();
// System.out.println(entry.getValue());
j ++;
for (int i = 0; i < projectNames.size(); i++) {
String projectName = projectNames.get(i);
String key = projectName + "," + violationType;
Integer value = perVperProjMap.get(key);
if (value != null) {
String category = categories.get(violationType);
if (category == null || category.equals("null")) {
category = "Other";
}
ssbuilder2.append(violationType + "," + j + "," + value + "," + category + "\n");
}
}
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Distribution-per-project-per-" + type + "type3.csv", ssbuilder2, false);
// StringBuilder pVpPBuilder = new StringBuilder();
// for (Map.Entry<String, Integer> entry : perVperProjMap.entrySet()) {
// pVpPBuilder.append(entry.getKey() + "," + entry.getValue() + "\n");
// }
// FileHelper.outputToFile("../FPM_Violations/RQ1/Per-project-per-" + type + "type.csv", pVpPBuilder, false);
}
private static void addToMap(Map<String, Integer> map, String key, int value) {
if (map.containsKey(key)) {
map.put(key, map.get(key) + value);
} else {
map.put(key, value);
}
}
public static void quantityOfEachViolationType() {
Map<String, Integer> violationQuantities = readTypeQuantityMap("../FPM_Violations/RQ1/distinct-per-vtype.csv");
Map<String, Integer> violationWidespread = readWidespread("../FPM_Violations/RQ1/distinct-per-project-vtype.csv");
StringBuilder buidler = new StringBuilder("Violation Type, Identifier, Quantity, Quantity of Projects, Ratio\n");
int identifier = 0;
double totality = 15961605d;
int sum = 0;
for (Map.Entry<String, Integer> entry : violationQuantities.entrySet()) {
String key = entry.getKey();
identifier ++;
int quantity = entry.getValue();
sum += quantity;
buidler.append(key + "," + identifier + "," + quantity + "," + violationWidespread.get(key) + "," + (sum / totality) + "\n");
// if (identifier >= 50) break;
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Quantity-per-V-type.csv", buidler, false);
}
public static void widespreadOfEachViolationType() {
Map<String, Integer> violationWidespread = readWidespread("../FPM_Violations/RQ1/distinct-per-project-vtype.csv");
System.out.println("Violation types: " + violationWidespread.size());
StringBuilder buidler = new StringBuilder("Violation Type, Identifier, Quantity of Projects\n");
int identifier = 0;
for (Map.Entry<String, Integer> entry : violationWidespread.entrySet()) {
identifier ++;
buidler.append(entry.getKey() + "," + identifier + "," + entry.getValue() + "\n");
// if (identifier >= 50) break;
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Widespread-per-V-type.csv", buidler, false);
}
public static void statisticWithCategories() throws IOException {
Map<String, String> categories = new HashMap<>();
String content = FileHelper.readFile("../FPM_Violations/RQ1/ViolationCategory.list");
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
while ((line = reader.readLine()) != null) {
String[] elements = line.split("@@");
categories.put(elements[1], elements[0]); // Violation type, category
}
reader.close();
Map<String, Integer> quantityOfCategory = new HashMap<>();
String content2 = FileHelper.readFile("../FPM_Violations/RQ1/distinct-per-vtype.csv");
reader = new BufferedReader(new StringReader(content2));
line = reader.readLine();
while ((line = reader.readLine()) != null) {
String[] elements = line.split(",");
String violationType = elements[0];
int quantity = Integer.parseInt(elements[1]);
String category = categories.get(violationType);
if (quantityOfCategory.containsKey(category)) {
quantityOfCategory.put(category, quantityOfCategory.get(category) + quantity);
} else {
quantityOfCategory.put(category, quantity);
}
}
reader.close();
String categoryQuantity = "Category Type, Quantity\n";
for (Map.Entry<String, Integer> entry : quantityOfCategory.entrySet()) {
categoryQuantity += entry.getKey() + "," + entry.getValue() + "\n";
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Quantity-per-Category.csv", categoryQuantity, false);
}
public static void reloadData() {
// ordered projects by quantity of violations
Map<String, Integer> projectQuantities = readTypeQuantityMap("../FPM_Violations/RQ1/distinct-per-project.csv");
List<String> projectNames = new ArrayList<>();
for (Map.Entry<String, Integer> entry : projectQuantities.entrySet()) {
projectNames.add(entry.getKey());
}
// get ordered types by quantity of violations.
Map<String, Integer> violations = readTypeQuantityMap("../FPM_Violations/RQ1/distinct-per-vtype.csv");
List<String> violationTypes = new ArrayList<>();
for (Map.Entry<String, Integer> entry : violations.entrySet()) {
violationTypes.add(entry.getKey());
}
Map<String, String> perVperProjs = readData("../FPM_Violations/RQ1/distinct-per-project-vtype.csv");
StringBuilder builder = new StringBuilder("Projects");
for (int i = 0; i < violationTypes.size(); i ++) {
builder.append("," + violationTypes.get(i));
}
builder.append("\n");
for (int i = 0; i < projectNames.size(); i++) {
String projectName = projectNames.get(i);
builder.append(projectName);
for (int j = 0; j < violationTypes.size(); j ++) {
String violationType = violationTypes.get(j);
String key = projectName + "," + violationType;
String value = perVperProjs.get(key);
if (value == null) {
value = "0";
}
builder.append("," + value);
}
builder.append("\n");
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Distribution-per-project-per-type.csv", builder, false);
}
public static void statisticOfFixedViolations() throws IOException {
String fileName = "../FPM_Violations/RQ1/fixed-alarms-v1.0.list";
FileInputStream fis = new FileInputStream(fileName);
Scanner scanner = new Scanner(fis);
Map<String, Integer> violations = new HashMap<>();
Map<String, Integer> projects = new HashMap<>();
Map<String, Integer> perVperPro = new HashMap<>();
Map<String, List<String>> widespreadFixedV = new HashMap<>();
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
String[] elements = line.split(":");
String violationType = elements[0];
String projectName = elements[1];
if (violations.containsKey(violationType)) {
violations.put(violationType, violations.get(violationType) + 1);
} else {
violations.put(violationType, 1);
}
if (projects.containsKey(projectName)) {
projects.put(projectName, projects.get(projectName) + 1);
} else {
projects.put(projectName, 1);
}
String perVperProj = projectName + "," + violationType;
if (perVperPro.containsKey(perVperProj)) {
perVperPro.put(perVperProj, perVperPro.get(perVperProj) + 1);
} else {
perVperPro.put(perVperProj, 1);
}
if (widespreadFixedV.containsKey(violationType)) {
List<String> projectList = widespreadFixedV.get(violationType);
if (!projectList.contains(projectName)) {
projectList.add(projectName);
}
} else {
List<String> projectList = new ArrayList<>();
projectList.add(projectName);
widespreadFixedV.put(violationType, projectList);
}
}
scanner.close();
fis.close();
MapSorter<String, Integer> sorter = new MapSorter<String, Integer>();
violations = sorter.sortByValueDescending(violations);
projects = sorter.sortByValueDescending(projects);
Map<String, String> categories = new HashMap<>();
String content = FileHelper.readFile("../FPM_Violations/RQ1/ViolationCategory.list");
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
while ((line = reader.readLine()) != null) {
String[] elements = line.split("@@");
categories.put(elements[1], elements[0]); // Violation type, category
}
reader.close();
List<String> violationTypes = new ArrayList<>();
Map<String, Integer> quantityOfCategory = new HashMap<>();
StringBuilder violationsBuilder = new StringBuilder();
for (Map.Entry<String, Integer> entry : violations.entrySet()) {
String violationType = entry.getKey();
int quantity = entry.getValue();
violationsBuilder.append(violationType + "," + quantity + "," + widespreadFixedV.get(violationType).size() + "\n");
String category = categories.get(violationType);
if (quantityOfCategory.containsKey(category)) {
quantityOfCategory.put(category, quantityOfCategory.get(category) + quantity);
} else {
quantityOfCategory.put(category, quantity);
}
violationTypes.add(violationType);
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Quantity-per-FixedV-Type.csv", violationsBuilder, false);
StringBuilder pBuilder = new StringBuilder();
List<String> projectNames = new ArrayList<>();
for (Map.Entry<String, Integer> entry : projects.entrySet()) {
String project = entry.getKey();
pBuilder.append(project + "," + entry.getValue() + "\n");
projectNames.add(project);
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Quantity-per-FixedV-Proj.csv", pBuilder, false);
StringBuilder categoryBuilder = new StringBuilder();
for (Map.Entry<String, Integer> entry : quantityOfCategory.entrySet()) {
categoryBuilder.append(entry.getKey() + "," + entry.getValue() + "\n");
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Quantity-per-Fixed-Category.csv", categoryBuilder, false);
StringBuilder builder = new StringBuilder("Projects");
for (int i = 0; i < violationTypes.size(); i ++) {
builder.append("," + violationTypes.get(i));
if (i < 500) {
}
}
builder.append("\n");
for (int i = 0; i < projectNames.size(); i++) {
String projectName = projectNames.get(i);
builder.append(projectName);
for (int j = 0; j < violationTypes.size(); j ++) {
String violationType = violationTypes.get(j);
String key = projectName + "," + violationType;
Integer value = perVperPro.get(key);
if (value == null) {
value = 0;
}
builder.append("," + value);
}
builder.append("\n");
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Distribution-per-project-per-Fixed-type.csv", builder, false);
StringBuilder pVpPBuilder = new StringBuilder();
for (Map.Entry<String, Integer> entry : perVperPro.entrySet()) {
pVpPBuilder.append(entry.getKey() + "," + entry.getValue() + "\n");
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Per-project-per-Fixed-type.csv", pVpPBuilder, false);
}
public static void fixedVSunfixed() throws IOException {
String fileName = "../FPM_Violations/RQ1/Quantity-per-V-type.csv"; // all violations
Map<String, Integer> allViolations = readTypeQuantityMap(fileName);
String file = "../FPM_Violations/RQ1/Quantity-per-FixedV-Type.csv"; // fixed violations
Map<String, Integer> fixedViolations = readTypeQuantityMap(file);
MapSorter<String, Integer> sorter = new MapSorter<>();
allViolations = sorter.sortByValueDescending(allViolations);
StringBuilder builder = new StringBuilder("Type,fixed,unfixed,all,fixed Ratio,unfixed Ratio\n");
for (Map.Entry<String, Integer> entry : allViolations.entrySet()) {
String violationType = entry.getKey();
int quantity = entry.getValue();
Integer fixedQuantity = fixedViolations.get(violationType);
if (fixedQuantity == null) {
builder.append(violationType + "," + 0 + "," + quantity + "," + quantity + ",0.0,1\n");
} else {
builder.append(violationType + "," + fixedQuantity + "," + (quantity)
+ "," + quantity + "," + ((double)fixedQuantity) / ((double) quantity)
+ "," + ((double)(quantity)) / ((double) quantity) + "\n");
}
}
FileHelper.outputToFile("../FPM_Violations/RQ1/Distribution-per-Fixed-type-VS-per-unFixed-type.csv", builder, false);
}
private static Map<String, String> readData(String fileName) {
Map<String, String> perVperPros = new HashMap<>();
FileInputStream fis = null;
Scanner scanner = null;
try {
fis = new FileInputStream(fileName);
scanner = new Scanner(fis);
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
String[] elements = line.split(",");
String key = elements[0] + "," + elements[1];
String value = elements[2];
perVperPros.put(key, value);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
try {
scanner.close();
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return perVperPros;
}
private static Map<String, Integer> readTypeQuantityMap(String fileName) {
Map<String, Integer> map = new HashMap<>();
String fileContent = FileHelper.readFile(fileName);
BufferedReader reader = new BufferedReader(new StringReader(fileContent));
try {
String line= reader.readLine();
while ((line = reader.readLine()) != null) {
String[] elements = line.split(",");
map.put(elements[0], Integer.valueOf(elements[2]));
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
MapSorter<String, Integer> sorter = new MapSorter<>();
map = sorter.sortByValueDescending(map);
return map;
}
private static Map<String, Integer> readWidespread(String fileName) {
Map<String, Integer> violationWidespread = new HashMap<String, Integer>();
String fileContent = FileHelper.readFile(fileName);
BufferedReader reader = new BufferedReader(new StringReader(fileContent));
try {
String line = null;
while ((line = reader.readLine()) != null) {
String[] strArray = line.split(",");
String key = strArray[1];
if (violationWidespread.containsKey(key)) {
violationWidespread.put(key, violationWidespread.get(key) + 1);
} else {
violationWidespread.put(key, 1);
}
// String[] elements = line.split(",");
// violationWidespread.put(elements[0], Integer.valueOf(elements[1]));
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
MapSorter<String, Integer> sorter = new MapSorter<>();
violationWidespread = sorter.sortByValueDescending(violationWidespread);
return violationWidespread;
}
}
src/main/java/edu/lu/uni/serval/statistics/TenFoldPossibilities.java
-243
View File
@@ -1,243 +0,0 @@
package edu.lu.uni.serval.statistics;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.utils.MapSorter;
public class TenFoldPossibilities {
public static void main(String[] args) throws IOException {
rankingInTenFolds("../FPM_Violations/RQ1/TenFolds/", "../FPM_Violations/RQ1/Quantity-per-V-type.csv", "TenFolds"); // ten folds of all projects.
rankingInTenFolds("../FPM_Violations/RQ1/FixedTenFolds-1/", "../FPM_Violations/RQ1/Quantity-per-FixedV-Type.csv", "FixedTenFolds-1");// all fixed violation types are contained.
}
public static void rankingInTenFolds(String tenFoldFiles, String typeFile, String outputPath) throws NumberFormatException, IOException {
List<File> files = FileHelper.getAllFilesInCurrentDiectory(tenFoldFiles, ".list");
Map<Integer, Map<String, Double>> tenFoldsOfAllViolationTypes = new HashMap<>();
Map<Integer, Map<String, Double>> tenFoldsOfAllFixedViolationTypes1 = new HashMap<>();
Map<Integer, Map<String, Double>> tenFoldsOfAllFixedViolationTypes2 = new HashMap<>();
Map<Integer, Map<String, Double>> testingTenFoldsOfAllViolationTypes = new HashMap<>();
Map<Integer, Map<String, Double>> testingTenFoldsOfAllFixedViolationTypes1 = new HashMap<>();
Map<Integer, Map<String, Double>> testingTenFoldsOfAllFixedViolationTypes2 = new HashMap<>();
for (int i = 0; i < 10; i ++) {
List<String> projects = selectPorjects(files, i);
// the distribution of all violation types in these 9 sub-sets.
Map<String, Integer> violations = new HashMap<>();
int quantityOfAllViolations = readViolationsPerProject(violations, projects, "../FPM_Violations/RQ1/all-leafnodes-per-project-vtype.csv");
// the distribution of all fixed violation types in these 9 sub-sets.
Map<String, Integer> fixedViolations = new HashMap<>();
int quantityOfAllFixedViolations = readViolationsPerProject(fixedViolations, projects, "../FPM_Violations/RQ1/distinct-fixed-summary-per-project-vtype.csv");
// Ratio of each violation type in all violations.
Map<String, Double> ratioOfEachViolationType = new HashMap<>(); // ratio of each violation type
Map<String, Double> ratioOfEachFixedViolationType1 = new HashMap<>(); // quantity of each fixed violation type / quantity of each violation type.
Map<String, Double> ratioOfEachFixedViolationType2 = new HashMap<>(); // quantity of each fixed violation type / quantityOfAllFixedViolations
for (Map.Entry<String, Integer> entry : violations.entrySet()) {
String violationType = entry.getKey();
int quantity = entry.getValue();
double ratio = (double) quantity / quantityOfAllViolations;
ratioOfEachViolationType.put(violationType, ratio);
if (fixedViolations.containsKey(violationType)) {
int quantity1 = fixedViolations.get(violationType);
double ratio1 = (double)quantity1 / quantity;
double ratio2 = (double)quantity1 / quantityOfAllFixedViolations;
ratioOfEachFixedViolationType1.put(violationType, ratio1);
ratioOfEachFixedViolationType2.put(violationType, ratio2);
} else {
ratioOfEachFixedViolationType1.put(violationType, 0d);
}
}
tenFoldsOfAllViolationTypes.put(i, ratioOfEachViolationType);
tenFoldsOfAllFixedViolationTypes1.put(i, ratioOfEachFixedViolationType1);
tenFoldsOfAllFixedViolationTypes2.put(i, ratioOfEachFixedViolationType2);
// Testing Data
List<String> testingProjects = readList(files.get(i));
Map<String, Integer> testingViolations = new HashMap<>();
int testingQuantitifOfAllViolations = readViolationsPerProject(testingViolations, testingProjects, "../FPM_Violations/RQ1/all-leafnodes-per-project-vtype.csv");
Map<String, Integer> testingFixedViolations = new HashMap<>();
int testingQuantityOfAllFixedViolations = readViolationsPerProject(testingFixedViolations, testingProjects, "../FPM_Violations/RQ1/distinct-fixed-summary-per-project-vtype.csv");
Map<String, Double> testingRatioOfEachViolationType = new HashMap<>();
Map<String, Double> testingRatioOfEachFixedViolationType1 = new HashMap<>();
Map<String, Double> testingRatioOfEachFixedViolationType2 = new HashMap<>();
for (Map.Entry<String, Integer> entry : testingViolations.entrySet()) {
String violationType = entry.getKey();
int quantity = entry.getValue();
double ratio = (double) quantity / testingQuantitifOfAllViolations;
testingRatioOfEachViolationType.put(violationType, ratio);
if (testingFixedViolations.containsKey(violationType)) {
int quantity1 = testingFixedViolations.get(violationType);
testingRatioOfEachFixedViolationType1.put(violationType, (double)quantity1 / quantity);
testingRatioOfEachFixedViolationType2.put(violationType, (double)quantity1 / testingQuantityOfAllFixedViolations);
} else {
testingRatioOfEachFixedViolationType1.put(violationType, 0d);
}
}
testingTenFoldsOfAllViolationTypes.put(i, testingRatioOfEachViolationType);
testingTenFoldsOfAllFixedViolationTypes1.put(i, testingRatioOfEachFixedViolationType1);
testingTenFoldsOfAllFixedViolationTypes2.put(i, testingRatioOfEachFixedViolationType2);
}
List<String> violationTypes = readTypes(typeFile);
outputTenFolds(tenFoldsOfAllViolationTypes, testingTenFoldsOfAllViolationTypes, violationTypes, "../FPM_Violations/RQ1/" + outputPath + "/Ten-fold-all-violation-type.csv");
outputTenFolds(tenFoldsOfAllFixedViolationTypes1, testingTenFoldsOfAllFixedViolationTypes1, violationTypes, "../FPM_Violations/RQ1/" + outputPath + "/Ten-fold-all-fixed-violation-type1.csv");
outputTenFolds(tenFoldsOfAllFixedViolationTypes2, testingTenFoldsOfAllFixedViolationTypes2, violationTypes, "../FPM_Violations/RQ1/" + outputPath + "/Ten-fold-all-fixed-violation-type2.csv");
}
private static List<String> readTypes(String fileName) throws IOException {
List<String> violationTypes = new ArrayList<>();
String content = FileHelper.readFile(fileName);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = reader.readLine();
while ((line = reader.readLine()) != null) {
String[] elements = line.split(",");
violationTypes.add(elements[0]);
}
reader.close();
return violationTypes;
}
private static void outputTenFolds(Map<Integer, Map<String, Double>> ratiosMap, Map<Integer, Map<String, Double>> testingMap, List<String> violationTypes, String fileName) {
StringBuilder builder = new StringBuilder("Violatype, 0, 0-R, 0-T, 0-T-R, 1, 1-R, 1-T, 1-T-R, 2, 2-R, 2-T, 2-T-R, 3, 3-R, 3-T, 3-T-R, 4, 4-R, 4-T, 4-T-R, 5, 5-R, 5-T, 5-T-R, 6, 6-R, 6-T, 6-T-R, 7, 7-R, 7-T, 7-T-R, 8, 8-R, 8-T, 8-T-R, 9, 9-R, 9-T, 9-T-R\n");
Map<Integer, Map<String, Integer>> ratiosRankingMap = new HashMap<>();
Map<Integer, Map<String, Integer>> testingDataRankingMap = new HashMap<>();
for (String type : violationTypes) {
builder.append(type);
for (int i = 0; i < 10; i ++) {
Map<String, Double> ratios = ratiosMap.get(i);
Map<String, Double> testingData = testingMap.get(i);
Map<String, Integer> ratiosRanking;
Map<String, Integer> testingDataRanking;
if (!ratiosRankingMap.containsKey(i)) {
ratiosRanking = rankAlarmTypes(ratios);
testingDataRanking = rankAlarmTypes(testingData);
} else {
ratiosRanking = ratiosRankingMap.get(i);
testingDataRanking = testingDataRankingMap.get(i);
}
if (ratios.containsKey(type)) {
builder.append(", " + ratios.get(type));//.toString().replace(".", ",")
} else {
builder.append(", 0.0");
}
if (ratiosRanking.containsKey(type)) {
builder.append(", " + ratiosRanking.get(type));
} else {
builder.append(", " + (ratiosRanking.size() + 1));
}
if (testingData.containsKey(type)) {
builder.append(", " + testingData.get(type));
} else {
builder.append(", 0.0");
}
if (testingDataRanking.containsKey(type)) {
builder.append(", " + testingDataRanking.get(type));
} else {
builder.append(", " + (testingDataRanking.size() + 1));
}
}
builder.append("\n");
}
FileHelper.outputToFile(fileName, builder, false);
}
private static Map<String, Integer> rankAlarmTypes(Map<String, Double> possibilityMap) {
MapSorter<String, Double> sorter = new MapSorter<>();
possibilityMap = sorter.sortByValueDescending(possibilityMap);
Map<String, Integer> ranking = new HashMap<>();
int ranker = 0;
double possibility = 0;
for (Map.Entry<String, Double> entry : possibilityMap.entrySet()) {
String alarmType = entry.getKey();
double value = entry.getValue();
if (possibility != value) {
ranker = ranking.size() + 1;
}
ranking.put(alarmType, ranker);
}
return ranking;
}
private static int readViolationsPerProject(Map<String, Integer> violations, List<String> projects, String fileName) throws NumberFormatException, IOException {
int quantityOfAllViolations = 0;
String content = FileHelper.readFile(fileName);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
while ((line = reader.readLine()) != null) {
String[] elements = line.split(",");
String projectName = elements[0];
if (projects.contains(projectName)) {
String alarmType = elements[1];
int quantity = Integer.parseInt(elements[2]);
quantityOfAllViolations += quantity;
if (violations.containsKey(alarmType)) {
violations.put(alarmType, violations.get(alarmType) + quantity);
} else {
violations.put(alarmType, quantity);
}
}
}
reader.close();
return quantityOfAllViolations;
}
private static List<String> selectPorjects(List<File> files, int i) {
List<String> projects = new ArrayList<>();
for (File file : files) {
if (file.getName().equals("Fold_" + i + ".list")) {
continue;
}
projects.addAll(readList(file));
}
return projects;
}
private static List<String> readList(File file) {
List<String> projects = new ArrayList<>();
String content = FileHelper.readFile(file);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
try {
while ((line = reader.readLine()) != null) {
projects.add(line);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return projects;
}
}
src/main/java/edu/lu/uni/serval/statistics/TenFoldPossibilities2.java
-256
View File
@@ -1,256 +0,0 @@
package edu.lu.uni.serval.statistics;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import edu.lu.uni.serval.utils.FileHelper;
import edu.lu.uni.serval.utils.MapSorter;
/**
* Fixed violation types are selected by their widespread.
*
* @author kui.liu
*
*/
public class TenFoldPossibilities2 {
public static void main(String[] args) throws IOException {
rankingInTenFolds("../FPM_Violations/RQ1/FixedTenFolds-W/", "../FPM_Violations/RQ1/Widespread-per-FixedV-Type.csv", "FixedTenFolds-W");
}
public static void rankingInTenFolds(String tenFoldFiles, String typeFile, String outputPath) throws NumberFormatException, IOException {
int topNumber = 10;
List<String> violationTypes = readTypes(typeFile, topNumber);
List<File> files = FileHelper.getAllFilesInCurrentDiectory(tenFoldFiles, ".list");
Map<Integer, Map<String, Double>> tenFoldsOfAllViolationTypes = new HashMap<>();
Map<Integer, Map<String, Double>> tenFoldsOfAllFixedViolationTypes1 = new HashMap<>();
Map<Integer, Map<String, Double>> tenFoldsOfAllFixedViolationTypes2 = new HashMap<>();
Map<Integer, Map<String, Double>> testingTenFoldsOfAllViolationTypes = new HashMap<>();
Map<Integer, Map<String, Double>> testingTenFoldsOfAllFixedViolationTypes1 = new HashMap<>();
Map<Integer, Map<String, Double>> testingTenFoldsOfAllFixedViolationTypes2 = new HashMap<>();
for (int i = 0; i < 10; i ++) {
List<String> projects = selectPorjects(files, i);
// all violations in these 9 sub-sets.
Map<String, Integer> violations = new HashMap<>();
int quantityOfAllViolations = readViolationsPerProject(violationTypes, violations, projects, "../FPM_Violations/RQ1/all-leafnodes-per-project-vtype.csv");
Map<String, Integer> fixedViolations = new HashMap<>();
int quantityOfAllFixedViolations = readViolationsPerProject(violationTypes, fixedViolations, projects, "../FPM_Violations/RQ1/distinct-fixed-summary-per-project-vtype.csv");
// Ratio of each violation type in all violations.
Map<String, Double> ratioOfEachViolationType = new HashMap<>(); // ratio of each violation type
Map<String, Double> ratioOfEachFixedViolationType1 = new HashMap<>(); // quantity of each fixed violation type / quantity of each violation type.
Map<String, Double> ratioOfEachFixedViolationType2 = new HashMap<>(); // quantity of each fixed violation type / quantityOfAllFixedViolations
for (Map.Entry<String, Integer> entry : violations.entrySet()) {
String violationType = entry.getKey();
int quantity = entry.getValue();
double ratio = (double) quantity / quantityOfAllViolations;
ratioOfEachViolationType.put(violationType, ratio);
if (fixedViolations.containsKey(violationType)) {
int quantity1 = fixedViolations.get(violationType);
double ratio1 = (double)quantity1 / quantity;
double ratio2 = (double)quantity1 / quantityOfAllFixedViolations;
ratioOfEachFixedViolationType1.put(violationType, ratio1);
ratioOfEachFixedViolationType2.put(violationType, ratio2);
} else {
ratioOfEachFixedViolationType1.put(violationType, 0d);
}
}
tenFoldsOfAllViolationTypes.put(i, ratioOfEachViolationType);
tenFoldsOfAllFixedViolationTypes1.put(i, ratioOfEachFixedViolationType1);
tenFoldsOfAllFixedViolationTypes2.put(i, ratioOfEachFixedViolationType2);
// Testing Data
List<String> testingProjects = readList(files.get(i));
Map<String, Integer> testingViolations = new HashMap<>();
int testingQuantitifOfAllViolations = readViolationsPerProject(violationTypes, testingViolations, testingProjects, "../FPM_Violations/RQ1/all-leafnodes-per-project-vtype.csv");
Map<String, Integer> testingFixedViolations = new HashMap<>();
int testingQuantityOfAllFixedViolations = readViolationsPerProject(violationTypes, testingFixedViolations, testingProjects, "../FPM_Violations/RQ1/distinct-fixed-summary-per-project-vtype.csv");
Map<String, Double> testingRatioOfEachViolationType = new HashMap<>();
Map<String, Double> testingRatioOfEachFixedViolationType1 = new HashMap<>();
Map<String, Double> testingRatioOfEachFixedViolationType2 = new HashMap<>();
for (Map.Entry<String, Integer> entry : testingViolations.entrySet()) {
String violationType = entry.getKey();
int quantity = entry.getValue();
double ratio = (double) quantity / testingQuantitifOfAllViolations;
testingRatioOfEachViolationType.put(violationType, ratio);
if (testingFixedViolations.containsKey(violationType)) {
int quantity1 = testingFixedViolations.get(violationType);
testingRatioOfEachFixedViolationType1.put(violationType, (double)quantity1 / quantity);
testingRatioOfEachFixedViolationType2.put(violationType, (double)quantity1 / testingQuantityOfAllFixedViolations);
} else {
testingRatioOfEachFixedViolationType1.put(violationType, 0d);
}
}
testingTenFoldsOfAllViolationTypes.put(i, testingRatioOfEachViolationType);
testingTenFoldsOfAllFixedViolationTypes1.put(i, testingRatioOfEachFixedViolationType1);
testingTenFoldsOfAllFixedViolationTypes2.put(i, testingRatioOfEachFixedViolationType2);
}
outputTenFolds(tenFoldsOfAllViolationTypes, testingTenFoldsOfAllViolationTypes, violationTypes, "../FPM_Violations/RQ1/" + outputPath + "/Ten-fold-all-violation-type-" + topNumber + ".csv");
outputTenFolds(tenFoldsOfAllFixedViolationTypes1, testingTenFoldsOfAllFixedViolationTypes1, violationTypes, "../FPM_Violations/RQ1/" + outputPath + "/Ten-fold-all-fixed-violation-type1-" + topNumber + ".csv");
outputTenFolds(tenFoldsOfAllFixedViolationTypes2, testingTenFoldsOfAllFixedViolationTypes2, violationTypes, "../FPM_Violations/RQ1/" + outputPath + "/Ten-fold-all-fixed-violation-type2-" + topNumber + ".csv");
}
private static List<String> readTypes(String fileName, int number) throws IOException {
List<String> violationTypes = new ArrayList<>();
String content = FileHelper.readFile(fileName);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = reader.readLine();
int counter = 0;
while ((line = reader.readLine()) != null) {
String[] elements = line.split(",");
violationTypes.add(elements[0]);
if (++counter == number) break;
}
reader.close();
return violationTypes;
}
private static void outputTenFolds(Map<Integer, Map<String, Double>> ratiosMap, Map<Integer, Map<String, Double>> testingMap, List<String> violationTypes, String fileName) {
StringBuilder builder = new StringBuilder("Violatype, 0, 0-R, 0-T, 0-T-R, 1, 1-R, 1-T, 1-T-R, 2, 2-R, 2-T, 2-T-R, 3, 3-R, 3-T, 3-T-R, 4, 4-R, 4-T, 4-T-R, 5, 5-R, 5-T, 5-T-R, 6, 6-R, 6-T, 6-T-R, 7, 7-R, 7-T, 7-T-R, 8, 8-R, 8-T, 8-T-R, 9, 9-R, 9-T, 9-T-R\n");
Map<Integer, Map<String, Integer>> ratiosRankingMap = new HashMap<>();
Map<Integer, Map<String, Integer>> testingDataRankingMap = new HashMap<>();
for (String type : violationTypes) {
builder.append(type);
for (int i = 0; i < 10; i ++) {
Map<String, Double> ratios = ratiosMap.get(i);
Map<String, Double> testingData = testingMap.get(i);
Map<String, Integer> ratiosRanking;
Map<String, Integer> testingDataRanking;
if (!ratiosRankingMap.containsKey(i)) {
ratiosRanking = rankAlarmTypes(ratios);
testingDataRanking = rankAlarmTypes(testingData);
} else {
ratiosRanking = ratiosRankingMap.get(i);
testingDataRanking = testingDataRankingMap.get(i);
}
if (ratios.containsKey(type)) {
builder.append(", " + ratios.get(type));//.toString().replace(".", ",")
} else {
builder.append(", 0.0");
}
if (ratiosRanking.containsKey(type)) {
builder.append(", " + ratiosRanking.get(type));
} else {
builder.append(", " + (ratiosRanking.size() + 1));
}
if (testingData.containsKey(type)) {
builder.append(", " + testingData.get(type));
} else {
builder.append(", 0.0");
}
if (testingDataRanking.containsKey(type)) {
builder.append(", " + testingDataRanking.get(type));
} else {
builder.append(", " + (testingDataRanking.size() + 1));
}
}
builder.append("\n");
}
FileHelper.outputToFile(fileName, builder, false);
}
private static Map<String, Integer> rankAlarmTypes(Map<String, Double> possibilityMap) {
MapSorter<String, Double> sorter = new MapSorter<>();
possibilityMap = sorter.sortByValueDescending(possibilityMap);
Map<String, Integer> ranking = new HashMap<>();
int ranker = 0;
double possibility = 0;
for (Map.Entry<String, Double> entry : possibilityMap.entrySet()) {
String alarmType = entry.getKey();
double value = entry.getValue();
if (possibility != value) {
ranker = ranking.size() + 1;
}
ranking.put(alarmType, ranker);
}
return ranking;
}
private static int readViolationsPerProject(List<String> violationTypes, Map<String, Integer> violations, List<String> projects, String fileName) throws NumberFormatException, IOException {
int quantityOfAllViolations = 0;
String content = FileHelper.readFile(fileName);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
while ((line = reader.readLine()) != null) {
String[] elements = line.split(",");
String projectName = elements[0];
if (projects.contains(projectName)) {
String alarmType = elements[1];
if (violationTypes.contains(alarmType)) {
int quantity = Integer.parseInt(elements[2]);
quantityOfAllViolations += quantity;
if (violations.containsKey(alarmType)) {
violations.put(alarmType, violations.get(alarmType) + quantity);
} else {
violations.put(alarmType, quantity);
}
}
}
}
reader.close();
return quantityOfAllViolations;
}
private static List<String> selectPorjects(List<File> files, int i) {
List<String> projects = new ArrayList<>();
for (File file : files) {
if (file.getName().equals("Fold_" + i + ".list")) {
continue;
}
projects.addAll(readList(file));
}
return projects;
}
private static List<String> readList(File file) {
List<String> projects = new ArrayList<>();
String content = FileHelper.readFile(file);
BufferedReader reader = new BufferedReader(new StringReader(content));
String line = null;
try {
while ((line = reader.readLine()) != null) {
projects.add(line);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return projects;
}
}

Some files were not shown because too many files have changed in this diff Show More