Skip to content

Commit 07bd715

Browse files
TeamworkGuy2TeamworkGuy2
committed
0.14.0 - Moved twg2.parser.text package to new jtext-tokenizer library, renamed classes/methods with the word 'Fragment' to 'Token'; i.e. CodeFragmentType -> CodeTokenType, see CHANGELOG for full details.
1 parent 76734b1 commit 07bd715

File tree

94 files changed

+758
-3923
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

94 files changed

+758
-3923
lines changed

‎.classpath

+3-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
1+
<?xml version="1.0" encoding="UTF-8"?>
22
<classpath>
33
<classpathentry kind="src" path="src"/>
44
<classpathentry kind="src" path="plugins"/>
@@ -44,6 +44,8 @@
4444
<classpathentry kind="lib" path="C:/Users/TeamworkGuy2/Documents/Java/Libraries/jstreamish/bin/jstreamish.jar" sourcepath="/JStreamish"/>
4545
<classpathentry kind="lib" path="C:/Users/TeamworkGuy2/Documents/Java/Libraries/jtext-parser/bin/jtext_parser.jar" sourcepath="/JTextParser"/>
4646
<classpathentry kind="lib" path="C:/Users/TeamworkGuy2/Documents/Java/Libraries/jtext-template/bin/jtext_template.jar" sourcepath="/JTextTemplate"/>
47+
<classpathentry kind="lib" path="C:/Users/TeamworkGuy2/Documents/Java/Libraries/jtext-tokenizer/bin/jtext_tokenizer-with-tests.jar" sourcepath="/JTextTokenizer"/>
48+
<classpathentry kind="lib" path="C:/Users/TeamworkGuy2/Documents/Java/Libraries/jtext-tokenizer/bin/jtext_tokenizer.jar" sourcepath="/JTextTokenizer"/>
4749
<classpathentry kind="lib" path="C:/Users/TeamworkGuy2/Documents/Java/Libraries/jtext-util/bin/jtext_util.jar" sourcepath="/JTextUtil"/>
4850
<classpathentry kind="lib" path="C:/Users/TeamworkGuy2/Documents/Java/Libraries/jtree-walker/bin/jtree_walker.jar" sourcepath="/JTreeWalker"/>
4951
<classpathentry kind="lib" path="C:/Users/TeamworkGuy2/Documents/Java/Libraries/jtuples/bin/jtuples.jar" sourcepath="/JTuples"/>

‎CHANGELOG.md

+22-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,28 @@ This project does its best to adhere to [Semantic Versioning](http://semver.org/
44

55

66
--------
7-
###[0.13.0](N/A) - 2016-10-26
7+
###[0.14.0](N/A) - 2016-10-30
8+
#### Changed
9+
__Reduced library complexity/scope by moving twg2.parser.text conditions/tokenizers to separate [jtext-tokenizer](https://github.com/TeamworkGuy2/JTextTokenizer) library__
10+
* Moved twg2.parser.text package to jtext-tokenizer library
11+
* Moved twg2.parser.Inclusion to jtext-parser library
12+
* Renamed classes *Fragment -> *Token:
13+
* CodeFragment -> CodeToken
14+
* CodeFragmentType -> CodeTokenType
15+
* DocumentFragment -> TextToken and renamed methods:
16+
* getTextFragment() -> getToken()
17+
* getFragmentType() -> getTokenType()
18+
* DocumentFragmentText -> TextFragmentRefToken
19+
* CodeFragmentEnumSubSet -> CodeTokenEnumSubSet
20+
* Renamed twg2.parser.codeParser.tools.performance -> twg2.parser.codeParser.analytics
21+
* TokenizeStepDetails -> TokenizeStepLogger and now implements TypedLogger from jtext-tokenizer library
22+
23+
#### Removed
24+
* Removed twg2.parser.text package (moved to jtext-tokenizer library)
25+
26+
27+
--------
28+
###[0.13.0](https://github.com/TeamworkGuy2/JParseCode/commit/76734b17d16c67a89df7245a2cea2a1133c3b6b0) - 2016-10-26
829
#### Added
930
* Added ParameterSig and ParameterSigResolved 'parameterModifiers' field
1031
* Added KeywordUtil parameterModifiers() and isParameterModifier()

‎README.md

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
JParseCode
22
==============
3-
version: 0.13.0
3+
version: 0.14.0
44

55
In progress C#/Java/TypeScript parser tools built atop [JTextParser] (https://github.com/TeamworkGuy2/JTextParser), [Jackson] (https://github.com/FasterXML/jackson-core/) (core, databind, annotations) and half a dozen other utility libraries.
66

@@ -9,7 +9,7 @@ In progress C#/Java/TypeScript parser tools built atop [JTextParser] (https://gi
99
* A code first parser aimed at manipulating the resulting AST and writing it back as source code or JSON. With the goal of allowing simple language constructs like interfaces and data models to be transpiled to different languages.
1010

1111
### Not Goals:
12-
* NOT to create another compiler for C#, Java, or JS/TS. This project's parser expects valid code as input, the few error messages that are present are NOT design to highlight syntax errors in the input.
12+
* NOT to create another compiler for C#, Java, or JavaScript/TypeScript. This project's parser expects valid code as input, the few error messages that are present are NOT design to highlight syntax errors in the input.
1313
* NOT to create a valid AST for each supported language. Rather an AST like structure that supports the lowest common denominator between the targeted languages. This means many AST compromises are inevitable due to differences in language specs.
1414

1515

@@ -65,7 +65,7 @@ Java code to parser SimpleCs.cs (simple_cs_source_string is a string containing
6565
CodeFileSrc<CodeLanguage> simpleCsAst = ParseCodeFile.parseCode("SimpleCs.cs", CodeLanguageOptions.C_SHARP, simple_cs_source_string);
6666
WriteSettings ws = new WriteSettings(true, true, true, true);
6767

68-
for(Map.Entry<SimpleTree<DocumentFragmentText<CodeFragmentType>>, IntermClass.SimpleImpl<CsBlock>> block : CodeLanguageOptions.C_SHARP.getExtractor().extractClassFieldsAndMethodSignatures(simpleCsAst.getDoc())) {
68+
for(Map.Entry<SimpleTree<CodeToken>, ClassAst.SimpleImpl<CsBlock>> block : CodeLanguageOptions.C_SHARP.getExtractor().extractClassFieldsAndMethodSignatures(simpleCsAst.getDoc())) {
6969
CodeFileParsed.Simple<String, CsBlock> fileParsed = new CodeFileParsed.Simple<>("SimpleCs.cs", block.getValue(), block.getKey());
7070

7171
StringBuilder sb = new StringBuilder();

‎bin/jparse_code-with-tests.jar

-67.8 KB
Binary file not shown.

‎bin/jparse_code.jar

-68.4 KB
Binary file not shown.

‎package-lib.json

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"version" : "0.13.0",
2+
"version" : "0.14.0",
33
"name" : "jparse-code",
44
"description" : "An in-progress suite of parsing/transpilation tools for C#, Java, and TypeScript code. Generates simple JSON ASTs.",
55
"homepage" : "https://github.com/TeamworkGuy2/JParseCode",
@@ -25,6 +25,7 @@
2525
"json-stringify": "*",
2626
"jstreamish": "*",
2727
"jtext-parser": "*",
28+
"jtext-tokenizer": "*",
2829
"jtext-util": "*",
2930
"jtree-walker": "*",
3031
"jtuples": "*"

‎src/twg2/ast/interm/block/BlockAst.java

+3-3
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
import twg2.annotations.Immutable;
55
import twg2.ast.interm.classes.ClassSig;
66
import twg2.parser.codeParser.BlockType;
7-
import twg2.parser.fragment.CodeFragment;
7+
import twg2.parser.fragment.CodeToken;
88
import twg2.treeLike.simpleTree.SimpleTree;
99

1010
/**
@@ -14,11 +14,11 @@
1414
@Immutable
1515
public class BlockAst<T_BLOCK extends BlockType> {
1616
private final @Getter ClassSig.SimpleImpl declaration;
17-
private final @Getter SimpleTree<CodeFragment> blockTree;
17+
private final @Getter SimpleTree<CodeToken> blockTree;
1818
private final @Getter T_BLOCK blockType;
1919

2020

21-
public BlockAst(ClassSig.SimpleImpl declaration, SimpleTree<CodeFragment> blockTree, T_BLOCK blockType) {
21+
public BlockAst(ClassSig.SimpleImpl declaration, SimpleTree<CodeToken> blockTree, T_BLOCK blockType) {
2222
this.declaration = declaration;
2323
this.blockTree = blockTree;
2424
this.blockType = blockType;

‎src/twg2/ast/interm/field/FieldDef.java

+8-8
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@
1212
import twg2.parser.codeParser.AccessModifier;
1313
import twg2.parser.codeParser.extractors.DataTypeExtractor;
1414
import twg2.parser.codeParser.tools.NameUtil;
15-
import twg2.parser.fragment.CodeFragment;
16-
import twg2.parser.fragment.CodeFragmentType;
15+
import twg2.parser.fragment.CodeToken;
16+
import twg2.parser.fragment.CodeTokenType;
1717
import twg2.parser.output.WriteSettings;
1818
import twg2.text.stringEscape.StringEscapeJson;
1919
import twg2.treeLike.simpleTree.SimpleTree;
@@ -24,11 +24,11 @@
2424
*/
2525
@Immutable
2626
public class FieldDef extends FieldSig {
27-
private final @Getter SimpleTree<CodeFragment> initializer;
27+
private final @Getter SimpleTree<CodeToken> initializer;
2828

2929

3030
public FieldDef(String name, List<String> fullName, TypeSigSimple fieldType, List<AccessModifier> accessModifiers,
31-
List<AnnotationSig> annotations, List<String> comments, SimpleTree<CodeFragment> initializer) {
31+
List<AnnotationSig> annotations, List<String> comments, SimpleTree<CodeToken> initializer) {
3232
super(name, fullName, fieldType, accessModifiers, annotations, comments);
3333
this.initializer = initializer;
3434
}
@@ -68,12 +68,12 @@ public String toString() {
6868
/** Write a field initializer to a JSON field named 'initializer' if the value is a number, boolean, string, or null literal, else write it to a field named 'initializerExpression'
6969
* @throws IOException
7070
*/
71-
public static void initializerToJson(SimpleTree<CodeFragment> astNode, boolean preClosingComma, Appendable dst, WriteSettings st) throws IOException {
72-
CodeFragment data = null;
71+
public static void initializerToJson(SimpleTree<CodeToken> astNode, boolean preClosingComma, Appendable dst, WriteSettings st) throws IOException {
72+
CodeToken data = null;
7373
boolean isNumOrBoolOrNull = false;
7474
if(astNode != null && !astNode.hasChildren() && (data = astNode.getData()) != null &&
75-
(data.getFragmentType() == CodeFragmentType.STRING ||
76-
(isNumOrBoolOrNull = (data.getFragmentType() == CodeFragmentType.NUMBER || DataTypeExtractor.isBooleanLiteral(data) || DataTypeExtractor.isNullLiteral(data))))) {
75+
(data.getTokenType() == CodeTokenType.STRING ||
76+
(isNumOrBoolOrNull = (data.getTokenType() == CodeTokenType.NUMBER || DataTypeExtractor.isBooleanLiteral(data) || DataTypeExtractor.isNullLiteral(data))))) {
7777
if(preClosingComma) {
7878
dst.append(", ");
7979
}

‎src/twg2/ast/interm/field/FieldDefResolved.java

+3-3
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
import twg2.io.json.stringify.JsonStringify;
1212
import twg2.parser.codeParser.AccessModifier;
1313
import twg2.parser.codeParser.tools.NameUtil;
14-
import twg2.parser.fragment.CodeFragment;
14+
import twg2.parser.fragment.CodeToken;
1515
import twg2.parser.output.WriteSettings;
1616
import twg2.treeLike.simpleTree.SimpleTree;
1717

@@ -21,11 +21,11 @@
2121
*/
2222
@Immutable
2323
public class FieldDefResolved extends FieldSigResolved {
24-
private final @Getter SimpleTree<CodeFragment> initializer;
24+
private final @Getter SimpleTree<CodeToken> initializer;
2525

2626

2727
public FieldDefResolved(String name, List<String> fullName, TypeSigResolved fieldType, List<AccessModifier> accessModifiers,
28-
List<AnnotationSig> annotations, List<String> comments, SimpleTree<CodeFragment> initializer) {
28+
List<AnnotationSig> annotations, List<String> comments, SimpleTree<CodeToken> initializer) {
2929
super(name, fullName, fieldType, accessModifiers, annotations, comments);
3030
this.initializer = initializer;
3131
}

‎src/twg2/parser/Inclusion.java

-11
This file was deleted.

‎src/twg2/parser/codeParser/AstExtractor.java

+4-4
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
import twg2.ast.interm.field.FieldSig;
1111
import twg2.ast.interm.method.MethodSig;
1212
import twg2.ast.interm.type.TypeSig;
13-
import twg2.parser.fragment.CodeFragment;
13+
import twg2.parser.fragment.CodeToken;
1414
import twg2.parser.stateMachine.AstParser;
1515
import twg2.treeLike.simpleTree.SimpleTree;
1616

@@ -40,12 +40,12 @@ public interface AstExtractor<T_BLOCK extends BlockType> {
4040
* @param astTree the AST tree containing blocks to extract
4141
* @param parentScope the current blockTree's parent node or null if the parent is null (only possible if blockTree is a child of a tree with a null root or astTree is the root and has no parent)
4242
*/
43-
public List<BlockAst<T_BLOCK>> extractBlocks(List<String> nameScope, SimpleTree<CodeFragment> astTree, BlockAst<T_BLOCK> parentScope);
43+
public List<BlockAst<T_BLOCK>> extractBlocks(List<String> nameScope, SimpleTree<CodeToken> astTree, BlockAst<T_BLOCK> parentScope);
4444

4545
/** Parses a simple AST tree
46-
* @param astTree the tree of basic {@link CodeFragment} tokens
46+
* @param astTree the tree of basic {@link CodeToken} tokens
4747
* @return a list of entries with simple AST tree blocks as keys and classes ({@link ClassAst} instances) as values containing the annotations, comments, fields, and methods found inside the AST tree
4848
*/
49-
public List<Entry<SimpleTree<CodeFragment>, ClassAst.SimpleImpl<T_BLOCK>>> extractClassFieldsAndMethodSignatures(SimpleTree<CodeFragment> astTree);
49+
public List<Entry<SimpleTree<CodeToken>, ClassAst.SimpleImpl<T_BLOCK>>> extractClassFieldsAndMethodSignatures(SimpleTree<CodeToken> astTree);
5050

5151
}

‎src/twg2/parser/codeParser/AstNodeConsumer.java

+5-5
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,11 @@
22

33
import java.util.List;
44

5-
import twg2.parser.fragment.DocumentFragmentText;
5+
import twg2.parser.fragment.TextFragmentRefToken;
66
import twg2.treeLike.IndexedSubtreeConsumer;
77
import twg2.treeLike.simpleTree.SimpleTree;
88

9-
/** Interface which adds additional parameters to {@link IndexedSubtreeConsumer} designed to consume a {@link SimpleTree} of {@link DocumentFragmentText}.
9+
/** Interface which adds additional parameters to {@link IndexedSubtreeConsumer} designed to consume a {@link SimpleTree} of {@link TextFragmentRefToken}.
1010
* New parameters are:
1111
* <ul>
1212
* <li>node - the current tree node</li>
@@ -34,8 +34,8 @@ public interface AstNodeConsumer<T> {
3434
* @param parentNode the parent tree node of the current node
3535
* @param parentNodeFrag the parent tree node's data
3636
*/
37-
public void accept(SimpleTree<DocumentFragmentText<T>> node, DocumentFragmentText<T> nodeFrag,
38-
int idx, int size, int depth, List<SimpleTree<DocumentFragmentText<T>>> siblings,
39-
SimpleTree<DocumentFragmentText<T>> parentNode, DocumentFragmentText<T> parentNodeFrag);
37+
public void accept(SimpleTree<TextFragmentRefToken<T>> node, TextFragmentRefToken<T> nodeFrag,
38+
int idx, int size, int depth, List<SimpleTree<TextFragmentRefToken<T>>> siblings,
39+
SimpleTree<TextFragmentRefToken<T>> parentNode, TextFragmentRefToken<T> parentNodeFrag);
4040

4141
}

‎src/twg2/parser/codeParser/AstNodePredicate.java

+5-5
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,11 @@
22

33
import java.util.List;
44

5-
import twg2.parser.fragment.DocumentFragmentText;
5+
import twg2.parser.fragment.TextFragmentRefToken;
66
import twg2.treeLike.IndexedSubtreeConsumer;
77
import twg2.treeLike.simpleTree.SimpleTree;
88

9-
/** Interface which adds additional parameters to {@link IndexedSubtreeConsumer} designed to consume a {@link SimpleTree} of {@link DocumentFragmentText}.
9+
/** Interface which adds additional parameters to {@link IndexedSubtreeConsumer} designed to consume a {@link SimpleTree} of {@link TextFragmentRefToken}.
1010
* New parameters are:
1111
* <ul>
1212
* <li>node - the current tree node</li>
@@ -34,8 +34,8 @@ public interface AstNodePredicate<T> {
3434
* @param parentNode the parent tree node of the current node
3535
* @param parentNodeFrag the parent tree node's data
3636
*/
37-
public boolean test(SimpleTree<DocumentFragmentText<T>> node, DocumentFragmentText<T> nodeFrag,
38-
int idx, int size, int depth, List<SimpleTree<DocumentFragmentText<T>>> siblings,
39-
SimpleTree<DocumentFragmentText<T>> parentNode, DocumentFragmentText<T> parentNodeFrag);
37+
public boolean test(SimpleTree<TextFragmentRefToken<T>> node, TextFragmentRefToken<T> nodeFrag,
38+
int idx, int size, int depth, List<SimpleTree<TextFragmentRefToken<T>>> siblings,
39+
SimpleTree<TextFragmentRefToken<T>> parentNode, TextFragmentRefToken<T> parentNodeFrag);
4040

4141
}

‎src/twg2/parser/codeParser/KeywordUtil.java

+8-8
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
package twg2.parser.codeParser;
22

3-
import twg2.parser.codeParser.tools.CodeFragmentEnumSubSet;
3+
import twg2.parser.codeParser.tools.CodeTokenEnumSubSet;
44

55
/**
66
* @author TeamworkGuy2
@@ -40,30 +40,30 @@ public interface KeywordUtil<T_KEYWORD extends AccessModifier> {
4040

4141
/** Checks for block identifying keywords (i.e. 'namespace', 'module', 'class', 'interface')
4242
*/
43-
public CodeFragmentEnumSubSet<T_KEYWORD> blockModifiers();
43+
public CodeTokenEnumSubSet<T_KEYWORD> blockModifiers();
4444

4545
/** Checks for class/interface block modifier keywords (i.e. 'abstract', 'static', 'final', 'sealed')
4646
*/
47-
public CodeFragmentEnumSubSet<T_KEYWORD> classModifiers();
47+
public CodeTokenEnumSubSet<T_KEYWORD> classModifiers();
4848

4949
/** Checks for field modifier keywords (i.e. 'volatile', 'readonly', 'static', 'private')
5050
*/
51-
public CodeFragmentEnumSubSet<T_KEYWORD> fieldModifiers();
51+
public CodeTokenEnumSubSet<T_KEYWORD> fieldModifiers();
5252

5353
/** Checks for method modifier keywords (i.e. 'synchronized', 'static', 'final', 'protected')
5454
*/
55-
public CodeFragmentEnumSubSet<T_KEYWORD> methodModifiers();
55+
public CodeTokenEnumSubSet<T_KEYWORD> methodModifiers();
5656

5757
/** Checks for method parameter modifier keywords (i.e. 'out' , 'final')
5858
*/
59-
public CodeFragmentEnumSubSet<T_KEYWORD> parameterModifiers();
59+
public CodeTokenEnumSubSet<T_KEYWORD> parameterModifiers();
6060

6161
/** Checks for operator keywords (i.e. 'As', 'Is', 'instanceof')
6262
*/
63-
public CodeFragmentEnumSubSet<T_KEYWORD> operators();
63+
public CodeTokenEnumSubSet<T_KEYWORD> operators();
6464

6565
/** Checks for type literal keywords (i.e. 'true', 'false', 'null')
6666
*/
67-
public CodeFragmentEnumSubSet<T_KEYWORD> typeLiterals();
67+
public CodeTokenEnumSubSet<T_KEYWORD> typeLiterals();
6868

6969
}

‎src/twg2/parser/codeParser/OperatorUtil.java

+9-9
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,27 @@
11
package twg2.parser.codeParser;
22

3-
import twg2.parser.codeParser.tools.CodeFragmentEnumSubSet;
3+
import twg2.parser.codeParser.tools.CodeTokenEnumSubSet;
44

55
/**
66
* @author TeamworkGuy2
77
* @since 2016-4-9
88
*/
99
public interface OperatorUtil<T_OP extends Operator> {
1010

11-
public CodeFragmentEnumSubSet<T_OP> arithmeticOperators();
11+
public CodeTokenEnumSubSet<T_OP> arithmeticOperators();
1212

13-
public CodeFragmentEnumSubSet<T_OP> assignmentOperators();
13+
public CodeTokenEnumSubSet<T_OP> assignmentOperators();
1414

15-
public CodeFragmentEnumSubSet<T_OP> bitwiseOperators();
15+
public CodeTokenEnumSubSet<T_OP> bitwiseOperators();
1616

17-
public CodeFragmentEnumSubSet<T_OP> concatOperators();
17+
public CodeTokenEnumSubSet<T_OP> concatOperators();
1818

19-
public CodeFragmentEnumSubSet<T_OP> conditionalOperators();
19+
public CodeTokenEnumSubSet<T_OP> conditionalOperators();
2020

21-
public CodeFragmentEnumSubSet<T_OP> equalityOperators();
21+
public CodeTokenEnumSubSet<T_OP> equalityOperators();
2222

23-
public CodeFragmentEnumSubSet<T_OP> typeCheckOperators();
23+
public CodeTokenEnumSubSet<T_OP> typeCheckOperators();
2424

25-
public CodeFragmentEnumSubSet<T_OP> unaryOperators();
25+
public CodeTokenEnumSubSet<T_OP> unaryOperators();
2626

2727
}

‎src/twg2/parser/codeParser/tools/performance/ParseTimes.java ‎src/twg2/parser/codeParser/analytics/ParseTimes.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
package twg2.parser.codeParser.tools.performance;
1+
package twg2.parser.codeParser.analytics;
22

33
import java.io.IOException;
44
import java.util.Map;

0 commit comments

Comments
 (0)