Implement parsing to a list of tokens

[reid-spencer]

Added to TopLevelParser:

• def parseToTokens(input:RiddlParseInput): Either[Messages,List[Token]]

Added these case classes to AST as the set of possible results
returned from TopLevelParser.parseToTokens:

• case class PunctuationTKN(at: At) extends Token
• case class QuotedStringTKN(at: At) extends Token
• case class ReadabilityTKN(at: At) extends Token
• case class PredefinedTKN(at: At) extends Token
• case class KeywordTKN(at: At) extends Token
• case class CommentTKN(at: At) extends Token
• case class LiteralStringTKN(at: At) extends Token
• case class MarkdownLinesTKN(at: At) extends Token
• case class IdentifierTKN(at: At) extends Token
• case class OtherTKN(at: At) extends Token

This allows for distinguishing the basic kinds of input being parsed.

[sonarqubecloud]

Please retry analysis of this Pull-Request directly on SonarQube Cloud

[coveralls]

Pull Request Test Coverage Report for Build 11990233828

Details

• 72 of 98 (73.47%) changed or added relevant lines in 9 files are covered.
• 84 unchanged lines in 24 files lost coverage.
• Overall coverage decreased (-0.003%) to 73.429%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
language/shared/src/main/scala/com/ossuminc/riddl/language/parsing/ParsingContext.scala	12	14	85.71%
language/shared/src/main/scala/com/ossuminc/riddl/language/parsing/PredefTypes.scala	5	7	71.43%
language/shared/src/main/scala/com/ossuminc/riddl/language/parsing/Punctuation.scala	1	3	33.33%
language/shared/src/main/scala/com/ossuminc/riddl/language/parsing/Keywords.scala	1	4	25.0%
language/shared/src/main/scala/com/ossuminc/riddl/language/parsing/ExtensibleTopLevelParser.scala	19	24	79.17%
language/shared/src/main/scala/com/ossuminc/riddl/language/parsing/TokenStreamParser.scala	26	38	68.42%

Files with Coverage Reduction	New Missed Lines	%
language/shared/src/main/scala/com/ossuminc/riddl/language/parsing/ContextParser.scala	1	81.25%
language/shared/src/main/scala/com/ossuminc/riddl/language/parsing/ModuleParser.scala	1	90.91%
language/shared/src/main/scala/com/ossuminc/riddl/language/parsing/ExtensibleTopLevelParser.scala	1	53.33%
language/shared/src/main/scala/com/ossuminc/riddl/language/parsing/EntityParser.scala	1	89.47%
language/shared/src/main/scala/com/ossuminc/riddl/language/parsing/NebulaParser.scala	1	77.78%
language/shared/src/main/scala/com/ossuminc/riddl/language/parsing/ParsingErrors.scala	1	87.88%
language/shared/src/main/scala/com/ossuminc/riddl/language/parsing/RootParser.scala	1	77.78%
language/shared/src/main/scala/com/ossuminc/riddl/language/parsing/FunctionParser.scala	2	82.35%
language/shared/src/main/scala/com/ossuminc/riddl/language/parsing/StreamingParser.scala	2	90.16%
language/shared/src/main/scala/com/ossuminc/riddl/language/parsing/Keywords.scala	2	67.72%

Totals
Change from base Build 11987576559:	-0.003%
Covered Lines:	4977
Relevant Lines:	6778

---

💛 - Coveralls