Improve the docs #254

deltaidea · 2017-07-30T07:54:30Z

Preview of the progress so far

This is a joint effort to make the docs more straight-forward and complete.

I hijacked tjvr:ihatewriting branch and started this PR based on it. @tjvr can still push to this PR since he's a collaborator, and I didn't want to deal with plain-text conflicts.

Relevant:

Things to do (in a vaguely sensible order):

nearley can run in browsers too.

Run `doctoc`. The ToC got outdated.

Use consistent style: `#` headers, no extra newlines, `-` lists. I hope this is the rigth style, this is how the majority of the file looked like.

deltaidea · 2017-07-30T15:13:01Z

I apologize in advance, this will require at least one of @Hardmath123 and @tjvr to go through each commit and review the changes. Things to look for:

terminology (I'm not sure I got the difference between "rule" and "nonterminal" right),
style (are explanations too technical or too vague, should code samples be formatted differently)
typos and spelling (English is not my first language)

The whole PR can be squashed before merging, so expect small focused commits and force pushes (I'll tell if there're any sneaky changes).

deltaidea · 2017-07-30T15:28:55Z

Notes so far:

I used ini syntax for fenced code blocks with comments. INI also has comments starting with # and it highlights everything else in a sane way. I tried python and coffeescript and they mark -> as invalid or highlight stuff inconsistently.
I used ES6 import in examples. Do you want me to drop it and go back to require?

deltaidea · 2017-08-03T10:04:15Z

@Hardmath123
Using moo is recommended and obviously easier than writing a custom lexer. What do you think about moving custom tokenizing stuff to tokens.md and only showing basics with moo in the readme?

This would streamline the docs, make them easier to digest all at once and not be as intimidating.

tjvr · 2017-08-03T10:33:52Z

Thank you for working on this! :D This is definitely something that needs doing, and we keep not getting round to.

We'll definitely need to review this carefully, as you said. I suspect @Hardmath123 will have strong opinions :-)

What do you think about moving custom tokenizing stuff to tokens.md and only showing basics with moo in the readme?

I'm in favour.

These two topics are covered in the section "Using a parser", which doesn't make much sense. The basic usage is already at the top of readme, so let's remove "Using a parser" completely.

deltaidea · 2017-08-03T15:16:23Z

16fbd03 adds a snippet for compiling grammars on client-side. Do you think it's a good idea to add this feature directly to nearley instead? In a separate PR, later.

kach · 2017-08-03T18:01:13Z

I suspect @Hardmath123 will have strong opinions :-)

…not sure how I should feel that you said that; especially since you're probably right…

What do you think about moving custom tokenizing stuff to tokens.md and only showing basics with moo in the readme?

I propose something more radical: currently, the README is a behemoth of a tutorial/documentation/README crossover, while the webpage is what convinces you to use nearley. I think they should be swapped. The README should contain most of the contents of index.html (and some other stuff). /www should contain the docs, split up into chunks that make sense:

Tutorial
Using lexers
nearley on the front-end
Performance tips
etc.

There's probably a way to coax github into rendering those markdown documents into a nice self-contained webpage, perhaps with Jekyll or something.

kach · 2017-08-03T18:04:24Z

I used ES6 import in examples. Do you want me to drop it and go back to require?

Yes, please. We can add a note saying the import statement can also be used, but let's stick with the style nearley was written in for documentation purposes.

613ad00 readme: make all example code more readable, update to ES6

See above. Shouldn't be hard to revert.

kach · 2017-08-03T18:05:54Z

16fbd03 adds a snippet for compiling grammars on client-side. Do you think it's a good idea to add this feature directly to nearley instead? In a separate PR, later.

What is the use-case? I can't think of anything besides the "try nearley" page.

tjvr · 2017-08-04T11:17:45Z

Do you think it's a good idea to add this feature directly to nearley instead?

At some point we should probably rework the Nearley API, but that's certainly a v3 thing.

deltaidea · 2017-08-04T12:15:22Z

README is a behemoth of a tutorial/documentation/README crossover, while the webpage is what convinces you to use nearley. I think they should be swapped.

That's a great thought! Still, this PR is already a beast, and not in a good sense, so let's cut the scope, finish what's originally in the top comment and just merge it already. This PR is not my smartest idea.

tjvr · 2017-08-04T13:18:12Z

README.md

 ```

-Alternatively, to use a generated grammar in a browser runtime, include the
-`nearley.js` file in a `<script>` tag.
+Add a script to `scripts` in `package.json` that runs the command above if you only have a locally installed copy of nearley.


tjvr

This is a really substantial piece of work! :D Thanks so much @deltaidea for taking this on. I've had a read through your changes to the README and made some (mostly minor) comments. Thanks again!

tjvr · 2017-08-04T13:18:55Z

README.md

-  random strings that match your grammar.
- `nearley-railroad` generates pretty railroad diagrams from your parser. This
-  is mainly helpful for creating documentation, as (for example) on json.org.
+See [Using in frontend](docs/using-in-frontend.md) for instructions on how to use nearley in your browser code.

 You can uninstall the nearley compiler using `npm uninstall -g nearley`.


Let's drop this line.

tjvr · 2017-08-04T13:19:57Z

README.md

-  random strings that match your grammar.
- `nearley-railroad` generates pretty railroad diagrams from your parser. This
-  is mainly helpful for creating documentation, as (for example) on json.org.
+See [Using in frontend](docs/using-in-frontend.md) for instructions on how to use nearley in your browser code.


Good link! But let's move this sentence to the bottom of usage, and make clear that this is an unusual thing to do.

Well, to clarify, using a generated parser and nearley.js is a totally reasonable and common thing to do in the browser. Instructions on doing this should be prominent.

Using the nearley compiler in the browser is whacky and shouldn't be mentioned here.

tjvr · 2017-08-04T13:20:32Z

README.md

+classes at universities, as well as [file format parsers](https://github.com/raymond-h/node-dmi),
+[markup languages](https://github.com/bobbybee/uPresent) and
+[complete programming languages](https://github.com/bobbybee/carbon).
+It's an npm [staff pick](https://www.npmjs.com/package/npm-collection-staff-picks).


@Hardmath123 I think this should be a list. Looks better that way. :-)

It also should mention a better set of projects. :-)

In particular, I think the markup language should be Idyll and the complete language should be https://github.com/sizigi/lp5562 or something.

tjvr · 2017-08-04T16:36:23Z

README.md

+
+- A *terminal* is a string or a token. E.g. keyword `"if"` is a terminal.
+- A *nonterminal* is a combination of terminals and other nonterminals. E.g. an if statement defined as `"if" condition statement` is a nonteminal.
+- A *rule* (or production rule) is a definition of a nonterminal. E.g. `"if" condition statement` is the rule according to which the if statement nonterminal is parsed.


@Hardmath123 Explaining what a CFG rule looks like is hard, but I think it's worth making sure we get this right.

Embarrassing story: I started a glossary a while back, and accidentally committed it at some point.

https://github.com/Hardmath123/nearley/blob/master/glossary.md

Some things in it might be wrong, so we should probably remove it at some point. But, it's a good starting-point if we want to, as Tim says, "make sure we get this right".

tjvr · 2017-08-04T16:58:06Z

README.md

-want to give fancy error messages for runtime errors.
+- `data: Array` - array with the parsed parts of the rule. It will always be an array, even if there's only one part. If a rule contains nonterminals with their own postprocessors, the respective parts will already be transformed.
+- `location: number` - the index (zero-based) at which the rule match starts. It's useful to retain this information in the syntax tree if you're writing an interpreter.
+- `reject: Object` - return this object to signal that this rule doesn't actually match. This allows you to restrict or conditionally support language features.


Should probably mention this here. (cf. #258 (comment))

Also, we should probably warn people to avoid reject if possible. @Hardmath123

Also, "restrict or conditionally support language features" isn't the use-case here. The use-case is edge conditions like "I want [a-z]+ to match variables, EXCEPT for the keyword if…"

Many (but not all) of these get fixed by using a lexer.

tjvr · 2017-08-07T23:26:46Z

README.md

-```js
-var grammar = require("generated-code.js");
-var nearley = require("nearley");
+You can still use raw strings, but they will only match full tokens parsed by Moo:


Aside: this is (or was) actually a misfeature, but, hey, this isn't the place to discuss that. (Using raw strings is really convenient, but it interacted badly with moo's support for capture groups. Fortunately, we've removed those from Moo! :-) )

...except we added back value transforms. I should really fix that...

tjvr · 2017-08-07T23:27:31Z

README.md

-The `nearley.Parser` constructor takes an optional third parameter, `options`,
-which is an object with the following possible keys:
+- [Best practices for writing grammars](docs/how-to-grammar-good.md)
+- [Custom tokens and lexers, parsing arbitrary arrays instead of strings](docs/custom-tokens-and-lexers.md)


Are we missing the feed(array) bit in the linked file? @deltaidea

Whoops, no, we just never documented that properly in the first place.

tjvr · 2017-08-07T23:27:36Z

README.md

-  very large), so we recommend leaving this as `false` unless you are familiar
-  with the Earley parsing algorithm and are planning to do something exciting
-  with the parse table.
+## Recipies


tjvr · 2017-08-07T23:29:13Z

README.md

@@ -410,27 +407,22 @@ parser.

 This was previously called `bin/nearleythere.js` and written by Robin.


@Hardmath123 Can we drop this sentence now? :-)

Yeah, I guess if you want.

tjvr · 2017-08-07T23:30:25Z

README.md

@@ -496,19 +484,16 @@ Webpack users can use
 [nearley-loader](https://github.com/kozily/nearley-loader) by Andrés Arana to
 load grammars directly.

+## Still confused?


This section should probably move up nearer to Recipes. @deltaidea @Hardmath123

kach · 2017-08-08T00:02:24Z

README.md

-written in *itself* (this is called bootstrapping).
+## Introduction
+
+nearley compiles grammar definitions from a simple syntax resembling [BNF](https://en.wikipedia.org/wiki/Backus–Naur_form) to a JS representation.


Introduction should also describe why people like nearley: it's user-friendly, easy to get started, works on both the browser and node, etc.

I can take care of this change later though, don't worry about it.

kach · 2017-08-08T00:03:22Z

README.md

+nearley compiles grammar definitions from a simple syntax resembling [BNF](https://en.wikipedia.org/wiki/Backus–Naur_form) to a JS representation.
+You pass that representation to the nearley's tiny runtime, feed it data, and get the results.
+
+nearley uses the Earley parsing algorithm with Joop Leo's optimizations to parse complex data structures easily.


I think "parse complex data structures" is not quite what we wanted to say, even in the original README. This is a good opportunity to find better wording. Complex languages, perhaps?

kach · 2017-08-08T00:06:21Z

README.md


-To use nearley, you need both a *global* and a *local* installation. The two
-types of installations are described separately below.
+nearley is published as an [NPM](https://docs.npmjs.com/getting-started/what-is-npm) package compatible with [Node.js](https://nodejs.org/en/) and browsers.


*Most browsers! Even some old ones. :-)

kach · 2017-08-08T00:09:00Z

README.md

-> var nearley = require("nearley");
-> var grammar = require("./my-generated-grammar.js");
+
+Check out the wonderful [nearley playground](https://omrelli.ug/nearley-playground/) to explore nearley interactively in your browser.


Can this be moved to a more prominent location? "NOTE: You can follow along by using the nearley playground, an online interface for exploring nearley grammars in your browser."

kach · 2017-08-08T00:09:43Z

README.md

 ```

+See below for detailed API and grammar syntax specification.


Let's replace "below" with internal links to sections.

kach · 2017-08-08T00:13:36Z

README.md


-will match 0 or more `cow`s in a row.
+If you would like to support a different language, feel free to file a PR!

 ### Charsets

 You can use valid RegExp charsets in a rule:


*in scannerless mode

kach · 2017-08-08T00:17:20Z

docs/accessing-parse-table.md

+- `keepHistory` (boolean, default `false`) - whether to preserve and expose the internal state
+- `lexer` (object) - custom lexer, overrides `@lexer` in the grammar
+
+If you are familiar with the Earley parsing algorithm and are planning to do something exciting with the parse table, set `keepHistory`:


Wasn't the initial motivation for keepHistory rewinding? @tjvr

Yes, but we now have save() for that purpose.

TODO: document save/restore.

kach · 2017-08-08T00:20:03Z

docs/using-in-frontend.md

@@ -0,0 +1,37 @@
+# Using nearley in browsers
+
+Use a tool like [Webpack](https://webpack.js.org/) or [Rollup](https://rollupjs.org/) to include the `nearley` NPM package in your browser code.


Wait, doesn't nearley work fine if you just, y'know, include nearley.js?

Let's be very careful making the distinction between parsing with nearley and compiling with nearleyc. This file should be called compiling nearley grammars in browsers or something!

Yeah, I don’t see a need to mention bundlers here.

Wait, I thought we were still in README! This is fine as-is I think. (Ignoring the fact that I personally prefer browserify :-) )

kach · 2017-08-08T00:20:35Z

I made a rough pass too.

deltaidea · 2017-08-08T04:40:25Z

Thanks for the insight, guys! This is really helpful for my understanding. I'm very busy at the moment. Maybe I'll have some time on the weekend. Feel free to fix stuff on your own if you like.

tjvr · 2017-08-08T11:49:16Z

@tjvr can still push to this PR since he's a collaborator

I'm confused what you mean by this. Should I merge your PR to ihatewriting and then tweak it, or what? :-)

deltaidea · 2017-08-08T12:19:25Z

I mean both of you can just checkout this PR and push to it directly since the work is slow and there're no conflicts.

git fetch origin pull/254/head:improve-docs
git checkout improve-docs

this is a test commit. Can I commit to a PR? That seems odd.

:-)

tjvr · 2017-08-08T12:55:37Z

@deltaidea Oh, weird! I’m confused about why I have push access to your fork, but there we go. :-)

I fixed a bunch of my own review comments. @Hardmath123 wanted to rewrite some bits too, I think.

I agree we shouldn’t keep this open too much longer; this is already much improved from what we have! @deltaidea, at some point please could you:

have a look at the few comments left above
switch import -> require, but leave the code as ES6, I think that’s fine.

For now, leaving Tokenizers as part of the main README is fine. Leaving the documentation as markdown files in docs/ is also fine; @Hardmath123 is welcome to webify things later. :-)

deltaidea · 2017-08-08T14:53:03Z

Thanks for the feedback and help!

Isn't require already used everywhere?

Regarding everything else, I'll have a proper look in an hour.

deltaidea · 2017-08-08T14:55:57Z

Oh, weird! I’m confused about why I have push access to your fork, but there we go. :-)

GitHub has a checkbox called "Allow edits from maintainers" under every PR. It opens write access to the branch.

tjvr · 2017-08-08T17:30:42Z

Allow edits from maintainers

Ah, thanks! Never seen that before.

Looks like you have already replaced import with require -- thanks! :-)

@deltaidea

@deltaidea's mega-PR. A squash of these commits: Add some lexer stuff to README readme: remove trailing spaces readme: change the description to "Simple parsing in JavaScript" nearley can run in browsers too. readme: update the table of contents Run `doctoc`. The ToC got outdated. readme: make Markdown style consistent Use consistent style: `#` headers, no extra newlines, `-` lists. I hope this is the rigth style, this is how the majority of the file looked like. readme: add an intro and usage guide for beginners readme: reorder and reword parser specification readme: make all example code more readable, update to ES6 docs: move `glossary.md` and `how-to-grammar-good.md` to `docs/` docs: move making REPL and accessing parse table into their own files These two topics are covered in the section "Using a parser", which doesn't make much sense. The basic usage is already at the top of readme, so let's remove "Using a parser" completely. docs: add a clear Moo example to readme, separate out custom lexers docs: add a link in readme to "How to grammar good" docs: add a complete example of usage in browsers docs: use `require()` instead of `import` readme: link to the indentation-aware lexer by nathan docs: add an example of generating CST and AST readme: move recipies to section "Recipies" Make clear using `nearleyc` in a browser is unusual. this is a test commit. Can I commit to a PR? That seems odd. Fix a bunch of my review comments :-) adds link to nearley-gulp in README Add doctoc script to package.json

tjvr · 2017-08-09T10:10:15Z

This PR was getting scary, so I went ahead and merged it! :D

Thanks so much for your help @deltaidea.

I'm gonna open a couple new issues for the rest--we can continue work in smaller PRs.

danielo515 · 2018-07-08T16:13:31Z

Where is the context aware example?

tjvr and others added 6 commits July 30, 2017 12:18

Add some lexer stuff to README

7f8ea92

readme: remove trailing spaces

a5ac4ea

readme: change the description to "Simple parsing in JavaScript"

70d27f0

nearley can run in browsers too.

readme: update the table of contents

f21b305

Run `doctoc`. The ToC got outdated.

readme: make Markdown style consistent

6e7703f

Use consistent style: `#` headers, no extra newlines, `-` lists. I hope this is the rigth style, this is how the majority of the file looked like.

readme: add an intro and usage guide for beginners

73738cb

readme: reorder and reword parser specification

e3f200a

deltaidea added 6 commits August 3, 2017 18:08

readme: make all example code more readable, update to ES6

613ad00

docs: move glossary.md and how-to-grammar-good.md to docs/

e165ee1

docs: move making REPL and accessing parse table into their own files

d0775fc

These two topics are covered in the section "Using a parser", which doesn't make much sense. The basic usage is already at the top of readme, so let's remove "Using a parser" completely.

docs: add a clear Moo example to readme, separate out custom lexers

2cb509f

docs: add a link in readme to "How to grammar good"

5f2f724

docs: add a complete example of usage in browsers

16fbd03

deltaidea added 4 commits August 4, 2017 10:48

docs: use require() instead of import

6c987f5

readme: link to the indentation-aware lexer by nathan

5d963d4

docs: add an example of generating CST and AST

51fef61

readme: move recipies to section "Recipies"

d0d7564

deltaidea changed the title ~~Improve docs~~ [WIP] Improve the docs Aug 4, 2017

tjvr reviewed Aug 4, 2017

View reviewed changes

deltaidea mentioned this pull request Aug 4, 2017

Add ES2015 (ES6) generator #259

Closed

tjvr mentioned this pull request Aug 7, 2017

Document lexers and fromCompiled #248

Closed

tjvr approved these changes Aug 7, 2017

View reviewed changes

kach reviewed Aug 8, 2017

View reviewed changes

README.md Outdated

```

See below for detailed API and grammar syntax specification.

Copy link

Owner

kach Aug 8, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's replace "below" with internal links to sections.

kach reviewed Aug 8, 2017

View reviewed changes

tjvr added 2 commits August 8, 2017 13:25

Make clear using nearleyc in a browser is unusual.

c680d4e

this is a test commit. Can I commit to a PR? That seems odd.

Fix a bunch of my review comments

ac30729

:-)

tjvr added 2 commits August 9, 2017 10:46

Merge branch 'master' into improve-docs

7655bb5

Add doctoc script to package.json

66f3406

tjvr merged commit d479748 into kach:master Aug 9, 2017

This was referenced Aug 9, 2017

Further Doc improvements #261

Open

Consider switching docs with website #262

Closed

deltaidea changed the title ~~[WIP] Improve the docs~~ Improve the docs Aug 9, 2017

deltaidea deleted the improve-docs branch August 9, 2017 14:26

		@@ -410,27 +407,22 @@ parser.

		This was previously called `bin/nearleythere.js` and written by Robin.

		```

		See below for detailed API and grammar syntax specification.

		@@ -0,0 +1,37 @@
		# Using nearley in browsers

		Use a tool like [Webpack](https://webpack.js.org/) or [Rollup](https://rollupjs.org/) to include the `nearley` NPM package in your browser code.

Improve the docs #254

Improve the docs #254

Conversation

deltaidea commented Jul 30, 2017 • edited by tjvr Loading

deltaidea commented Jul 30, 2017 • edited Loading

deltaidea commented Jul 30, 2017 • edited Loading

deltaidea commented Aug 3, 2017

tjvr commented Aug 3, 2017

deltaidea commented Aug 3, 2017

kach commented Aug 3, 2017

kach commented Aug 3, 2017

kach commented Aug 3, 2017

tjvr commented Aug 4, 2017

deltaidea commented Aug 4, 2017

Choose a reason for hiding this comment

tjvr left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tjvr Aug 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tjvr Aug 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kach commented Aug 8, 2017

deltaidea commented Aug 8, 2017

tjvr commented Aug 8, 2017

deltaidea commented Aug 8, 2017

tjvr commented Aug 8, 2017 • edited Loading

deltaidea commented Aug 8, 2017

deltaidea commented Aug 8, 2017

tjvr commented Aug 8, 2017

tjvr commented Aug 9, 2017

danielo515 commented Jul 8, 2018

deltaidea commented Jul 30, 2017 •

edited by tjvr

Loading

deltaidea commented Jul 30, 2017 •

edited

Loading

deltaidea commented Jul 30, 2017 •

edited

Loading

tjvr left a comment •

edited

Loading

tjvr Aug 7, 2017 •

edited

Loading

tjvr Aug 7, 2017 •

edited

Loading

tjvr commented Aug 8, 2017 •

edited

Loading