Skip to content
This repository was archived by the owner on Jun 4, 2024. It is now read-only.

Improve Filtering #397

Merged
merged 56 commits into from
Apr 16, 2019
Merged

Improve Filtering #397

merged 56 commits into from
Apr 16, 2019

Conversation

Marc-Andre-Rivet
Copy link
Contributor

@Marc-Andre-Rivet Marc-Andre-Rivet commented Mar 13, 2019

  • isolate lexemes from lexicons and grammar/syntax
  • support multiple lexicons (single column, multiple columns, full query)
  • allow lexicons termination and validation logic for lexemes based on previous lexeme(s) (essentially a deterministic finite state-machine with these state transitions -- copy of table below)

Basics

  • enable short filter criteria on column filters (e.g. <binary op> <expression>, <unary op>, <expression>)
  • update operand syntax to expect {field} instead of field or "field", [{}] can be escaped with {\\{\\\\}field}, see unit tests for details
  • additional unit tests on all lexicons to test syntax / edge cases
  • refactor multi column AST's transformation into and from single column AST
  • rework expression lexeme to be consistent with changes made to operand lexeme
  • increase lexicon's granularity (one lexeme per binary operation, etc.) -- will make handling the structured cases simpler
  • expose query as structure (lhs, rhs, pivot, block, etc.) on derived prop -- consumable programmatically

After this PR

  • contains operator
  • regex operator
  • type-aware/conditional lexemes (e.g. can't contains on number)
  • type-aware default operator (single column query)
  • support column types implicitly
  • support for contextual conditionals (e.g. if in style_**_conditional -- allow expressing column_type, row_id, etc. as a filter instead of a structure)
  • fix python unit tests -- the remaining py visual tests could (all?) probably be rewritten as Cypress / functionality tests
  • suport eq(...) and and(...) syntax

Items with x are valid, nest refers to the level of nesting of the query up to that point -- each block open (open parentheses) increases the nesting by 1, each block close reduces the nesting by 1.

image

@chriddyp chriddyp temporarily deployed to dash-table-review-pr-397 March 13, 2019 03:05 Inactive
- rework validation logic (wip)
@chriddyp chriddyp temporarily deployed to dash-table-review-pr-397 March 13, 2019 15:24 Inactive
@chriddyp chriddyp temporarily deployed to dash-table-review-pr-397 March 13, 2019 22:56 Inactive
@chriddyp chriddyp temporarily deployed to dash-table-review-pr-397 March 13, 2019 23:14 Inactive
- add basic multi column tests
@chriddyp chriddyp temporarily deployed to dash-table-review-pr-397 March 14, 2019 00:44 Inactive
@chriddyp chriddyp temporarily deployed to dash-table-review-pr-397 March 14, 2019 14:31 Inactive
@chriddyp chriddyp temporarily deployed to dash-table-review-pr-397 March 14, 2019 16:26 Inactive
- style filter cells correctly
@chriddyp chriddyp temporarily deployed to dash-table-review-pr-397 March 14, 2019 21:38 Inactive
@chriddyp chriddyp temporarily deployed to dash-table-review-pr-397 March 14, 2019 22:14 Inactive
@chriddyp chriddyp temporarily deployed to dash-table-review-pr-397 March 15, 2019 00:04 Inactive
@Marc-Andre-Rivet Marc-Andre-Rivet changed the title [WIP] Refactor AST [WIP] Improve Filtering Mar 15, 2019

export default class SyntaxTree {
private result: ISyntaxerResult;
protected lexerResult: ILexerResult;
protected syntaxerResult: ISyntaxerResult;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping the intermediary results allows for finer manipulations in the derived syntax tree classes

@Marc-Andre-Rivet
Copy link
Contributor Author

Marc-Andre-Rivet commented Apr 12, 2019

I had put a stopgag on the selenium tests a while back, after removing it Python/Selenium tests are failing. Not sure why, just as before. Investigating.

@Marc-Andre-Rivet
Copy link
Contributor Author

Marc-Andre-Rivet commented Apr 16, 2019

@alexcjohnson: Seems like the python visual tests got broken with the move to react 16 -- since the table was only built / never in a real PR, I never realized there were visual differences -- same issue discussed last week. The latest commits rewrite all the Python tests that can be written as storybook tests. The storybook tests have shown themselves very resilient to change / to be more stable. In any case, some tests were clearly already broken, like the dropdown ones that did not use presentation: dropdown. The noise factor and general negligence on my part due to the noise was at play.

Some Python tests cannot be rewritten this way as they involve callbacks between the table and the graph. Technically these are not really visual tests, instead the visual result is an artefact indicating the dash app behaves correctly.. I hate those but there's no easy replacement for now. These seem to render just fine but the style is not being picked up when snapshot for whatever reason -- somehow seems related to the React 15 -> 16 update but that's no more than a hypothesis.

Still have to have look into \r, \n \t and the likes in the queries, fix issues to be found and add tests.

- add `contains` relational operator to all lexicons
- add `contains` test
},
type: LexemeType.LogicalOperator,
priority: 2,
regexp: /^(and\s|&&)/i,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably want to allow ( immediately after, like (1<2)and(3<4) - that works in Python anyway, will confuse ppl if it fails. Same for or.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is allowed in full query form -- e.g. {a} eq "1" and ({a} eq "0" or ({b} eq "1" or {b} eq "0"))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah! You mean the space?

),
subType: RelationalOperator.Contains,
regexp: /^(contains)/i
}, LEXEME_BASE);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added contains relational operator -- works only for strings -- to eventually be allowed on string columns/data

Copy link
Collaborator

@alexcjohnson alexcjohnson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beautiful. Time to get this out in the world! 💃

@Marc-Andre-Rivet Marc-Andre-Rivet changed the title [WIP] Improve Filtering Improve Filtering Apr 16, 2019
@Marc-Andre-Rivet
Copy link
Contributor Author

Marc-Andre-Rivet commented Apr 16, 2019

@alexcjohnson Adding a changelog entry. This might also be the last good time to rename filtering_settings. Leaning towards filter to stay consistent with other existing props and renaming derived_query_structure to derived_filter_structure. Is there a plotly.js precedent that we may want to follow?

@alexcjohnson
Copy link
Collaborator

👍 for filter. We're already using that name in conditional styles & tooltips, and even though the usage is a little different (can be simpler) it's similar enough that keeping the same name makes sense.

- rename filtering_settings to filter
@Marc-Andre-Rivet Marc-Andre-Rivet merged commit 989d8cd into master Apr 16, 2019
@Marc-Andre-Rivet Marc-Andre-Rivet deleted the refactor-syntax-tree branch April 16, 2019 20:46
@alexcjohnson alexcjohnson mentioned this pull request Apr 30, 2019
4 tasks
@chriddyp chriddyp mentioned this pull request Jun 4, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants