Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

475 Pattern Matching Callbacks (codenamed wildcards) #1103

Merged
merged 88 commits into from
Apr 7, 2020
Merged

Conversation

alexcjohnson
Copy link
Collaborator

@alexcjohnson alexcjohnson commented Jan 30, 2020

Pattern Matching Callbacks! (Codenamed, Wildcards!)

➡️ See official documentation here: https://dash.plotly.com/pattern-matching-callbacks

Closes #475 - using the API discussed there (I'll post some examples soon, but for now here's the description):

  • String IDs still work the same way in both layout and callbacks
  • Now you can also make dict IDs. Keys should be strings, values can be strings, numbers, booleans, like:
    • id={"type": "cat", "age": 4, "purrs": True}
    • IDs must still be unique
  • Callbacks can use dict IDs with wildcard values for any of the keys. The supported wildcards are MATCH, ALL, and ALLSMALLER - and can be imported from dash.dependencies. There are various rules about how these can combine, driven by the need to resolve these to a unique callback invocation for each set of outputs and identify uniquely the inputs required for this invocation...

To do before this is ready:

Contributor Checklist

  • I have run the tests locally and they passed. (refer to testing section in contributing)
  • I have added tests, or extended existing tests, to cover any new features or bugs fixed in this PR
  • I have added entry in the CHANGELOG.md
  • If this PR needs a follow-up in dash docs, community thread, I have mentioned the relevant URLS as follow
    • this github #PR number updates the dash docs
    • here is the show and tell thread in plotly dash community

Closes #832 (among other callback bugs - the others all have commits that close them with matching tests, but #832 is a bit heavy for a test of its own)
Closes #1146

@Marc-Andre-Rivet Marc-Andre-Rivet added this to the Dash v1.10 milestone Jan 30, 2020
@christianwengert
Copy link

Dear @alexcjohnson
This looks promising!
If you deem it useful, I might start some tests for my usecases next week.

@alexcjohnson
Copy link
Collaborator Author

@christianwengert Absolutely! Thank you for your patience with this. I will try to post some usage examples here later today.

@christianwengert
Copy link

christianwengert commented Jan 31, 2020 via email

@alexcjohnson
Copy link
Collaborator Author

alexcjohnson commented Feb 1, 2020

To-Do list example

The edit_list callback creates pairs of elements: a checkbox for marking items done and a div displaying the item text. Both of these have dict IDs, with matching item values, but the checkbox is distinguished by the "action": "done" entry. The callbacks then refer to these items with "item": ALL or "item": MATCH in their input/output/state wildcard component IDs.

import dash
import dash_core_components as dcc
import dash_html_components as html
from dash.dependencies import Input, Output, State, MATCH, ALL

app = dash.Dash(__name__)

app.layout = html.Div([
    html.Div('Dash To-Do list'),
    dcc.Input(id="new-item"),
    html.Button("Add", id="add"),
    html.Button("Clear Done", id="clear-done"),
    html.Div(id="list-container"),
    html.Hr(),
    html.Div(id="totals")
])

style_todo = {"display": "inline", "margin": "10px"}
style_done = {"textDecoration": "line-through", "color": "#888"}
style_done.update(style_todo)


@app.callback(
    [
        Output("list-container", "children"),
        Output("new-item", "value")
    ],
    [
        Input("add", "n_clicks"),
        Input("new-item", "n_submit"),
        Input("clear-done", "n_clicks")
    ],
    [
        State("new-item", "value"),
        State({"item": ALL}, "children"),
        State({"item": ALL, "action": "done"}, "value")
    ]
)
def edit_list(add, add2, clear, new_item, items, items_done):
    triggered = [t["prop_id"] for t in dash.callback_context.triggered]
    adding = len([1 for i in triggered if i in ("add.n_clicks", "new-item.n_submit")])
    clearing = len([1 for i in triggered if i == "clear-done.n_clicks"])
    new_spec = [
        (text, done) for text, done in zip(items, items_done)
        if not (clearing and done)
    ]
    if adding:
        new_spec.append((new_item, []))
    new_list = [
        html.Div([
            dcc.Checklist(
                id={"item": i, "action": "done"},
                options=[{"label": "", "value": "done"}],
                value=done,
                style={"display": "inline"}
            ),
            html.Div(text, id={"item": i}, style=style_done if done else style_todo)
        ], style={"clear": "both"})
        for i, (text, done) in enumerate(new_spec)
    ]
    return [new_list, "" if adding else new_item]


@app.callback(
    Output({"item": MATCH}, "style"),
    [Input({"item": MATCH, "action": "done"}, "value")]
)
def mark_done(done):
    return style_done if done else style_todo


@app.callback(
    Output("totals", "children"),
    [Input({"item": ALL, "action": "done"}, "value")]
)
def show_totals(done):
    count_all = len(done)
    count_done = len([d for d in done if d])
    result = "{} of {} items completed".format(count_done, count_all)
    if count_all:
        result += " - {}%".format(int(100 * count_done / count_all))
    return result


if __name__ == "__main__":
    app.run_server(debug=True)

@alexcjohnson
Copy link
Collaborator Author

I noticed a problem with the To-Do app: if you delete all the items, the totals callback will not fire to say "0 of 0 items completed" as it should. This is because we currently only fire callbacks based on inputs that exist when they're fired, but for ALL (and ALLSMALLER) an input being removed is itself a change to the input. In fact this could happen when removing just some of the items, but in this particular case the callback still fires because we replace all the other list items (having not yet implemented #968 array operation callbacks).

Working on a fix for this...

@alexcjohnson
Copy link
Collaborator Author

alexcjohnson commented Feb 2, 2020

Extended To-Do app using ALLSMALLER to link up the list items in sequence:
Screen Shot 2020-02-01 at 10 48 37 PM

ALLSMALLER can only be used in Input and State items, and must be used on a key that has MATCH in the Output item(s).

import dash
import dash_core_components as dcc
import dash_html_components as html
from dash.dependencies import Input, Output, State, MATCH, ALL, ALLSMALLER

app = dash.Dash(__name__)

app.layout = html.Div([
    html.Div('Dash To-Do list'),
    dcc.Input(id="new-item"),
    html.Button("Add", id="add"),
    html.Button("Clear Done", id="clear-done"),
    html.Div(id="list-container"),
    html.Hr(),
    html.Div(id="totals")
])

style_todo = {"display": "inline", "margin": "10px"}
style_done = {"textDecoration": "line-through", "color": "#888"}
style_done.update(style_todo)


@app.callback(
    [
        Output("list-container", "children"),
        Output("new-item", "value")
    ],
    [
        Input("add", "n_clicks"),
        Input("new-item", "n_submit"),
        Input("clear-done", "n_clicks")
    ],
    [
        State("new-item", "value"),
        State({"item": ALL}, "children"),
        State({"item": ALL, "action": "done"}, "value")
    ]
)
def edit_list(add, add2, clear, new_item, items, items_done):
    triggered = [t["prop_id"] for t in dash.callback_context.triggered]
    adding = len([1 for i in triggered if i in ("add.n_clicks", "new-item.n_submit")])
    clearing = len([1 for i in triggered if i == "clear-done.n_clicks"])
    new_spec = [
        (text, done) for text, done in zip(items, items_done)
        if not (clearing and done)
    ]
    if adding:
        new_spec.append((new_item, []))
    new_list = [
        html.Div([
            dcc.Checklist(
                id={"item": i, "action": "done"},
                options=[{"label": "", "value": "done"}],
                value=done,
                style={"display": "inline"}
            ),
            html.Div(text, id={"item": i}, style=style_done if done else style_todo),
            html.Div(id={"item": i, "preceding": True}, style=style_todo)
        ], style={"clear": "both"})
        for i, (text, done) in enumerate(new_spec)
    ]
    return [new_list, "" if adding else new_item]


@app.callback(
    Output({"item": MATCH}, "style"),
    [Input({"item": MATCH, "action": "done"}, "value")]
)
def mark_done(done):
    return style_done if done else style_todo


@app.callback(
    Output({"item": MATCH, "preceding": True}, "children"),
    [
        Input({"item": ALLSMALLER, "action": "done"}, "value"),
        Input({"item": MATCH, "action": "done"}, "value")
    ]
)
def show_preceding(done_before, this_done):
    if this_done:
        return ""
    all_before = len(done_before)
    done_before = len([1 for d in done_before if d])
    out = "{} of {} preceding items are done".format(done_before, all_before)
    if all_before == done_before:
        out += " DO THIS NEXT!"
    return out


@app.callback(
    Output("totals", "children"),
    [Input({"item": ALL, "action": "done"}, "value")]
)
def show_totals(done):
    count_all = len(done)
    count_done = len([d for d in done if d])
    result = "{} of {} items completed".format(count_done, count_all)
    if count_all:
        result += " - {}%".format(int(100 * count_done / count_all))
    return result


if __name__ == "__main__":
    app.run_server(debug=True)

@alexcjohnson
Copy link
Collaborator Author

At a meeting where I demoed this feature, the consensus was ALL and ANY are easy to confuse with each other (and in a separate discussion @christianwengert seems to agree) - The proposal I think we agreed on is to change ANY to MATCH - which makes it clear that the values will match across Input and Output items for a given callback invocation.

We didn't discuss ALLSMALLER, as I didn't have a demo ready for it yet. Is it clear from the extended To-Do demo above what this does and how to use it?

Give me a 👍 reaction (not a comment please) if the names ALL, MATCH, and ALLSMALLER look good to you. If not and you have another suggestion please comment.

@nicolaskruchten
Copy link
Contributor

ALLSMALLER seems oddly specific to me... why not also ALLBIGGER ?

Copy link
Contributor

@Marc-Andre-Rivet Marc-Andre-Rivet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💃💃💃 :shipit: 🚢 ⛵️ 🎉🎉🎉

@alexcjohnson
Copy link
Collaborator Author

The issue @chriddyp mentioned #1103 (comment) from the docs, that the title never goes back from "Updating...", is still present. Investigating...

@alexcjohnson
Copy link
Collaborator Author

Ah got it - we're creating and throwing some errors that I forgot to trap, so I could clear the associated callback from the queue and surface the error in the devtools.

The specific error we're hitting in the docs is (in case someone wants to fix it):

Uncaught (in promise) ReferenceError: A nonexistent object was used in an Input of a Dash callback. The id of this object is memory-button and the property is n_clicks. The string ids in the current layout are: [memory, memory-output, local, session, wait-for-layout, location, banner, logo-home, backlinks-top, chapter, backlinks-bottom, pagemenu]

so we can clear the pendingCallbacks queue and show errors in the devtools
@alexcjohnson
Copy link
Collaborator Author

After 7c47b05 (running docs with devtools on so we see the errors... with devtools off the title is still correct and the errors only appear in the js console)

Screen Shot 2020-04-07 at 5 18 58 PM

@MM-Lehmann
Copy link

whats that rationale behind the restriction that I cannot combine MATCH with other (static) outputs? I would like to have a MATCH callback on some GUI elements which save their current state into a dcc.Store object and retrieve it later from there. Is it because in the background multiple callbacks are created for each match of objects and those can't have duplicate outputs between each other?

@alexcjohnson
Copy link
Collaborator Author

That’s essentially it - the value to return into that single output would be ill-defined, as it would be trying to get a separate value from each matched input.

I see how specifically with Store it would be possible to construct the callback in such a way as to make it self-consistent. But that kind of construction would be tricky and fragile. Keeping MATCH keys the same across all outputs makes this much simpler for everyone to reason about (including us writing the rendering engine!) but in return you sometimes have to handle your data a little differently.

In this case I would suggest either making a separate Store for each MATCH value, or a separate callback that combines the inputs with ALL and returns the single Store value.

@lhsusan1016
Copy link

When I used two outputs, one of which uses MATCH and the other has nothing to do with 'index':
@app.callback(
[Output({'type': 'dynamic-output', 'index': MATCH}, 'children'),
Output('title_button', 'children')],
[Input({'type': 'dynamic-dropdown', 'index': MATCH}, 'value')],
[State({'type': 'dynamic-dropdown', 'index': MATCH}, 'id')],
)

it raised an error as follows:

In the callback for output(s):
{"index":MATCH,"type":"dynamic-output"}.children
title_button.children
Output 1 (title_button.children)
does not have MATCH wildcards on the same keys as
Output 0 ({"index":MATCH,"type":"dynamic-output"}.children).
MATCH wildcards must be on the same keys for all Outputs.
ALL wildcards need not match, only MATCH.

Why is it necessary to require all the outputs with MATCH, and how can I solve this problem?
Thanks!!!

@alexcjohnson
Copy link
Collaborator Author

@lhsusan1016 as described in #1103 (comment) if there were two different elements that satisfy your MATCH pattern, then there would need to be two different invocations of this callback, one for each. Then this would be two different values for title_button.children - so which one would we use? It’s ambiguous and therefore an error.

You can solve this by either switching to an ALL pattern to create all the dynamic outputs at once, or making a separate callback for title_button.children

I happened to see your comment, but old closed PRs are not great places to start conversations. This kind of question is better at https://community.plotly.com

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
8 participants