Add failing AsyncTokenFilterTest.

[htmldoug]

I've been following your great work in #57 and finally found a use-case to try async parsing in one of my apps, but I think I've hit a bug.

Summary

I'd like to use a TokenFilter over a NonBlockingJsonParser to do json projection, but wrapping a FilteringParserDelegate around NonBlockingJsonParser leads to an infinite loop when passing a complete json object over several separate chunks.

I've included a failing test in this PR.

Infinite loop

This works fine at chunkSize 28 and 27, but we get stuck in an infinite loop at chunkSize 26. Thread point dump points to:

  at com.fasterxml.jackson.core.base.ParserMinimalBase.skipChildren(ParserMinimalBase.java:253)
  at com.fasterxml.jackson.core.filter.FilteringParserDelegate.nextToken(FilteringParserDelegate.java:412)
  at com.fasterxml.jackson.core.json.async.AsyncTokenFilterTest.testFilteredNonBlockingParser(AsyncTokenFilterTest.java:46)

Questions

Is my usage supported, or is there another/better way to accomplish my goal?

[cowtowncoder]

It is not possible to make skipChildren() work for all cases of non-blocking parsing, because it may or may not have enough content to complete skipping; and if not, there's no way to retain information on what would still be needed. So skipping logic will have to be done from outside, keeping necessary state information.
However, I think that attempts to try to do that should not end in infinite cycle. Not sure if it should fail for all attempts, or such cases where JsonToken.NOT_AVAILABLE is returned. Maybe latter?
I will create a new issue for such improvement.

As to whether non-blocking parser and delegate can work together... that seems more plausible.
I don't think it has been tested, but to degree it is doable it would be nice to improve.
But I think unit tests for failing cases would be useful. It should not be very difficult to write some for simple cases, as variations; especially when using very small buffer (like one byte at a time).

[htmldoug]

Having skipChildren() throw seems reasonable, or at least less surprising.

I've worked around this in my app by moving the TokenFilter to generation-time (FilteringGeneratorDelegate) instead of parse-time. I suspect it's less performant than filtering upfront, but it's good enough for now.

I think unit tests for failing cases would be useful.

You may find the one in this PR helpful. It covers the 1 byte at a time case that you mentioned, along with several others.

As to whether non-blocking parser and delegate can work together... that seems more plausible.

Agreed. Might need to track more state. Not sure. However, I found FilteringParserDelegate a bit daunting. Those changes seem like a bigger investment than I'm able to make right now.

Really appreciate the response, and thank you for making async json processing possible. Way cool. Whatever you want to do with this PR is fine by me.

[cowtowncoder]

@htmldoug Excellent. I hope to make some little fixes still before 2.9.6, but will need to publish that relatively soon. Thanks for the PR, I will need to read it with thought. Should be useful.

Also thank you for checking out async parsing: I hesitated for quite a while until implementing it (based on my earlier work on async XML for aalto-xml), since adoption of earlier work had been lower than expected. But uptake for json seems to have been bit more rapid -- and as long as there is a steady stream of early adopters, things work out quite nicely: there's a stream of bug reports, fixes, but no torrent of duplicates. :)