Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing lines when using readline Interface as AsyncIterable #28565

Open
amcheshire opened this issue Jul 5, 2019 · 2 comments
Open

Missing lines when using readline Interface as AsyncIterable #28565

amcheshire opened this issue Jul 5, 2019 · 2 comments
Labels
readline Issues and PRs related to the built-in readline module.

Comments

@amcheshire
Copy link

  • Version: v12.6.0
  • Platform: Darwin Admins-MacBook-Pro-6.local 18.6.0 Darwin Kernel Version 18.6.0: Thu Apr 25 23:16:27 PDT 2019; root:xnu-4903.261.4~2/RELEASE_X86_64 x86_64
  • Subsystem: readline

We were using readline to convert a stream that gives bytes to an AsyncIterable of lines. We did this by using something like:

const iterableOfLines = readline.createInterface({ input: streamOfBytes })

We would then use this as an AsyncIterable (which looks like it should be supported) and write it to a file, but there would sometimes be lines missing. The number of lines missing was inconsistent and we couldn't identify any patterns relating the lines that were missing. They were not different from the rest in any way, they were not always the first or last lines, etc.

Unfortunately, it's hard for me to tell you more because we were unable to write an isolated test that would reliably show this behavior (we noticed it in production).

We are confident that the bug is in readline because we replaced the use of it with our own custom function that would convert the bytes to lines, and we consistently get all of the lines every time.

@Fishrock123 Fishrock123 added the readline Issues and PRs related to the built-in readline module. label Jul 8, 2019
@jasnell
Copy link
Member

jasnell commented Jun 26, 2020

@nodejs/stream @BridgeAR ... any ideas?

@jfriend00
Copy link

Probably related to #33463.

benjamingr added a commit to benjamingr/io.js that referenced this issue Jan 26, 2022
Beforehand the async iterator for readline was created lazily when the
stream was accessed which meant no events were buffered. In addition
stream handling was modernized and improved in the process to support
backpressure correctly. A test was added to assert this.

Fixes: nodejs#28565
Fixes: nodejs#33463
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
readline Issues and PRs related to the built-in readline module.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants