Error: RangeError: index out of range: 3 + 109 > 23

[James-Quigley]

See Elaniobro/MMM-nyc-transit#56

[ericandrewlewis]

This error happens when we attempt to decode a protocol buffer bytestring into a protocol buffer class instance.

Based on this comment, I understand that it means "the library tried to read 109 more bytes according to the protobuf wire format, but the buffer didn't contain these bytes".

I have occasionally seen this error in my subway arrivals app. I'd say the frequency is somewhere around 5%, although that's a guess-timate over the course of a few years of usage. The error lasts for a bit (10 minutes), and if I check back later the proto is parsed fine.

There is a list of common reasons this error happens, here are my notes on each:

Common error reason 1: The buffer has been corrupted somewhere on its way (usually through implicit string conversion, see also)

This issue implies that the binary data of the protocol buffer is forced into a string encoding like utf8, which is incorrect to do (link).

As far as I understand, the library handles the proto binary data correctly. We use the node-fetch package to make the HTTP request to the feed URL, retrieve the body in a buffer format (see code), and then pass that to the protobuf library to decode it (FeedMessage.decode()).

The protobuf.js library has an example of properly using buffers with the http library. Although we're using node-fetch instead of http, I don't that is a fundamental issue, especially since the problem is intermittent.

Common error reason 2: The schema doesn't match the data

There are updates to the gtfs-realtime.proto which we haven't pulled into this library. I would be surprised if this is the issue though. As I understand, a protobuf receiver (our library) should be able to accept messages from an updated sender (the MTA feed server), the receiver should ignore fields it doesn't recognize.

I think we should just do this anyway though. Will open a PR for this.

Common error reason 3: The buffer is / is not length delimited but is decoded as being / as being not length delimited

We call the FeedMessage.decode() function to decode our message. There is also FeedMessage.decodeDelimited(). To be honest I'm not sure if the buffer is or is not length delimited...maybe something to look into further that could break sometimes?

Common error reason 4: The data has additional, custom, non-protobuf bytes in it

This seems unlikely, unless something about the MTA's feed implementation is basically broken.

[ericandrewlewis]

I'm going to re-open because I see this in my app every once in a while.