Index out of Range Exception Randomely

[krforgit]

protobuf.js version: <6.8.0>

I am getting an index out of range exception randomly,
when I am subscribing a message from a server which is a console application,
this is the code snippet which is sending the message.

and the below is the JS snippet where the data is recieved but on continous recieving the protobufjs is throwing an index out of range exception randomely where I tried to read the data after the four bytes of the Sentmessage.

and the exception I am getting is

I am also facing an invalid wire type exception randomly and very rare.

Can anyone help me out getting the data continuously and root cause of the issue.

[taoxinyi]

I have the same problem that it randomly throws Index out of Range Exception. Did you solve it?

[dcodeIO]

One thing I notice there is that SendMessage.verify should not be used with a buffer (see). Other than that the error means that the decoder tried to read one more byte from the buffer at offset 1, but the buffer was only 1 bytes long, so this might either be an issue with the proto definition or on the side of the producer.

[taoxinyi]

@dcodeIO I used python django channels to send protobuf bytes data to the browser. I compare the the bytes data of the Uint8Array and the bytes data after SerializeToString in python, and they are identical, but in the python it can be decoded with ParseFromString but in the Javascript of the browser, it throws Index out of Range Exception when trying to decoded.

[dcodeIO]

I don't know what SerializeToString does under the hood, but usually serializing binary to a string can fail for a couple of reasons on the transport layer. If you can't transmit binary, maybe consider encoding as base64.

[taoxinyi]

@dcodeIO I find the there is something different from the bytes data I sent from python and the Uint8Array I convert in the browser.

Bytes data send from python:

18,150,1,8,3,18,4,84,97,111,66,34,55,47,109,101,100,105,97,47,117,115,101,114,47,112,104,111,116,111,47,50,48,49,54,47,49,49,47,50,52,47,81,81,229,155,190,231,137,135,50,48,49,54,49,49,50,51,50,48,50,56,53,56,46,112,110,103,40,206,183,160,216,5,50,33,109,101,100,105,97,47,117,115,101,114,47,51,47,49,95,50,48,49,56,48,53,50,53,50,50,50,49,48,50,46,100,111,99,56,128,168,147,1,66,5,49,46,100,111,99,51,220,236,109,97,170,194,130,112,227,129,178,43,193,56,119,182,187,95,133,2,235,26,9,232,159,138,68,225,51,30,170

I receive Blob in the client browser and I use FileReader into arrayBuffer , convert it to Uint8Array using Uint8Array(arrayBuffer), and the data in the Uint8Array is:

18,150,1,8,3,18,4,84,97,111,66,34,55,47,109,101,100,105,97,47,117,115,101,114,47,112,104,111,116,111,47,50,48,49,54,47,49,49,47,50,52,47,81,81,229,155,190,231,137,135,50,48,49,54,49,49,50,51,50,48,50,56,53,56,46,112,110,103,40,237,183,160,216,5,50,33,109,101,100,105,97,47,117,115,101,114,47,51,47,49,95,50,48,49,56,48,53,50,53,50,50,50,49,51,51,46,100,111,99,56,128,168,147,1,66,5,49,46,100,111,99,51,220,236,109,97,170,194,130,112,227,129,178,43,193,56,119,182,187,95,133,2,235,26,9,232,159,138,68,225,51,30,170

I don't know why the data is corrupted, am I doing wrong in converting Blob to Uint8Array?

[dcodeIO]

Things like these usually happen when the data is forced into some sort of string encoding like utf8. The best way to avoid this is to make sure that the data remains binary from its generation to its consumption, or, if string conversion is necessary for some reason, use base64 encoding.

[taoxinyi]

@dcodeIO In my case, the data server sent to client is binary. The client receives it using Websocket as Blob, and converts it to Uint8Arrary, so I don't understand which part of it contains string encoding.

[dcodeIO]

Hard to tell, but using a FileReader seems wrong as well (sounds like it involves string encoding to me). Should possibly use binaryType="arraybuffer" on the WebSocket instead (see). If that's not an option, base64 strings.

[taoxinyi]

@dcodeIO Thank you for the advice. I've replaced FileReader with binaryType="arraybuffer". But the result is the same. Maybe it's the problem of django channels that the binary data is somehow changed during the transmission. Thank you anyway.

[taoxinyi]

@dcodeIO Sorry to bother you again. Even the same bytes data cannot be decoded in the browser.

in Python

>>> b=bytearray()

>>> x=[18, 150, 1, 8, 3, 18, 4, 84, 97, 111, 66, 34, 55, 47, 109, 101, 100, 105, 97, 47, 117, 115, 101, 114, 47, 112, 104, 111, 116, 111, 47, 50, 48, 49, 54, 4
... 7, 49, 49, 47, 50, 52, 47, 81, 81, 229, 155, 190, 231, 137, 135, 50, 48, 49, 54, 49, 49, 50, 51, 50, 48, 50, 56, 53, 56, 46, 112, 110, 103, 40, 206, 183, 1
... 60, 216, 5, 50, 33, 109, 101, 100, 105, 97, 47, 117, 115, 101, 114, 47, 51, 47, 49, 95, 50, 48, 49, 56, 48, 53, 50, 53, 50, 50, 50, 49, 48, 50, 46, 100, 11
... 1, 99, 56, 128, 168, 147, 1, 66, 5, 49, 46, 100, 111, 99, 51, 220, 236, 109, 97, 170, 194, 130, 112, 227, 129, 178, 43, 193, 56, 119, 182, 187, 95, 133, 2,
...  235, 26, 9, 232, 159, 138, 68, 225, 51, 30, 170]

>>> for i in x:
...     b.append(i)

>>> b
bytearray(b'\x12\x96\x01\x08\x03\x12\x04TaoB"7/media/user/photo/2016/11/24/QQ\xe5\x9b\xbe\xe7\x89\x8720161123202858.png(\xce\xb7\xa0\xd8\x052!media/user/3/1_20180525222102.doc8\x80\xa8\x93\x01B\x051.doc3\xdc\xecma\xaa\xc2\x82p\xe3\x81\xb2+\xc18w\xb6\xbb_\x85\x02\xeb\x1a\t\xe8\x9f\x8aD\xe13\x1e\xaa')

>>> bytes_data=bytes(b)

>>> chat_message = ChatMessage.ChatMessage()

>>> chat_message.ParseFromString(bytes_data)

>>> chat_message
chat_message_item {
  clientId: 3
  clientName: "TaoB"
  imageUrl: "/media/user/photo/2016/11/24/QQ\345\233\276\347\211\20720161123202858.png"
  timeStamp: 1527258062
  fileUrl: "media/user/3/1_20180525222102.doc"
  fileSize: 2413568
  fileName: "1.doc"
}

in browser

var uint8Array = new Uint8Array([18, 150, 1, 8, 3, 18, 4, 84, 97, 111, 66, 34, 55, 47, 109, 101, 100, 105, 97, 47, 117, 115, 101, 114, 47, 112, 104, 111, 116, 111, 47, 50, 48, 49, 54, 47, 49, 49, 47, 50, 52, 47, 81, 81, 229, 155, 190, 231, 137, 135, 50, 48, 49, 54, 49, 49, 50, 51, 50, 48, 50, 56, 53, 56, 46, 112, 110, 103, 40, 206, 183, 160, 216, 5, 50, 33, 109, 101, 100, 105, 97, 47, 117, 115, 101, 114, 47, 51, 47, 49, 95, 50, 48, 49, 56, 48, 53, 50, 53, 50, 50, 50, 49, 48, 50, 46, 100, 111, 99, 56, 128, 168, 147, 1, 66, 5, 49, 46, 100, 111, 99, 51, 220, 236, 109, 97, 170, 194, 130, 112, 227, 129, 178, 43, 193, 56, 119, 182, 187, 95, 133, 2, 235, 26, 9, 232, 159, 138, 68, 225, 51, 30, 170]);
var chatMessageDecoded = chatMessage.decode(uint8Array);

And the Error

protobuf.min.js:7 Uncaught RangeError: index out of range: 125 + 1799772 > 153
    at r (protobuf.min.js:7)
    at n.bytes (protobuf.min.js:7)
    at n.string (protobuf.min.js:7)
    at n.ChatMessageItem$decode [as decode] (eval at i (protobuf.min.js:7), <anonymous>:26:15)
    at n.decode (protobuf.min.js:7)
    at n.ChatMessage$decode [as decode] (eval at i (protobuf.min.js:7), <anonymous>:16:35)
    at n.decode (protobuf.min.js:7)
    at WebSocket.chatSocket.onmessage ((index):555)

Here is my .proto file

syntax = "proto3";
package chat;

message ChatMessageItem {


    int32 clientId = 1;
    string clientName = 2;
    string message = 3;
    string imageUrl = 4;
    int32 timeStamp = 5;
    string fileUrl = 6;
    int32 fileSize = 7;
    string fileName = 8;


}

message ChatMessage {
    enum Type {
        CHAT_MESSAGE = 0;
        CLIENT_ENTER = 1;
        CLIENT_LEAVE = 2;
    }
    Type type = 1;
    repeated ChatMessageItem chat_message_item = 2;
    repeated ChatMessageItem current_client = 3;
}

I don't know what's wrong in this.

[drawrowfly]

Same issue. Works fine with identical data in python but not in js. Any solution?

[tutulee]

@dcodeIO 很抱歉再次打扰您。即使相同的字节数据也无法在浏览器中解码。

在Python中

>>> b=bytearray()

>>> x=[18, 150, 1, 8, 3, 18, 4, 84, 97, 111, 66, 34, 55, 47, 109, 101, 100, 105, 97, 47, 117, 115, 101, 114, 47, 112, 104, 111, 116, 111, 47, 50, 48, 49, 54, 4
... 7, 49, 49, 47, 50, 52, 47, 81, 81, 229, 155, 190, 231, 137, 135, 50, 48, 49, 54, 49, 49, 50, 51, 50, 48, 50, 56, 53, 56, 46, 112, 110, 103, 40, 206, 183, 1
... 60, 216, 5, 50, 33, 109, 101, 100, 105, 97, 47, 117, 115, 101, 114, 47, 51, 47, 49, 95, 50, 48, 49, 56, 48, 53, 50, 53, 50, 50, 50, 49, 48, 50, 46, 100, 11
... 1, 99, 56, 128, 168, 147, 1, 66, 5, 49, 46, 100, 111, 99, 51, 220, 236, 109, 97, 170, 194, 130, 112, 227, 129, 178, 43, 193, 56, 119, 182, 187, 95, 133, 2,
...  235, 26, 9, 232, 159, 138, 68, 225, 51, 30, 170]

>>> for i in x:
...     b.append(i)

>>> b
bytearray(b'\x12\x96\x01\x08\x03\x12\x04TaoB"7/media/user/photo/2016/11/24/QQ\xe5\x9b\xbe\xe7\x89\x8720161123202858.png(\xce\xb7\xa0\xd8\x052!media/user/3/1_20180525222102.doc8\x80\xa8\x93\x01B\x051.doc3\xdc\xecma\xaa\xc2\x82p\xe3\x81\xb2+\xc18w\xb6\xbb_\x85\x02\xeb\x1a\t\xe8\x9f\x8aD\xe13\x1e\xaa')

>>> bytes_data=bytes(b)

>>> chat_message = ChatMessage.ChatMessage()

>>> chat_message.ParseFromString(bytes_data)

>>> chat_message
chat_message_item {
  clientId: 3
  clientName: "TaoB"
  imageUrl: "/media/user/photo/2016/11/24/QQ\345\233\276\347\211\20720161123202858.png"
  timeStamp: 1527258062
  fileUrl: "media/user/3/1_20180525222102.doc"
  fileSize: 2413568
  fileName: "1.doc"
}

在浏览器中

var uint8Array = new Uint8Array([18, 150, 1, 8, 3, 18, 4, 84, 97, 111, 66, 34, 55, 47, 109, 101, 100, 105, 97, 47, 117, 115, 101, 114, 47, 112, 104, 111, 116, 111, 47, 50, 48, 49, 54, 47, 49, 49, 47, 50, 52, 47, 81, 81, 229, 155, 190, 231, 137, 135, 50, 48, 49, 54, 49, 49, 50, 51, 50, 48, 50, 56, 53, 56, 46, 112, 110, 103, 40, 206, 183, 160, 216, 5, 50, 33, 109, 101, 100, 105, 97, 47, 117, 115, 101, 114, 47, 51, 47, 49, 95, 50, 48, 49, 56, 48, 53, 50, 53, 50, 50, 50, 49, 48, 50, 46, 100, 111, 99, 56, 128, 168, 147, 1, 66, 5, 49, 46, 100, 111, 99, 51, 220, 236, 109, 97, 170, 194, 130, 112, 227, 129, 178, 43, 193, 56, 119, 182, 187, 95, 133, 2, 235, 26, 9, 232, 159, 138, 68, 225, 51, 30, 170]);
var chatMessageDecoded = chatMessage.decode(uint8Array);

和错误

protobuf.min.js:7 Uncaught RangeError: index out of range: 125 + 1799772 > 153
    at r (protobuf.min.js:7)
    at n.bytes (protobuf.min.js:7)
    at n.string (protobuf.min.js:7)
    at n.ChatMessageItem$decode [as decode] (eval at i (protobuf.min.js:7), <anonymous>:26:15)
    at n.decode (protobuf.min.js:7)
    at n.ChatMessage$decode [as decode] (eval at i (protobuf.min.js:7), <anonymous>:16:35)
    at n.decode (protobuf.min.js:7)
    at WebSocket.chatSocket.onmessage ((index):555)

这是我的.proto文件

syntax = "proto3";
package chat;

message ChatMessageItem {


    int32 clientId = 1;
    string clientName = 2;
    string message = 3;
    string imageUrl = 4;
    int32 timeStamp = 5;
    string fileUrl = 6;
    int32 fileSize = 7;
    string fileName = 8;


}

message ChatMessage {
    enum Type {
        CHAT_MESSAGE = 0;
        CLIENT_ENTER = 1;
        CLIENT_LEAVE = 2;
    }
    Type type = 1;
    repeated ChatMessageItem chat_message_item = 2;
    repeated ChatMessageItem current_client = 3;
}

我不知道这有什么问题。

请问 解决了吗

[icyzeroice]

Hi, I just met the same issue and solve it via patching protobufjs with the following tricky code in decoder.js:

diff --git a/src/decoder.js b/src/decoder.js
index 491dd3059df42d28c2f486e89651985433f650c6..53f6ef95e069b36e627a859301b9495391ec60d6 100644
--- a/src/decoder.js
+++ b/src/decoder.js
@@ -59,6 +59,14 @@ function decoder(mtype) {
                         ("case 2:");
 
             if (types.basic[type] === undefined) gen
+                            ("if (2 !== (t&7)) {")
+                            ("r.skipType(t&7)")
+                            ("break")
+                            ("}")
                             ("value=types[%i].decode(r,r.uint32())", i); // can't be groups
             else gen
                             ("value=r.%s()", type);
@@ -91,14 +99,29 @@ function decoder(mtype) {
                 ("}else");
 
             // Non-packed
-            if (types.basic[type] === undefined) gen(field.resolvedType.group
+            if (types.basic[type] === undefined) gen
+                ("if (2 !== (t&7)) {")
+                ("r.skipType(t&7)")
+                ("break")
+                ("}")
+                (field.resolvedType.group
                     ? "%s.push(types[%i].decode(r))"
                     : "%s.push(types[%i].decode(r,r.uint32()))", ref, i);
             else gen
                     ("%s.push(r.%s())", ref, type);
 
         // Non-repeated
-        } else if (types.basic[type] === undefined) gen(field.resolvedType.group
+        } else if (types.basic[type] === undefined) gen
+            ("if (2 !== (t&7)) {")
+            ("r.skipType(t&7)")
+            ("break")
+            ("}")
+            (field.resolvedType.group
                 ? "%s=types[%i].decode(r)"
                 : "%s=types[%i].decode(r,r.uint32())", ref, i);
         else gen

I also create a fork with the fix here: icyzeroice@53f6ef9

The reason why I got this error is that the protobuf data I use has some invalid messages only have the field name and wrong wire type without the field value in it. So, I skip decoding this kind of empty data with wrong wire type. I do not know if there are other new problems with the solution, but it just works fine.

PS: I came into two different errors, and both of them are solved with the patch above.

Range Error: index out of range: 271309 + 10 > 271309

invalid wire type 6 at offset 11398

[TvojTatk0]

Hi. I have very same issue. This is opened for 5 years now so I wanna ask if this is even considered as a issue in protobuf.js ? @icyzeroice posted solution so it would be nice to know if it is considered as a fix and will be included in feature release or it have to be handled by integrator. Thanks

[TvojTatk0]

FYI: I was able to get rid off exception. The problem was probably in the content itself. Originally, my main content was encrypted using the JSEncrypt and crypto-js libraries. After switching to using only the web crypto API that works with ArrayBuffer, I no longer get the Index out of range exception. Therefore, I think the problem is related to the encoding of the content.

[opeolluwa]

what's the fix for this

[smpeters-cb]

what's the fix for this

I ran into this issue with an object/message that was all strings. It would randomly fail depending on the values passed in and sometimes even the order of the elements in the object. I resolved this by using encodeDelimited and decodeDelimited instead of encode and decode.

[opeolluwa]

I was able to fix mine.

It was a pub sub pattern and one of the pairs had a missing field