smartcontract: escape non-ascii characters for manifest.Extra SI

[AnnaShaleva]

Problem

States diff at 284177 block of testnet.

Solution

Escape non-ASCII characters during manifest.Extra to stackitem conversion. This PR makes us compatible up to 290k of testnet.

Manifest bytes (got from transaction script):

7b226e616d65223a22696f75222c2267726f757073223a5b5d2c226665617475726573223a7b7d2c22737570706f727465647374616e6461726473223a5b5d2c22616269223a7b226d6574686f6473223a5b7b226e616d65223a22646563696d616c73222c22706172616d6574657273223a5b5d2c2272657475726e74797065223a22496e7465676572222c226f6666736574223a302c2273616665223a66616c73657d2c7b226e616d65223a2273796d626f6c222c22706172616d6574657273223a5b5d2c2272657475726e74797065223a22537472696e67222c226f6666736574223a322c2273616665223a66616c73657d2c7b226e616d65223a2262616c616e63654f66222c22706172616d6574657273223a5b5d2c2272657475726e74797065223a22496e7465676572222c226f6666736574223a382c2273616665223a66616c73657d2c7b226e616d65223a227472616e73666572222c22706172616d6574657273223a5b7b226e616d65223a2266726f6d222c2274797065223a2248617368313630227d2c7b226e616d65223a22746f222c2274797065223a2248617368313630227d2c7b226e616d65223a22616d6f756e74222c2274797065223a22496e7465676572227d2c7b226e616d65223a2264617461222c2274797065223a22416e79227d5d2c2272657475726e74797065223a22426f6f6c65616e222c226f6666736574223a31342c2273616665223a66616c73657d5d2c226576656e7473223a5b7b226e616d65223a225472616e73666572222c22706172616d6574657273223a5b7b226e616d65223a2261726731222c2274797065223a2248617368313630227d2c7b226e616d65223a2261726732222c2274797065223a2248617368313630227d2c7b226e616d65223a2261726733222c2274797065223a22496e7465676572227d5d7d5d7d2c227065726d697373696f6e73223a5b5d2c22747275737473223a5b5d2c226578747261223a7b22417574686f72223a226368656e7a6869746f6e67222c22456d61696c223a226368656e7a6869746f6e67406e67642e6e656f2e6f7267222c224465736372697074696f6e223a22494f555c75464630385c75364232305c75363736315c75354530315c75464630395c75464631415c75344530305c75373943445c75363532465c75363330315c75384431465c75363537305c75373638344e45502d31375c75464630385c75393735455c75344532355c75363833435c75363130465c75344534395c75344530415c75373638345c75464630395c75384434345c75344541375c75464630435c75353430385c75374541365c75363545305c75354235385c75353041385c75353333415c75464630435c75384432365c75363233375c75373533315c75353333415c75353735375c75393446455c75364434465c75383943385c75353636385c75374544465c7538424131227d7d

Items diff:

block 284177: value mismatch for key /////wj7OSJ9xTsoazY/iyD1aochwZCgDw==: QAUhApMAIQAoFPs5In3FOyhrNj+LIPVqhyHBkKAPKP09AU5FRjNOZW8uQ29tcGlsZXIuQ1NoYXJwIDMuMC4wLXJjMwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADvEUAMA0lPVUACAOH1BUBXAQR4cGgLlyYHEdsgIg14StkoUMoAFLOrqiYkDB9UaGUgYXJndW1lbnQgImZyb20iIGlzIGludmFsaWQuOnlwaAuXJgcR2yAiDXlK2ShQygAUs6uqJiIMHVRoZSBhcmd1bWVudCAidG8iIGlzIGludmFsaWQuOnoQtSYqDCVUaGUgYW1vdW50IG11c3QgYmUgYSBwb3NpdGl2ZSBudW1iZXIuOnhB+CfsjKomBxDbICIgwkp4z0p5z0p6zwwIVHJhbnNmZXJBlQFvYRHbICICQErZKFDKABSzq0BB+CfsjEAYZZ/uQQgoA2lvdUAASABAAEECQARBBSgIZGVjaW1hbHNAACEBESEAIABBBSgGc3ltYm9sQAAhARMhAQIgAEEFKAliYWxhbmNlT2ZAACEBESEBCCAAQQUoCHRyYW5zZmVyQARBAigEZnJvbSEBFEECKAJ0byEBFEECKAZhbW91bnQhARFBAigEZGF0YSEAIQEQIQEOIABAAUECKAhUcmFuc2ZlckADQQIoBGFyZzEhARRBAigEYXJnMiEBFEECKARhcmczIQERQABAACjVeyJBdXRob3IiOiJjaGVuemhpdG9uZyIsIkVtYWlsIjoiY2hlbnpoaXRvbmdAbmdkLm5lby5vcmciLCJEZXNjcmlwdGlvbiI6IklPVe+8iOasoOadoeW4ge+8ie+8muS4gOenjeaUr+aMgei0n+aVsOeahE5FUC0xN++8iOmdnuS4peagvOaEj+S5ieS4iueahO+8iei1hOS6p++8jOWQiOe6puaXoOWtmOWCqOWMuu+8jOi0puaIt+eUseWMuuWdl+mTvua1j+iniOWZqOe7n+iuoSJ9 vs QAUhApMAIQAoFPs5In3FOyhrNj+LIPVqhyHBkKAPKP09AU5FRjNOZW8uQ29tcGlsZXIuQ1NoYXJwIDMuMC4wLXJjMwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADvEUAMA0lPVUACAOH1BUBXAQR4cGgLlyYHEdsgIg14StkoUMoAFLOrqiYkDB9UaGUgYXJndW1lbnQgImZyb20iIGlzIGludmFsaWQuOnlwaAuXJgcR2yAiDXlK2ShQygAUs6uqJiIMHVRoZSBhcmd1bWVudCAidG8iIGlzIGludmFsaWQuOnoQtSYqDCVUaGUgYW1vdW50IG11c3QgYmUgYSBwb3NpdGl2ZSBudW1iZXIuOnhB+CfsjKomBxDbICIgwkp4z0p5z0p6zwwIVHJhbnNmZXJBlQFvYRHbICICQErZKFDKABSzq0BB+CfsjEAYZZ/uQQgoA2lvdUAASABAAEECQARBBSgIZGVjaW1hbHNAACEBESEAIABBBSgGc3ltYm9sQAAhARMhAQIgAEEFKAliYWxhbmNlT2ZAACEBESEBCCAAQQUoCHRyYW5zZmVyQARBAigEZnJvbSEBFEECKAJ0byEBFEECKAZhbW91bnQhARFBAigEZGF0YSEAIQEQIQEOIABAAUECKAhUcmFuc2ZlckADQQIoBGFyZzEhARRBAigEYXJnMiEBFEECKARhcmczIQERQABAACj9VgF7IkF1dGhvciI6ImNoZW56aGl0b25nIiwiRW1haWwiOiJjaGVuemhpdG9uZ0BuZ2QubmVvLm9yZyIsIkRlc2NyaXB0aW9uIjoiSU9VXHVGRjA4XHU2QjIwXHU2NzYxXHU1RTAxXHVGRjA5XHVGRjFBXHU0RTAwXHU3OUNEXHU2NTJGXHU2MzAxXHU4RDFGXHU2NTcwXHU3Njg0TkVQLTE3XHVGRjA4XHU5NzVFXHU0RTI1XHU2ODNDXHU2MTBGXHU0RTQ5XHU0RTBBXHU3Njg0XHVGRjA5XHU4RDQ0XHU0RUE3XHVGRjBDXHU1NDA4XHU3RUE2XHU2NUUwXHU1QjU4XHU1MEE4XHU1MzNBXHVGRjBDXHU4RDI2XHU2MjM3XHU3NTMxXHU1MzNBXHU1NzU3XHU5NEZFXHU2RDRGXHU4OUM4XHU1NjY4XHU3RURGXHU4QkExIn0=

String representation of manifest.Extra bytes for NeoGo node:

String representation of manifest.Extra bytes for C# node:

[codecov]

Codecov Report

Merging #2174 (e9cf2f0) into master (868835f) will increase coverage by 0.04%.
The diff coverage is 58.33%.

❗ Current head e9cf2f0 differs from pull request most recent head dfc0b25. Consider uploading reports for the commit dfc0b25 to get more accurate results

@@            Coverage Diff             @@
##           master    #2174      +/-   ##
==========================================
+ Coverage   83.75%   83.79%   +0.04%     
==========================================
  Files         295      295              
  Lines       28217    28228      +11     
==========================================
+ Hits        23633    23655      +22     
+ Misses       3242     3227      -15     
- Partials     1342     1346       +4     

Impacted Files	Coverage Δ
pkg/smartcontract/manifest/manifest.go	96.34% <58.33%> (-3.01%)	⬇️
pkg/core/native/ledger.go	90.51% <0.00%> (-3.45%)	⬇️
pkg/consensus/consensus.go	73.15% <0.00%> (+0.54%)	⬆️
pkg/services/oracle/request.go	62.50% <0.00%> (+3.26%)	⬆️
pkg/services/oracle/oracle.go	86.48% <0.00%> (+9.00%)	⬆️
pkg/network/metrics/metrics.go	84.61% <0.00%> (+15.38%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 868835f...dfc0b25. Read the comment docs.

[AnnaShaleva]

@roman-khimov, I wasn't able to find standard library solution of how to escape non-ascii characters (https://golang.org/pkg/strconv/?m=old#QuoteToASCII is similar to what we're looking for, but we don't need to apply it to the whole string and don't need quotes also). golang/go#39137 is somehow related to that.

This PR contains a cludge, so could you review it and tell whether it is appropriate for us to solve the problem this way?

[AnnaShaleva]

Hi, neo-project/neo#2345.

[AnnaShaleva]

#2026 related.

[AnnaShaleva]

I found out that escaping non-ASCII characters is the wrong solution. The problem is in go-ordered-json implementation that is used to remove indentations from the source raw json. go-ordered-json prettifies escaped runes, see https://github.com/virtuald/go-ordered-json/blob/master/decode.go#L1243, so after unmarshalling extra by go-ordered-json we got non-escaped bytes: https://github.com/nspcc-dev/neo-go/blob/master/pkg/smartcontract/manifest/manifest.go#L186.

[roman-khimov]

Unescaping during decoding is actually a correct behaviour. The funny thing is that we've tried to avoid this decoding/encoding, but we've got #2026. Now we decode/encode and get this. C# compatibility costs a lot...

[roman-khimov]

We can fork go-ordered-json into some neo-ordered-json and add all the quirks needed directly there. Yeah, not the funniest thing ever, but go-ordered-json itself is very stable (doesn't have any changes in four years, we're not likely to have any problems from this fork), we already have some compatibility quirks and we'll inevitably have more of them (just because C# is very different in how it works with JSON and we must be byte-to-byte compatible with it). So we can concentrate all the kludges needed in this package (and make them at least a bit more efficient, current solution makes another output copy) and use it in the node then (and wherever this JSON dialect is needed).

[AnnaShaleva]

Finally, the issue is fixed in nspcc-dev/go-ordered-json#1, so just update the dependencies.