Stream File Problems 2.6.2

[laercionit]

##Platform

• Hardware: ESP-12
• Core Version: 2.6.2
• Development Env: Arduino
• Operating System: Windows

Settings in IDE

Problem Description

Webserver stream problems:

bool ServerFileReadFS(String Path) {
  Waiting();
  if (Path.endsWith(F("/"))) Path += F("home.htm");
  String contentType = ServerContentType(Path);
  String PathWithGz = Path + F(".gz");
  if (FilesFSExists(PathWithGz))  Path += F(".gz");
  if (FilesFSOpen(Path)) {
    DebugSerial.println(String(F("ServerFileReadFS: Load ")) + (Path));
    if (server.streamFile(FilesFS, contentType) == FilesFS.size()) {
      FilesFSClose();
      return 1;
    } //else DebugSerial.println(String(F("ServerLoadFile: FS Error stream File ")) + (Path));
  } //else DebugSerial.println(String(F("ServerLoadFile: FS File not Exists ")) + (Path));
  FilesFSClose();
  return 0;
}

Debug Messages

ServerFileReadFS: Load /home.htm.gz

User exception (panic/abort/assert)
Panic src/include/DataSource.h:93 const uint8_t* BufferedStreamDataSource<TStream>::get_buffer(size_t) [with TStream = Stream; uint8_t = unsigned char; size_t: Assertion 'cb == stream_rem' failed.

>>>stack>>>

ctx: cont
sp: 3ffffd30 end: 3fffffc0 offset: 0000
3ffffd30:  3fff6364 0000036f 3fff300c 40245e0d  
3ffffd40:  000000fe 00000000 00000000 00000000  
3ffffd50:  00000000 00000000 00000000 4028f974  
3ffffd60:  3fff1768 00000000 3fff4b0c 0000036f  
3ffffd70:  0000036f 00000000 3fff4b0c 402427c6  
3ffffd80:  00000000 00000000 4bc6a7f0 4024280e  
3ffffd90:  3fff5b5c 00000000 3fff4b0c 40234f6a  
3ffffda0:  00000000 3fff02c8 00000020 0000036f  
3ffffdb0:  00000000 0000036f 3fff5b5c 40235145  
3ffffdc0:  3fff5b5c 0000086a 0000001c 0000086a  
3ffffdd0:  00000001 3fff0bfc 3fff5b5c 40235206  
3ffffde0:  3fff5b5c 3fff0bfc 00000000 40235740  
3ffffdf0:  80fffe88 3ffffe10 3fff0bfc 0000086a  
3ffffe00:  3ffffe88 3fff02c8 3fff0bfc 4022f5e1  
3ffffe10:  3fff4b00 000c000f 80fffe4c 40240a51  
3ffffe20:  3fff2134 3ffffeb0 3ffffeb0 3fff1728  
3ffffe30:  00000000 00000001 3fff0bfc 4022f916  
3ffffe40:  3fff5f00 0023002f 80ff006d 3fff4b00  
3ffffe50:  0017001f 8028d524 3fff4b00 000c000f  
3ffffe60:  80000000 3fff2100 000c000f 80240c30  
3ffffe70:  3fff2100 000c000f 80ff006d 3fff4aec  
3ffffe80:  000c000f 00ff0308 74786574 6d74682f  
3ffffe90:  89ff006c 59524f54 80ff002f 40240e14  
3ffffea0:  00000000 3fff0308 3ffffed0 4022fab2  
3ffffeb0:  3fff2134 000c000f 09fe006d 4023f648  
3ffffec0:  00000000 3fff0308 3fff02c8 4022fc7a  
3ffffed0:  0000002f 00000019 81ff15f8 4023fcc8  
3ffffee0:  00000000 3fff0308 3fff02c8 401000e1  
3ffffef0:  00544500 3fff0308 3fff02c8 40230b3a  
3fffff00:  00000000 80000000 4bc6a700 00000000  
3fffff10:  8023f63c 3fff6000 0000007f 80005054  
3fffff20:  3fff02ec 3fff0308 00000001 40201974  
3fffff30:  00794235 0000000a 3fff15f8 4023fcc8  
3fffff40:  3fffdad0 00000000 3fff02c8 3fff1728  
3fffff50:  00000001 3fff02ec 3fff02c8 40231f74  
3fffff60:  4024d0b0 00000000 00001388 4022e934  
3fffff70:  00000000 3fff5b5c 80ff16e8 40203248  
3fffff80:  3fffdad0 00000000 3fff16e8 40232038  
3fffff90:  3fffdad0 00000000 3fff16e8 40232063  
3fffffa0:  feefeffe feefeffe feefeffe 402421a0  
3fffffb0:  feefeffe feefeffe 3ffe97a4 40101665  
<<<stack<<<

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

Decoding 28 results
0x40245e0d: spiffs_hydro_read at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\cores\esp8266\spiffs/spiffs_hydrogen.cpp line 418
:  (inlined by) SPIFFS_read at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\cores\esp8266\spiffs/spiffs_hydrogen.cpp line 429
0x4028f974: chip_v6_unset_chanfreq at ?? line ?
0x402427c6: raise_exception at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\cores\esp8266/core_esp8266_postmortem.cpp line 245
0x4024280e: __assert_func at ?? line ?
0x40234f6a: BufferedStreamDataSource ::get_buffer(unsigned int) at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\libraries\ESP8266WiFi\src/include/DataSource.h line 95
0x40235145: ClientContext::_write_some() at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\libraries\ESP8266WiFi\src/include/ClientContext.h line 495
0x40235206: ClientContext::_write_from_source(DataSource*) at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\libraries\ESP8266WiFi\src/include/ClientContext.h line 450
0x40235740: WiFiClient::write(Stream&) at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\libraries\ESP8266WiFi\src/include/ClientContext.h line 372
:  (inlined by) WiFiClient::write(Stream&) at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\libraries\ESP8266WiFi\src/WiFiClient.cpp line 242
0x4022f5e1: unsigned int esp8266webserver::ESP8266WebServerTemplate ::streamFile (fs::File&, String const&) at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\libraries\ESP8266WebServer\src/ESP8266WebServer.h line 157 (discriminator 1)
0x40240a51: String::invalidate() at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\cores\esp8266/WString.cpp line 140
0x4022f916: ServerFileReadFS(String) at C:\Users\Laercio Mendon�a\AppData\Local\Temp\arduino_build_763520\sketch/Server.h line 121
0x40240e14: String::operator=(String const&) at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\cores\esp8266/WString.cpp line 262
0x4022fab2: ServerLoadFile(String) at C:\Users\Laercio Mendon�a\AppData\Local\Temp\arduino_build_763520\sketch/Server.h line 134
0x4023f648: HardwareSerial::write(unsigned char const*, unsigned int) at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\cores\esp8266/HardwareSerial.h line 165
0x4022fc7a: operator() at C:\Users\Laercio Mendon�a\AppData\Local\Temp\arduino_build_763520\sketch/Server.h line 402
:  (inlined by) _M_invoke at d:\setup\programas\desenvolvimento\arduinocoretest\portable\packages\esp8266\tools\xtensa-lx106-elf-gcc\2.5.0-4-b40a506\xtensa-lx106-elf\include\c++\4.8.2/functional line 2071
0x4023fcc8: Print::println() at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\cores\esp8266/Print.cpp line 178
0x401000e1: std::function ::operator()() const at d:\setup\programas\desenvolvimento\arduinocoretest\portable\packages\esp8266\tools\xtensa-lx106-elf-gcc\2.5.0-4-b40a506\xtensa-lx106-elf\include\c++\4.8.2/functional line 2465
0x40230b3a: esp8266webserver::ESP8266WebServerTemplate ::_handleRequest() at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\libraries\ESP8266WebServer\src/ESP8266WebServer-impl.h line 673
0x40201974: esp8266webserver::FunctionRequestHandler ::canHandle(HTTPMethod, String) at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\libraries\ESP8266WebServer\src/detail/RequestHandlersImpl.h line 23
0x4023fcc8: Print::println() at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\cores\esp8266/Print.cpp line 178
0x40231f74: esp8266webserver::ESP8266WebServerTemplate ::handleClient() at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\libraries\ESP8266WebServer\src/ESP8266WebServer-impl.h line 335
0x4024d0b0: spiffs_cache_page_get_by_fd at ?? line ?
0x4022e934: SerialDebugMonitor() at C:\Users\Laercio Mendon�a\AppData\Local\Temp\arduino_build_763520\sketch/Serial.h line 20 (discriminator 1)
0x40203248: RTCMonitor() at C:\Users\Laercio Mendon�a\AppData\Local\Temp\arduino_build_763520\sketch/RTC.h line 227
0x40232038: Waiting at C:\Users\Laercio Mendon�a\AppData\Local\Temp\arduino_build_763520\sketch/Server.h line 410
:  (inlined by) ServerMonitor() at C:\Users\Laercio Mendon�a\AppData\Local\Temp\arduino_build_763520\sketch/Server.h line 411
0x40232063: loop at D:\Save\Aquarino\Projects\Aquarino-ControllerBeta/Aquarino-ControllerBeta.ino line 90
0x402421a0: loop_wrapper() at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\cores\esp8266/core_esp8266_main.cpp line 180
0x40101665: cont_wrapper at D:\SETUP\Programas\Desenvolvimento\ArduinoCoreTest\portable\packages\esp8266\hardware\esp8266\2.6.2\cores\esp8266/cont.S line 81

[d-a-v]

This is not an MCVE for us to be able to reproduce.
Can you also enable all debug options (including OOM)
or print getFreeHeap() around where your issue happens ?

[laercionit]

request: GET / HTTP/1.1
method: GET url: / search:
headerName: Host
headerValue: 192.168.1.110
headerName: Connection
headerValue: keep-alive
headerName: Upgrade-Insecure-Requests
headerValue: 1
headerName: User-Agent
headerValue: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36
headerName: Accept
headerValue: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3
headerName: Accept-Encoding
headerValue: gzip, deflate
headerName: Accept-Language
headerValue: pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7
:c0 1, 436
args:
args count: 0
args:
args count: 0
Request: /
Arguments:
final list of key/value pairs:
request handler not found
24248 HEAP FREE
ServerFileReadFS: Load /home.htm.gz
:wr 193 0
:wrc 193 193 0
:wr 2154 0
SPIFFS_read rc=-10014

User exception (panic/abort/assert)
Panic src/include/DataSource.h:93 const uint8_t* BufferedStreamDataSource::get_buffer(size_t) [with TStream = Stream; uint8_t = unsigned char; size_t: Assertion 'cb == stream_rem' failed.

[d-a-v]

SPIFFS_read rc=-10014

That seems to be the issue.
SPIFFS reports a file size,
Datasource tries to read that exact size

Arduino/libraries/ESP8266WiFi/src/include/DataSource.h

Lines 92 to 93 in 1d05283

	const size_t cb = _stream.readBytes(reinterpret_cast<char*>(_buffer.get() + stream_read), stream_rem);
	assert(cb == stream_rem);

but it is a short read. The following assert kills the esp.

The API is such that this function get_buffer() is not supposed to read anything shorter than the requested size.

@laercionit Can you verify your file is really unreadable ?

Anyway, this has to be fixed.
Marking it for v3.0.0

[laercionit]

OK, I just changed the code:

    if (server.streamFile (FilesFS, contentType) == FilesFS.size ()) {

for

server.streamFile (FilesFS, contentType);

and it worked correctly.

[laercionit]

Taking advantage of the subject, what are these new options for exeptions? What behavior for each of them?

[d-a-v]

First: in case of oom, new will return nullptr
Second: in case of oom, new will call abort()
Third: in case of oom, a real c++ exception is raised and is catchable (try..catch)

[array81]

I think I have a similar problem with 2.7.0. My webserver have a html page with same JS and CSS files.
When I try to load the page I get a similar problem.
Can you get the right code to read file from SPIFFS and send it to client and avoid this problem?

[MagicBugsBunny]

Also have the Problem when the web page load a CSV file from sd Card.

[schimkat]

Just upgraded to 2.7.1 .. and got the problem. It worked fine with the older version - but I cannot recall which version. Any workarounds out there? :-)

[d-a-v]

Can you have a look to this example which is able to stream files from FSes ?
https://github.com/esp8266/Arduino/tree/master/libraries/ESP8266WebServer/examples/FSBrowser
Don't forget to enable OOM debug option.

[MagicBugsBunny]

I've tested the example and it Looks ok.
Then i replace the files on sd Card with my "web page" files.
Same issue. As it load the CSV file the exception from post 1 appear.

[d-a-v]

I would be very nice if you could provide such files so we can reproduce and hopefully fix.

[MagicBugsBunny]

Edit by @devyte : removed attached file

Here the Example with my files for sd Card. Simple "Data Logger" with CSV upload.
I use a Wemos D1 mini (clone). Upload all and try to open http;//IP/index.html

My ESP crashes immediately on each try.
If i disconnect power (usb cable) and reconnect again, it is better?
I can load the page and can referesh or reload the page.
But after a while the esp crashes again.

[devyte]

Please don't attach files. Post your MCVE directly In this case.

[MagicBugsBunny]

sorry, i don't understand what you mean?
How should i post all my "Web Page" files?

don't know if a link is allowed …. http://www.schafhof24.de/3D/WebBrowser.zip

[devyte]

Direct file attachments are discouraged for security reasons.
External links are discouraged because they can be broken in the future, so the context is lost.
MCVE stands for Minimal Complete Verifiable Example. It means: reduce your code/files as much as you can to something small and manageable that reproduces the issue, then post the content directly as a comment, preferably with markdown.

[MagicBugsBunny]

@d-a-v so you can try to reproduces the issue?

[d-a-v]

@MagicBugsBunny I did try with SPIFFS and I get no error (although no graphics is displayed). I could see the file was transmitted using tcpdump).
I shall try again with SDFS just in case.
Did you try to Erase Flash: All flash content ?
Weird things can happen when firmware is changed.

[d-a-v]

@MagicBugsBunny Still working with SDFS.
This time I could see the 3KW load on Apr, 20, at 12PM.

@schimkat Same question,
can you try the Arduino IDE menu option Erase Flash: All flash content ?

@laercionit Does your issue still apply ?

[MagicBugsBunny]

i also tried with complete erase Flash but don't help.
Also used other wemos pro an other esp8266 node mcu, all the same :-(

if i load it to my wemos d1 mini and try to open the index.html i get the exception :-(
also if i try again and again … even i disconnect the power.

Can it be a Problem of the sd Card? Power Supply?

[d-a-v]

Please print ESP.getFullVersion().
Enable all/full debug messages, disable MDNS which is too noisy, then post serial logs.

(power supply has been the issue in some cases, try another if you can)

[MagicBugsBunny]

I changed the Sketch and upload it …. and than try to open the index.html

Changed already to another usb power supply and soldered a C over 5V.
Maybe with this it is a littel bit better, i think.

[MagicBugsBunny]

Connected! IP address: 192.168.178.47
[MDNSResponder] STA interface selected
[MDNSResponder] _allocUDPContext
[MDNSResponder] addService: Succeeded to add 'fsbrowser.http.tcp'!
Open http://fsbrowser.local/edit to open the FileSystem Browser
HTTP server started
SDK:2.2.2-dev(38a443e)/Core:2.7.1=20701000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-30-g92add50/BearSSL:5c771be
:urn 57
[MDNSResponder] _callProcess (9665, triggered by: 192.168.178.39)
[MDNSResponder] _parseMessage (Time: 9666 ms, heap: 46320 bytes, from 192.168.178.39(5353), to 224.0.0.251(5353))
:urd 2, 57, 0
:urd 1, 57, 2
:urd 1, 57, 3
:urd 2, 57, 4
:urd 2, 57, 6
:urd 2, 57, 8
:urd 2, 57, 10
[MDNSResponder] _parseMessage: Done (Succeeded after 25 ms, ate 0 bytes, remaining 46320)

pm open,type:2 0
:urch 85, 57
[MDNSResponder] _callProcess (14683, triggered by: 192.168.178.39)
[MDNSResponder] _parseMessage (Time: 14684 ms, heap: 45800 bytes, from 192.168.178.39(5353), to 224.0.0.251(5353))
:urd 2, 57, 0
:urd 1, 57, 2
:urd 1, 57, 3
:urd 2, 57, 4
:urd 2, 57, 6
:urd 2, 57, 8
:urd 2, 57, 10
[MDNSResponder] _parseMessage: Done (Succeeded after 24 ms, ate 0 bytes, remaining 45800)

:urch 85, 57
[MDNSResponder] _callProcess (19701, triggered by: 192.168.178.39)
[MDNSResponder] _parseMessage (Time: 19701 ms, heap: 45800 bytes, from 192.168.178.39(5353), to 224.0.0.251(5353))
:urd 2, 57, 0
:urd 1, 57, 2
:urd 1, 57, 3
:urd 2, 57, 4
:urd 2, 57, 6
:urd 2, 57, 8
:urd 2, 57, 10
[MDNSResponder] _parseMessage: Done (Succeeded after 25 ms, ate 0 bytes, remaining 45800)

:urch 85, 57
[MDNSResponder] _callProcess (24717, triggered by: 192.168.178.39)
[MDNSResponder] _parseMessage (Time: 24718 ms, heap: 45800 bytes, from 192.168.178.39(5353), to 224.0.0.251(5353))
:urd 2, 57, 0
:urd 1, 57, 2
:urd 1, 57, 3
:urd 2, 57, 4
:urd 2, 57, 6
:urd 2, 57, 8
:urd 2, 57, 10
[MDNSResponder] _parseMessage: Done (Succeeded after 25 ms, ate 0 bytes, remaining 45800)

:urch 85, 57
[MDNSResponder] _callProcess (29735, triggered by: 192.168.178.39)
[MDNSResponder] _parseMessage (Time: 29736 ms, heap: 45800 bytes, from 192.168.178.39(5353), to 224.0.0.251(5353))
:urd 2, 57, 0
:urd 1, 57, 2
:urd 1, 57, 3
:urd 2, 57, 4
:urd 2, 57, 6
:urd 2, 57, 8
:urd 2, 57, 10
[MDNSResponder] _parseMessage: Done (Succeeded after 24 ms, ate 0 bytes, remaining 45800)

WS:ac
:rn 376
:ref 1
WS:av status=4 WCav=376
:ref 2
:ur 2
New client
:ref 2
:ur 2
request: GET /index.html HTTP/1.1
method: GET url: /index.html search: 
headerName: Accept
headerValue: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
headerName: Accept-Language
headerValue: de-DE
headerName: Upgrade-Insecure-Requests
headerValue: 1
headerName: User-Agent
headerValue: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18363
headerName: Accept-Encoding
headerValue: gzip, deflate
headerName: Host
headerValue: 192.168.178.47
headerName: Connection
headerValue: Keep-Alive
:c0 1, 376
args: 
args count: 0
args: 
args count: 0
Request: /index.html
Arguments: 
final list of key/value pairs:
request handler not found
handleFileRead: /index.html
:wr 85 0
:wrc 85 85 0
:wr 2071 0
:wrc 987 2071 0
:ack 85
:ack 536
:ack 451
:wr 1084 987

User exception (panic/abort/assert)
Panic src/include/DataSource.h:93 const uint8_t* BufferedStreamDataSource<TStream>::get_buffer(size_t) [with TStream = Stream; uint8_t = unsigned char; size_t: Assertion 'cb == stream_rem' failed.

>>>stack>>>

ctx: cont
sp: 3ffffd50 end: 3fffffc0 offset: 0000
3ffffd50:  3fff0efc 00000200 0000040b 4020f778  
3ffffd60:  000000fe 00000000 00000000 00000000  
3ffffd70:  00000000 00000000 00000000 40251674  
3ffffd80:  3ffeeea8 ffffffff 3fff0ffc 00000430  
3ffffd90:  00000430 00000000 3fff0ffc 4021480a  
3ffffda0:  00000430 402061ac 00000430 4021483e  
3ffffdb0:  007a1200 00000000 3fff0ffc 402061ea  
3ffffdc0:  00000000 00000000 3ffeeec8 00000430  
3ffffdd0:  3fff0ffc 00000430 3fff0004 402064f5  
3ffffde0:  00000000 3ffeeec8 00000001 402142f8  
3ffffdf0:  4021490d 00000001 00000817 00000817  
3ffffe00:  00000001 00000004 3fff0004 402065d1  
3ffffe10:  3fff0004 3ffffe40 00000000 40206b60  
3ffffe20:  3ffeed1c 3ffffe64 00000817 3ffeeb14  
3ffffe30:  3ffeed1c 3ffe8528 00000817 40204210  
3ffffe40:  40216fc8 00000000 000003e8 402131e4  
3ffffe50:  3fff0f5c 3fff0fd4 00000000 00000000  
3ffffe60:  3ffeece4 65646e00 74682e78 80006c6d  
3ffffe70:  74786574 6d74682f 89ff006c 402133c8  
3ffffe80:  00000000 3ffeeb54 3ffffeb0 402133f8  
3ffffe90:  3ffffec8 00000000 000a0d27 3ffeee68  
3ffffea0:  00000000 3ffeeb54 3ffeeb14 402042af  
3ffffeb0:  3fff0ecc 000b000f 00ff1100 3ffeee68  
3ffffec0:  4021210c 3ffeed1c 3fff08ac 000b000f  
3ffffed0:  0a006d74 3ffeed1c 3ffe87fd 40212519  
3ffffee0:  00000000 3ffeeb54 3ffeeb14 40215c4e  
3ffffef0:  00000000 3ffeeb54 3ffeed1c 401000e1  
3fffff00:  3fff0800 3ffeeb54 3ffeeb14 40204489  
3fffff10:  3fffff30 80ffff20 3fff1100 0000008f  
3fffff20:  80007865 00000002 3ffeec1c 00004bc6  
3fffff30:  3ffeeb54 00000001 3ffe87fd 40212519  
3fffff40:  00000000 00000001 0000000a 40206328  
3fffff50:  3fffdad0 00000000 3ffeeb14 3ffeee68  
3fffff60:  00000001 3ffeeb38 3ffeeb14 40204c1b  
3fffff70:  40216c60 00000000 00001388 80006c6d  
3fffff80:  00000000 3fff0004 00000001 40100364  
3fffff90:  3fffdad0 00000000 3ffeee28 40204cbc  
3fffffa0:  3fffdad0 00000000 3ffeee28 40214410  
3fffffb0:  feefeffe feefeffe 3ffe8548 40101535  
<<<stack<<<

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3456, room 16 
tail 0
chksum 0x84
csum 0x84
va5432625
~ld

SDK:2.2.2-dev(38a443e)/Core:2.7.1=20701000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-30-g92add50/BearSSL:5c771be


Filesystem initialized.
Connecting to WLAN_43
scandone
wifi evt: 2
.....scandone
state: 0 -> 2 (b0)
.state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 10
cnt 

connected with WLAN_43, channel 3
dhcp client start...
wifi evt: 0
ip:192.168.178.47,mask:255.255.255.0,gw:192.168.178.1
wifi evt: 3
.
Connected! IP address: 192.168.178.47
[MDNSResponder] STA interface selected
[MDNSResponder] _allocUDPContext
[MDNSResponder] addService: Succeeded to add 'fsbrowser.http.tcp'!
Open http://fsbrowser.local/edit to open the FileSystem Browser
HTTP server started
SDK:2.2.2-dev(38a443e)/Core:2.7.1=20701000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-30-g92add50/BearSSL:5c771be

[MagicBugsBunny]

I use "old" 2GB Micro SD Cards, up to 5 different one, with a normal sd adaptor.

Now i tried a newer 32GB sd Card and it Looks ok?
Tried 8GB Micro sd Card with the Adapter and it Looks ok?
With a older normal 8GB sd Card it Looks also ok.

So think there is a Problem with this 2GB Micro sd Cards?

I can't belive it.

[d-a-v]

Please keep your old 2GB card safe. Thanks for the log. That's valuable information.
I shall try to artificially slow down things or find another way to reproduce.
Code needs to be read again between lines.

[MagicBugsBunny]

ok, i have enough of them laught

one of them i Format with Windows, fat or fat32, quick and normal but it doesn't matter (also done before)

no i tried an old Programm to Format sd Cards, overwrite them or erase them. I let Format one of the 2GB Card with an full overwrite before.
Now it Looks also ok with this Card?

[d-a-v]

I shall try with an SD card with read errors then.
Improving the error message should help us with further issues of that kind.

[MagicBugsBunny]

I went back to my original Sketch and get the exception again with the "bad 2GB Micro sd Card".
With a Standard 8GB sd Card it is also ok.

The main difference is, at start of my Sketch it open the csv file a wrtite the time with a value.
So i think the main Problem is the write Operation?

[d-a-v]

The issue occurs when a stream reports a size but ::read(thisSize) is short (returns less than the reported size).
Maybe there is a bug somewhere in SD code where when a write is done/buffered/not/written or quite so, reported size will not match what happens with the following read.
If you have an MCVE (your initial write, the network transfer, all in the good configuration for everything to fall into custard), we'd be happy (I at least) to debug and see how it can be fixed.

[devyte]

If it's an sd code problem, it should be possible to reproduce without any network code. I suggest trying to reproduce by doing writes with synthetic data.

[MagicBugsBunny]

If i put this at the end of Setup :

File dataFile = SDFS.open("/load.csv", "w");
dataFile.print("1590062229");
dataFile.print(',');
dataFile.println("-1");
dataFile.close();

I can't load the web page …. exception …
Some times i can load but when i reload the page -> exception

When i remove the above Code from Setup, all is ok with the 2GB Micro sd Card.

[d-a-v]

@MagicBugsBunny your SD card must be in bad shape.
With this added code, not only it works, but I can see the -1 dot on the "yesterday" graph.
Do you not have this issue with another sd card ?

[MagicBugsBunny]

This issue i have only with those 2GB Micro sd Card …. but 5 of them can't be bad?
If i check them with a test Programm for Flash Memory all is ok. This will write the complete Memory and verify the read.

And with other sd Card > 2GB there is no Problem and all works fine.

[d-a-v]

@MagicBugsBunny
Can you apply this patch and restart with your SD card (with debug mode enabled) so we can have a little more input ?

diff --git a/libraries/ESP8266WiFi/src/include/DataSource.h b/libraries/ESP8266WiFi/src/include/DataSource.h
index 2a0bfed2..2436a0dd 100644
--- a/libraries/ESP8266WiFi/src/include/DataSource.h
+++ b/libraries/ESP8266WiFi/src/include/DataSource.h
@@ -90,6 +90,9 @@ public:
             //Remaining bytes to read from stream
             const size_t stream_rem = size - stream_read;
             const size_t cb = _stream.readBytes(reinterpret_cast<char*>(_buffer.get() + stream_read), stream_rem);
+            if (cb != stream_rem) {
+                DEBUGV(":DS-FAIL total=%z asked=rem=%z got=%z\r\n", size, stream_rem, cb);
+            }
             assert(cb == stream_rem);
             (void)cb;
             _streamPos += stream_rem;

[MagicBugsBunny]

ok, now in the Right one :-) DS-FAIL is present but with no values

SDK:2.2.2-dev(38a443e)/Core:2.7.1=20701000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-30-g92add50/BearSSL:5c771be


Filesystem initialized.
Connecting to WLAN_43
scandone
wifi evt: 2
.....scandone
state: 0 -> 2 (b0)
.state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 2
cnt 

connected with WLAN_43, channel 3
dhcp client start...
wifi evt: 0
ip:192.168.178.47,mask:255.255.255.0,gw:192.168.178.1
wifi evt: 3
.
Connected! IP address: 192.168.178.47
HTTP server started
SDK:2.2.2-dev(38a443e)/Core:2.7.1=20701000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-30-g92add50/BearSSL:5c771be
WS:ac
:ref 1
WS:av status=4 WCav=0
:ref 2
:ur 2
New client
:ref 2
:ur 2
:rn 402
request: GET /index.html HTTP/1.1
method: GET url: /index.html search: 
headerName: Cache-Control
headerValue: max-age=0
headerName: Accept
headerValue: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
headerName: Accept-Language
headerValue: de-DE
headerName: Upgrade-Insecure-Requests
headerValue: 1
headerName: User-Agent
headerValue: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18363
headerName: Accept-Encoding
headerValue: gzip, deflate
headerName: Host
headerValue: 192.168.178.47
headerName: Connection
headerValue: Keep-Alive
:c0 1, 402
args: 
args count: 0
args: 
args count: 0
Request: /index.html
Arguments: 
final list of key/value pairs:
request handler not found
handleFileRead: /index.html
:wr 85 0
:wrc 85 85 0
:wr 2071 0
:wrc 987 2071 0
:ack 85
:wr 1084 987
:wrc 85 1084 0
:ack 536
:wr 999 1072
:wrc 536 999 0
:ack 451
:wr 463 1608
:wrc 451 463 0
:ack 85
:ack 536
:wr 12 2059
:wrc 12 12 0
:ack 451
WS:ac
:rn 364
:ack 12
:rcl pb=0 sz=-1
:abort
:ur 1
:dsrcv 0
WS:dis
:del
:ref 1
WS:av status=4 WCav=364
:ref 2
:ur 2
New client
:ref 2
:ur 2
request: GET /temperatureGraph.js HTTP/1.1
method: GET url: /temperatureGraph.js search: 
headerName: Referer
headerValue: http://192.168.178.47/index.html
headerName: Cache-Control
headerValue: max-age=0
headerName: Accept
headerValue: */*
headerName: Accept-Language
headerValue: de-DE
headerName: Accept-Encoding
headerValue: gzip, deflate
headerName: User-Agent
headerValue: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18363
headerName: Host
headerValue: 192.168.178.47
headerName: Connection
headerValue: Keep-Alive
:c0 1, 364
args: 
args count: 0
args: 
args count: 0
Request: /temperatureGraph.js
Arguments: 
final list of key/value pairs:
request handler not found
handleFileRead: /temperatureGraph.js
:wr 98 0
:wrc 98 98 0
:wr 6351 0
:wrc 974 6351 0
:ack 98
:ack 536
:ack 438
:wr 5377 974
:wrc 1072 5377 0
:wr 4305 2046
:ack 536
:ack 536
:wr 4305 2046
:DS-FAIL total= asked=rem= got=

User exception (panic/abort/assert)
Panic src/include/DataSource.h:98 const uint8_t* BufferedStreamDataSource<TStream>::get_buffer(size_t) [with TStream = Stream; uint8_t = unsigned char; size_t: Assertion 'cb == stream_rem' failed.

>>>stack>>>

ctx: cont
sp: 3ffffd50 end: 3fffffc0 offset: 0000
3ffffd50:  00000001 00000000 0000042e 402090f8  
3ffffd60:  000000fe 00000000 00000000 00000000  
3ffffd70:  00000000 00000000 00000000 4024a958  
3ffffd80:  3ffeec2c 00000000 00000004 00000430  
3ffffd90:  00000430 00000000 3fff07c4 4020de82  
3ffffda0:  00000000 ffffffff 00000004 4020deb6  
3ffffdb0:  007a1200 00000000 3fff07c4 40205e1a  
3ffffdc0:  00000000 00000000 3ffeec4c 00000430  
3ffffdd0:  3fff07c4 00000430 3fff087c 40206129  
3ffffde0:  00000000 3ffeec4c 00000001 4020d970  
3ffffdf0:  4020df85 00000001 000018cf 000018cf  
3ffffe00:  00000001 00000006 3fff087c 40206205  
3ffffe10:  3fff087c 3ffffe40 00000000 40206794  
3ffffe20:  3ffeeab8 3ffffe64 000018cf 3ffee964  
3ffffe30:  3ffeeab8 3ffe8524 000018cf 402041f4  
3ffffe40:  402105b0 00000000 000003e8 4020c854  
3ffffe50:  3fff049c 3fff079c 00000000 00000000  
3ffffe60:  3ffeea80 3fff0700 0013001f 8020c854  
3ffffe70:  3fff0334 0016001f 00fffeb0 4020ca38  
3ffffe80:  00000000 3ffee9a4 3ffffeb0 4020ca68  
3ffffe90:  3ffffec8 00000000 7a25000a 3ffeebec  
3ffffea0:  00000000 3ffee9a4 3ffee964 40204293  
3ffffeb0:  3fff0304 0014001f 00ff0300 3ffeebec  
3ffffec0:  4020ba74 3ffeeab8 3fff03c4 0014001f  
3ffffed0:  0a007574 3ffeeab8 3ffe8803 4020be85  
3ffffee0:  00000000 3ffee9a4 3ffee964 4020f5a6  
3ffffef0:  00000000 3ffee9a4 3ffeeab8 401000e1  
3fffff00:  3fff0300 3ffee9a4 3ffee964 4020446d  
3fffff10:  3fffff30 80ffff20 3fff0700 0000008f  
3fffff20:  80006570 00000002 3ffeea6c 00004bc6  
3fffff30:  3ffee9a4 00000001 3ffe8803 4020be85  
3fffff40:  00000000 00000001 0000000a 40205f5c  
3fffff50:  3fffdad0 00000000 3ffee964 3ffeebec  
3fffff60:  00000001 3ffee988 3ffee964 40204bff  
3fffff70:  402102b8 00000000 00001388 feefeffe  
3fffff80:  00000000 3fff087c 00000001 40100198  
3fffff90:  3fffdad0 00000000 3ffeebac 40204ca0  
3fffffa0:  3fffdad0 00000000 3ffeebac 4020da88  
3fffffb0:  feefeffe feefeffe 3ffe8544 40101369  
<<<stack<<<

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3456, room 16 
tail 0
chksum 0x84
csum 0x84
va5432625
~ld

[d-a-v]

@MagicBugsBunny No mistake from your side, thanks for the report
I was using the wrong printf. Can you please replace %z by %u in the patch and retry ?

    DEBUGV(":DS-FAIL total=%u asked=rem=%u got=%u\r\n",
        (int)size, 
        (int)stream_rem,
        (int)cb);

[MagicBugsBunny]

SDK:2.2.2-dev(38a443e)/Core:2.7.1=20701000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-30-g92add50/BearSSL:5c771be

Filesystem initialized.
Connecting to WLAN_43
bcn 0
del if1
usl
mode : sta(2c:f4:32:5d:c3:64)
add if0
wifi evt: 8
wifi evt: 2
.....scandone
state: 0 -> 2 (b0)
.state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 9
cnt

connected with WLAN_43, channel 3
dhcp client start...
wifi evt: 0
ip:192.168.178.47,mask:255.255.255.0,gw:192.168.178.1
wifi evt: 3
.
Connected! IP address: 192.168.178.47
HTTP server started
SDK:2.2.2-dev(38a443e)/Core:2.7.1=20701000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-30-g92add50/BearSSL:5c771be
WS:ac
:rn 376
:ref 1
WS:av status=4 WCav=376
:ref 2
:ur 2
New client
:ref 2
:ur 2
request: GET /index.html HTTP/1.1
method: GET url: /index.html search:
headerName: Accept
headerValue: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
headerName: Accept-Language
headerValue: de-DE
headerName: Upgrade-Insecure-Requests
headerValue: 1
headerName: User-Agent
headerValue: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18363
headerName: Accept-Encoding
headerValue: gzip, deflate
headerName: Host
headerValue: 192.168.178.47
headerName: Connection
headerValue: Keep-Alive
:c0 1, 376
args:
args count: 0
args:
args count: 0
Request: /index.html
Arguments:
final list of key/value pairs:
request handler not found
handleFileRead: /index.html
:wr 85 0
:wrc 85 85 0
:wr 2071 0
:wrc 987 2071 0
:ack 85
:ack 536
:ack 451
:wr 1084 987
:DS-FAIL total=1072 asked=rem=1072 got=4294967295

User exception (panic/abort/assert)
Panic src/include/DataSource.h:101 const uint8_t* BufferedStreamDataSource::get_buffer(size_t) [with TStream = Stream; uint8_t = unsigned char; size_: Assertion 'cb == stream_rem' failed.

stack>>>

ctx: cont
sp: 3ffffd50 end: 3fffffc0 offset: 0000
3ffffd50: 00000002 00000000 0000040b 402090f8
3ffffd60: 000000fe 00000000 00000000 00000000
3ffffd70: 00000000 00000000 00000000 4024a958
3ffffd80: 3ffeec2c 00000000 00000004 00000430
3ffffd90: 00000430 00000000 3fff072c 4020de82
3ffffda0: 00000000 ffffffff 00000430 4020deb6
3ffffdb0: 007a1200 00000000 3fff072c 40205e1a
3ffffdc0: 00000000 00000000 3ffeec4c 00000430
3ffffdd0: 3fff072c 00000430 3fff0584 40206129
3ffffde0: 00000000 3ffeec4c 00000001 4020d970
3ffffdf0: 4020df85 00000001 00000817 00000817
3ffffe00: 00000001 00000004 3fff0584 40206205
3ffffe10: 3fff0584 3ffffe40 00000000 40206794
3ffffe20: 3ffeeab8 3ffffe64 00000817 3ffee964
3ffffe30: 3ffeeab8 3ffe8524 00000817 402041f4
3ffffe40: 402105b0 00000000 000003e8 4020c854
3ffffe50: 3fff05ec 3fff0644 00000000 00000000
3ffffe60: 3ffeea80 65646e00 74682e78 80006c6d
3ffffe70: 74786574 6d74682f 89ff006c 4020ca38
3ffffe80: 00000000 3ffee9a4 3ffffeb0 4020ca68
3ffffe90: 3ffffec8 00000000 7a25000a 3ffeebec
3ffffea0: 00000000 3ffee9a4 3ffee964 40204293
3ffffeb0: 3fff04fc 000b000f 00ff0900 3ffeebec
3ffffec0: 4020ba74 3ffeeab8 3fff022c 000b000f
3ffffed0: 0a006d74 3ffeeab8 3ffe8803 4020be85
3ffffee0: 00000000 3ffee9a4 3ffee964 4020f5a6
3ffffef0: 00000000 3ffee9a4 3ffeeab8 401000e1
3fffff00: 3fff0200 3ffee9a4 3ffee964 4020446d
3fffff10: 3fffff30 80ffff20 3fff0900 0000008f
3fffff20: 80007865 00000002 3ffeea6c 00004bc6
3fffff30: 3ffee9a4 00000001 3ffe8803 4020be85
3fffff40: 00000000 00000001 0000000a 40205f5c
3fffff50: 3fffdad0 00000000 3ffee964 3ffeebec
3fffff60: 00000001 3ffee988 3ffee964 40204bff
3fffff70: 402102b8 00000000 00001388 feefeffe
3fffff80: 00000000 3fff0584 00000001 40100198
3fffff90: 3fffdad0 00000000 3ffeebac 40204ca0
3fffffa0: 3fffdad0 00000000 3ffeebac 4020da88
3fffffb0: feefeffe feefeffe 3ffe8544 40101369
<<<stack<<<

ets Jan 8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3456, room 16
tail 0
chksum 0x84
csum 0x84
va5432625
~ld

SDK:2.2.2-dev(38a443e)/Core:2.7.1=20701000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-30-g92add50/BearSSL:5c771be

Filesystem initialized.
Connecting to WLAN_43
scandone
wifi evt: 2
.....scandone
state: 0 -> 2 (b0)
.state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 9
cnt

connected with WLAN_43, channel 3
dhcp client start...
wifi evt: 0
ip:192.168.178.47,mask:255.255.255.0,gw:192.168.178.1
wifi evt: 3
.
Connected! IP address: 192.168.178.47
HTTP server started
SDK:2.2.2-dev(38a443e)/Core:2.7.1=20701000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-30-g92add50/BearSSL:5c771be
pm open,type:2 0

[d-a-v]

:DS-FAIL total=1072 asked=rem=1072 got=4294967295

Thanks for the help, that's -1.
We will investigate because a size_t can't be negative but somewhere -1 is returned, probably in upstream SD library. This datasource assert message should also be improved.

EDIT: fixed by #6979

[MagicBugsBunny]

I have to thank. My program now runs with other SD cards.
If there is something to test again, just report it.