Multiple HTTP writes fail

[amotl]

Dear Giorgio,

Introduction

Our background is that we are currently building a test suite based on CPython/Pytest for invoking MicroPython programs, see also micropython/micropython#5786 and micropython/micropython#4955. It works pretty good so far. Thanks again for Mocket, it helped us tremendously already.

Problem

So, after successfully creating a Pytest LoRa socket mock fixture, we are now struggling creating a corresponding thing for the urequests module.

The gist is that urequests will work like that:

sock.connect(address[-1])
sock.write("%s %s HTTP/1.0\r\n" % (method, path))
sock.write("Host: %s\r\n" % host)

which makes Mocket only see the first line:

b'POST /api/data HTTP/1.0\r\n'

so that it will croak with the following exception.

Exception

>       _, self.body = decode_from_bytes(data).split('\r\n\r\n', 1)
E       ValueError: not enough values to unpack (expected 2, got 1)
.venv3/lib/python3.8/site-packages/mocket/mockhttp.py:23: ValueError

Reproduction

I have been able to create a repro using pytest.
https://gist.github.com/amotl/015ef6b336db55128798d7f1a9a67dea

Thoughts

I believe @sjaensch observed this within #66 already and apparently #67 didn't fix it yet.

Thanks already for looking into this!

With kind regards,
Andreas.

[mindflayer]

I had a look at your gist. So, at the moment mocket supports multiple read operations, but definitely does not support multiple write ones. I can see if that's an easy task to implement, but I cannot promise anything.
Issue #67 was definitely about something different.

[amotl]

I can see if that's an easy task to implement, but I cannot promise anything.

Thanks!

In that case, Mocket should honor the semantics of HTTP that a Content-Length header is required and read that many bytes from the HTTP request body before passing the request down to its handler.

When the client signals a Connection: close header, the socket connection should be closed afterwards.

[mindflayer]

Good morning, yesterday night I made a few changes which fixed your issue, but broke other test cases. I hope I'll be able to sort everything out.

[amotl]

Hi Giorgio,

I've also spent some time on further researching this issue last night. In the meanwhile, I am not so sure that urequests acts standards-compliant at all, especially when using standard (CPython) socket semantics.

The usocket documentation says

For efficiency and consistency, socket objects in MicroPython implement a stream (file-like) interface directly. In CPython, you need to convert a socket to a file-like object using makefile() method. This method is still supported by MicroPython (but is a no-op), so where compatibility with CPython matters, be sure to use it.

That's actually the reason usocket already provides the .write() method without further ado.

From looking at this, I figure that, when invoked on MicroPython, usocket might apply some buffering on its stream interface which will make the bytes written to it being emitted as a request en bloc to the HTTP server. However, as far as I know, HTTP messages must be sent contiguous. While there are mechanisms for streaming data in HTTP, they usually only apply to the request or response bodies.

So, I am inclined to believe that talking to a HTTP server like implemented through urequests is not standards-compliant. However, it works when a) when run on MicroPython and b) when using a real HTTP server. I also did a quick test using pytest-httpserver and it works flawlessly also when invoked on CPython.

The socket mocking machinery applied by the other popular contender httpretty didn't help either. The module will even completely reject data sent to a socket using the .send() method - apparently it will only accept .sendall(), sending all data contiguous.

So, while I appreciate your investigations, we might just recommend using pytest-httpserver to others. If you feel solving this within python-mocket will add too much complexity or weirdness, feel free to close this issue as [wontfix].

With kind regards,
Andreas.

[mindflayer]

Hi @amotl, I have a PR with a few changes which make your tests pass but I could not spend enough time to see if they can work with all the rest.
In case you'd like to try, it's about fixing PR #113.

[mindflayer]

For now, I am flagging it as [wontfix] and closing the issue. Feel free to reopen it in case you have an update.

[mindflayer]

Hi @amotl, I've got good news for you, Micropython's folks.

[mindflayer]

Here you are: pip install mocket==3.9.1

https://pypi.org/project/mocket/3.9.1/

[mindflayer]

@amotl This is your test added to mocket's suite: https://github.com/mindflayer/python-mocket/blob/master/tests/main/test_http.py#L327