Use pytest servers #8

dtrifiro · 2022-08-18T16:47:11Z

closes #2

efiop · 2022-08-18T17:06:15Z

dvc_gs/tests/fixtures.py

 @pytest.fixture
-def make_gs(gs_creds):
-    def _make_gs():
-        return GCP(GCP.get_url(), gs_creds)
+def make_gs(request):
+    if os.getenv("DVC_GS_NO_EMULATOR", None):
+
+        def _make_gs():
+            return GCP(GCP.get_url(), "")
+
+    else:
+        gcs_path = request.getfixturevalue("gcs_path")
+        fake_gcs_server = request.getfixturevalue("fake_gcs_server")
+
+        def _make_gs():
+            return FakeGCP(
+                str(gcs_path).replace("gcs://", "gs://"),
+                endpoint_url=fake_gcs_server,
+            )

    return _make_gs


Let's do similar to dvc-s3: have gs fixture that is always using fake server and have a real_gs fixture that uses real gs. Tests should use gs(fake) always, but if one would need to debug stuff, he could use real_gs manually.

setup.cfg

.github/workflows/tests.yaml

setup.cfg

dtrifiro · 2022-08-25T20:12:11Z

We were getting errors when setting up pytest-servers fixtures because of -n=auto: we were trying to setup multiple session-scoped fixtures at once. I'll work out something in pytest-servers in the future to support (see iterative/pytest-servers#50), but for now I got rid of -nauto in gha.

Note that test_import is still broken, and this will be taken care of in #9

efiop · 2022-08-26T13:04:55Z

@dtrifiro Not sure I understand, so is this ready for review/merge? Seems like there are still some cloud credentials issues in the log?

dtrifiro · 2022-08-26T13:39:25Z

The cloud credentials messages in the logs are just debug messages, and are related to GCSFileSystem trying to get credentials from various sources (including environment) before giving up, see
https://github.com/fsspec/gcsfs/blob/fef6c0b098342a95990fd5aad8c3353d257f8d1b/gcsfs/credentials.py#L225-L242.

Note that these credentials are not used to access storage.googleapis.com because endpoint_url is overriden.

We could get rid of those warnings by setting GOOGLE_APPLICATION_CREDENTIALS to the path of a service account private key json, something similar to what we do with S3 with fake_creds_file. Trying to decide whether it's better to have a fixture for that here or in pytest-servers, although I'm opting for the former.

efiop · 2022-08-26T16:37:16Z

@dtrifiro So this is ready, right? Feel free to merge then.

dtrifiro requested a review from efiop August 18, 2022 16:47

dtrifiro mentioned this pull request Aug 18, 2022

Fix import test #9

Merged

dtrifiro marked this pull request as draft August 18, 2022 16:54

dtrifiro marked this pull request as ready for review August 18, 2022 16:54

This comment was marked as outdated.

Sign in to view