Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a helper function for connecting to Snowflake. #662

Merged
merged 5 commits into from
May 15, 2024

Conversation

atheriel
Copy link
Collaborator

@atheriel atheriel commented Dec 8, 2023

This commit introduces a new odbc::snowflake() function intended to help setting up connections to Snowflake and handling auth correctly on platforms that provide Snowflake-native OAuth credentials. It generally follows the pattern established with odbc::databricks().

We've been testing this internally on Snowflake's container platform, and it works as expected.

@atheriel
Copy link
Collaborator Author

atheriel commented Apr 8, 2024

Tests are currently failing because we now have both a Snowflake.Rd and a snowflake.Rd file. I'm not sure what you'd suggest in this case, but happy to fix what's needed.

@simonpcouch
Copy link
Collaborator

Ah, interesting. From my perspective, I'd say we transition existing @rdname Snowflakes to @rdname driver-Snowflake and allow ?snowflake to head to docs on the wrapper implemented in this PR.

@atheriel atheriel force-pushed the aj-snowflake-helper branch from 2bf7ad0 to 425347d Compare April 8, 2024 18:27
@atheriel
Copy link
Collaborator Author

atheriel commented Apr 8, 2024

@simonpcouch Done.

@atheriel atheriel force-pushed the aj-snowflake-helper branch 2 times, most recently from 0cd3170 to 842b08b Compare April 8, 2024 18:28
@hadley
Copy link
Member

hadley commented Apr 8, 2024

Since we just added snowflake to our CI, should we add a couple of very basic tests?

@atheriel atheriel force-pushed the aj-snowflake-helper branch 2 times, most recently from 47173f0 to 5af2483 Compare April 8, 2024 20:44
@atheriel
Copy link
Collaborator Author

atheriel commented Apr 8, 2024

I've added a number of unit tests (and fixed some issues in the process!), but due to the design of the current CI secrets -- which supply entire connection strings -- I can't see an easy way to pass real credentials through the new odbc::snowflake() function.

@hadley
Copy link
Member

hadley commented Apr 8, 2024

@simonpcouch could you please take a look at how we're managing secrets to see if we could facilitate their use here?

@@ -35,7 +35,7 @@ getCatalogSchema <- function(conn, catalog_name = NULL, schema_name = NULL) {
#' to aid with performance, as the SQLColumns method is more performant
#' when restricted to a particular DB/schema.
#' @inheritParams DBI::dbListFields
#' @rdname Snowflake
#' @rdname driver-Snowflake
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be tempted to document them alltogether, adding a new section that details the snowflake method variations. But might be easiest for me to do this after the PR is merged.

@simonpcouch
Copy link
Collaborator

Just pushed a quick demo of using the secret in tests--fine by me to have both the PWD embedded in a connection string and the PWD itself in the GITHUB_ENV. The remaining connection information is in .github/odbc/odbc.ini.

@chendaniely
Copy link

quarto render seems to be unable to connect to snowflake. Manually running the code chunk and runnign the code in the console work just fine.
but pressing the render button seems to break:

processing file: reprex.qmd
  |...................................                 |  67% [unnamed-chunk-1]
Quitting from lines  at lines 7-20 [unnamed-chunk-1] (reprex.qmd)
Error:
! nanodbc/nanodbc.cpp:1139: 00000
[RStudio][DSI] (20032) Required setting 'UID' is not present in the connection settings. 
Backtrace:
  1. DBI::dbConnect(...)
  2. DBI::dbConnect(...)
  3. odbc (local) .local(drv, ...)
  6. odbc::dbConnect(...)
  7. odbc (local) .local(drv, ...)
  8. odbc:::OdbcConnection(...)
 10. odbc:::odbc_connect(...)
                                                                                                            
Execution halted

reprex quarto file using the cybersyn webtraffic database: https://docs.cybersyn.com/consumer-insights/web-traffic-foundation

---
title: "Untitled"
format: html
---

```{r}
library(dplyr)
library(tidyr)
library(DBI)
library(dbplyr)
library(odbc)

conn <-
  DBI::dbConnect(
    odbc::snowflake(),
    warehouse = "DEVREL_WH_LARGE",
    database = "WEB_TRAFFIC_FOUNDATION_EXPERIMENTAL",
    schema = "CYBERSYN"
  )
```

@simonpcouch
Copy link
Collaborator

simonpcouch commented Apr 24, 2024

In the above, "ambient" credentials are those set by default in the Posit Workbench build hosted at dev palm ptd:

grep("SNOWFLAKE", names(Sys.getenv()), value = TRUE)
#> [1] "SNOWFLAKE_ACCOUNT" "SNOWFLAKE_HOME"    "SNOWFLAKE_ROLE"  

If this is just a config issue, we should detect the conditions when this error will be raised and raise a more informative one before dispatching to dbConnect.OdbcConnection().

EDIT: noting that I can reproduce Daniel's issue.

@atheriel atheriel force-pushed the aj-snowflake-helper branch 4 times, most recently from aec47f3 to f7de3f4 Compare May 7, 2024 13:16
@atheriel
Copy link
Collaborator Author

atheriel commented May 7, 2024

This is ready for re-review:

  • Added additional tests, including a smoke test to make sure that odbc::snowflake() can be used to connect to the real Snowflake account in our CI system.
  • I've fixed the Quarto rendering issue by removing the call to in-process RStudio APIs; instead we parse the Workbench-generated file directly, which should be a lot more portable and future-proof.
  • CI failures seem to be related to code I haven't changed, but I may be mistaken.

@atheriel atheriel requested a review from simonpcouch May 7, 2024 13:26
Copy link
Collaborator

@simonpcouch simonpcouch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking solid! That Snowflake CI failure for the "real" connection did indeed crop up elsewhere and is uninformative due to #798—no need to address here. I do see, in R CMD check runs:

❯ checking Rd \usage sections ... WARNING
  Undocumented arguments in Rd file 'driver-Snowflake.Rd'
    ‘catalog_name’ ‘schema_name’

Could you look into those?

Submitting review as "Comment" rather than "Approve" only so as to defer to Hadley on the timeline to merge this one. :)

@atheriel atheriel force-pushed the aj-snowflake-helper branch from f7de3f4 to 46e0f71 Compare May 7, 2024 15:44
@atheriel
Copy link
Collaborator Author

atheriel commented May 7, 2024

The warning about missing docs is unrelated to this PR but I fixed it anyway to try and make CI happy :)

@simonpcouch simonpcouch requested a review from hadley May 7, 2024 18:05
@atheriel atheriel force-pushed the aj-snowflake-helper branch from 46e0f71 to beb23cd Compare May 7, 2024 20:01
@garrettgman
Copy link

cc @skaltman

Copy link
Collaborator

@simonpcouch simonpcouch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GHA failure:

── Failed tests ────────────────────────────────────────────────────────────────
Error ('test-driver-snowflake.R:2:3'): can connect to snowflake
Error in `test_con("SNOWFLAKE")`: ! ODBC failed with error 00000 from [Snowflake][Snowflake].
*  Password not found.
i From 'nanodbc/nanodbc.cpp:1147'.

...is due to the relevant repository secret not being available and is fine to ignore. Game to merge with your thumbs-up, @atheriel—just let us know whether you want to address the warning -> error conversion review comment in this PR or file a separate issue to make that happen.

atheriel added 3 commits May 15, 2024 10:52
This commit introduces a new odbc::snowflake() function intended to help
setting up connections to Snowflake and handling auth correctly on
platforms that provide Snowflake-native OAuth credentials.

Due to a naming clash, the Snowflake.Rd file has also been renamed
driver-Snowflake.Rd.

Because we want to smoke test arguments to the snowflake() function
directly rather than a full connection string, this commit also exposes
a new environment variable with the GitHub secret for the Snowflake
password.

Unit tests are included.

Signed-off-by: Aaron Jacobs <aaron.jacobs@rstudio.com>
Signed-off-by: Aaron Jacobs <aaron.jacobs@rstudio.com>
Signed-off-by: Aaron Jacobs <aaron.jacobs@rstudio.com>
@atheriel atheriel force-pushed the aj-snowflake-helper branch from 8a1df83 to 16ef158 Compare May 15, 2024 14:54
@atheriel atheriel merged commit 889242b into r-dbi:main May 15, 2024
16 checks passed
@atheriel atheriel deleted the aj-snowflake-helper branch May 15, 2024 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants