Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File CDK: S3 config adapter (JSONL Options) #28134

Closed
2 tasks
clnoll opened this issue Jul 11, 2023 · 1 comment
Closed
2 tasks

File CDK: S3 config adapter (JSONL Options) #28134

clnoll opened this issue Jul 11, 2023 · 1 comment

Comments

@clnoll
Copy link
Contributor

clnoll commented Jul 11, 2023

The existing S3 connector offers a couple of options for configuring connections for JSONL file types. To ensure backwards compatibility, we'll want to update the config adapter to handle them.

In #28131, we're creating a new S3 FileBasedConfig object. This ticket involves extending that object to handle JSONL-specific options, and will also require the creation of a custom parser that handles the old options.

  • Verify that all options that we still support are appropriately mapped to the name in the file-based CDK:
    • newlines_in_values
    • unexpected_field_behavior: Options provided to Pyarrow for handling types that aren't in the schema used by a handful of connectors. This should be mapped to the new schema validation policies.

Acceptance Criteria

  • The existing JSONL config options are mapped and handled appropriately by the file-based CDK.
  • Any options that we cannot support are identified, along with the connectors that will be impacted.
@clnoll
Copy link
Contributor Author

clnoll commented Aug 9, 2023

Not needed

@clnoll clnoll closed this as completed Aug 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants