Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use the same library when Reading and Writing CSV #30

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

abendada
Copy link

  • Added Escape Character
  • Use CSVPrinter when writing CSV Files

@falaki
Copy link
Member

falaki commented Mar 25, 2015

@abendada Would you add tests for this and make it pass?

@abendada
Copy link
Author

I added a test for escape
The test consist of:

  • reading a csv file with "" as escape character
  • check the parsed content
  • save the dataFrame as a temp file without providing an escape character the default (") is used if necessary
  • check that temp file is well formed
  • read the temp file and check that the content well parsed

Note: quote and escape should have Character as type so nullable.

@@ -33,7 +33,8 @@ case class CsvRelation protected[spark] (
location: String,
useHeader: Boolean,
delimiter: Char,
quote: Char,
quote: Character,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use Char

@falaki
Copy link
Member

falaki commented Mar 26, 2015

@abendada left a few comments. Would you please also update the README for the new escape character option, and rebase this? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants