Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize writeNext Method #103

Closed
tmdgusya opened this issue Jul 23, 2022 · 3 comments
Closed

Optimize writeNext Method #103

tmdgusya opened this issue Jul 23, 2022 · 3 comments

Comments

@tmdgusya
Copy link
Contributor

tmdgusya commented Jul 23, 2022

Current writeNext function have unnecessary map method.
It cause unnecessary memory usage.

  private fun writeNext(row: List<Any?>) {
      val rowStr = row.map { field ->
          if (field == null) {
              ctx.nullCode
          } else {
              attachQuote(field.toString())
          }
      }.joinToString(ctx.delimiter.toString())
      writer.print(rowStr)
  }

Performance test

image

Result : Java Heap Space out error


Remove unnecessary map method.

  private fun writeNext(row: List<Any?>) {
      val rowStr = row.joinToString(ctx.delimiter.toString()) { field ->
          if (field == null) {
              ctx.nullCode
          } else {
              attachQuote(field.toString())
          }
      }
      writer.print(rowStr)
  }

Performance test

image

The test code was excluded for fear of slow execution for test.
If you want to measure performance test, the performance test must be isolate alone. Because of, All tests execute independently and pararell then they use some memory. So, it might be on unexpected result


TEST CODE

"does not throw java heap space exception when execute writeRows" {
    val rows = mutableListOf<String>()

    for (i in 0..100000000) {
        rows.add(i.toString())
    }

    val sequence = listOf<List<String>>(rows).asSequence()

    assertDoesNotThrow {
        csvWriter().open(testFileName) {
            writeRows(sequence)
        }
    }
}
tmdgusya added a commit to tmdgusya/kotlin-csv that referenced this issue Jul 23, 2022
@tmdgusya
Copy link
Contributor Author

In the past, an error occurred when processing 10 million data, but now 100 million data can be processed.

@tmdgusya
Copy link
Contributor Author

tmdgusya commented Jul 23, 2022

Before fixed it

image


After fixed it

image

doyaaaaaken added a commit that referenced this issue Jul 24, 2022
[#103] Optimize writeNext Method
@doyaaaaaken
Copy link
Collaborator

released in v1.5.0 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants