Default ESQL data partitioning to DOC

[dnhatn]

The DOC data partitioning typically outperforms SEGMENT data partitioning. Since we've capped the number of concurrent operators to the number of threads in the ESQL worker as a safeguard, we should consider changing the default data partitioning from SEGMENT to DOC.

Relates #99189

[elasticsearchmachine]

Pinging @elastic/es-ql (Team:QL)

[elasticsearchmachine]

Pinging @elastic/elasticsearch-esql (:Query Languages/ES|QL)

[dnhatn]

Thanks, Nik!

[ChrisHegarty]

LGTM

[jpountz]

I haven't run benchmarks to confirm, but in theory this change makes latency potentially better but throughput worse until we add support for doc partitioning to Lucene: apache/lucene#9721. This is because some queries like range queries have no option but to evaluate the filter across the entire segment anyway. If you have a segment that you want to partition in, say, 10 partitions and evaluate a range filter across 10 threads (one per partition), each thread will actually evaluate the filter against the entire segment - not what we want.

Given how range queries on the @timestamp field are common, my preference would be to wait until we address this Lucene issue before we enable doc partitioning by default in ES|QL.

[dnhatn]

@jpountz Thank you for the feedback. I will take a look at the issue that you linked and leave this PR unmerged as you suggested.

[costin]

LGTM

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)