Open
Description
Currently each mapper executor fiber processes its file fully. This is simple and efficient approach but it creates sometimes a long tail latency when some mappers has finished but others are still processing their workload.
- Make sure the mappers incoming queue is short, possibly of length 1.
- When mappers finished they will register to take some work from other mappers.
- Need to think how we maintain "begin_shard/process/end_shard" flow with stealing mappers.
Metadata
Metadata
Assignees
Labels
No labels