Skip to content

Work stealing mappers #29

Open
Open
@romange

Description

@romange

Currently each mapper executor fiber processes its file fully. This is simple and efficient approach but it creates sometimes a long tail latency when some mappers has finished but others are still processing their workload.

  1. Make sure the mappers incoming queue is short, possibly of length 1.
  2. When mappers finished they will register to take some work from other mappers.
  3. Need to think how we maintain "begin_shard/process/end_shard" flow with stealing mappers.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions