Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

number of queries judged #1

Closed
yuhongqian opened this issue May 20, 2021 · 1 comment
Closed

number of queries judged #1

yuhongqian opened this issue May 20, 2021 · 1 comment

Comments

@yuhongqian
Copy link

Thanks for your good work!

I counted the number of queries judged in ./dataset/dl_hard-passage.qrels, and it seems that there are 48 queries instead of 50. In particular, queries 1056416 and 1103153 are not there. Am I missing anything?
@iain-mackie
Copy link
Collaborator

Thanks @yuhongqian - great catch!

It turns out that those 2 queries (1056416 & 1103153) we judged only for documents during TREC DL 2020 and not passages. This was missed when we created the DL-HARD additional judgments.

We've had our annotators create new judgements for both queries - qrels, judgments, and baseline evals have all been updated!

Any other thoughts / comments welcome! Cheers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuhongqian @iain-mackie