Add missing RetriBERT tokenizer tests

[mpoemsl]

What does this PR do?

Addresses issue #16627.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@LysandreJik
@SaulLu

Notes

1. There was no folder tests/retribert/ yet, so I created one and put an __init__.py in it. Is there anything else I have to do for these tests to get picked up by CI?

2. RetriBertTokenizer is identical to BertTokenizer, so I mostly just duplicated BertTokenizationTest. Is that fine or should I rather a) write new tests from scratch or b) figure out a way to reuse the code in BertTokenizationTest?

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[SaulLu]

Thank you very much for the addition and the explanations regarding your choices! I agree with those 😃

I've just left 2 suggestions of change in comment

[mpoemsl]

Hi @SaulLu, can you give me some guidance on how to proceed?

The CI is throwing the following error:
Make sure the names of these test files match the name of the module or utils they are testing, or adapt the constant `SPECIAL_MODULE_TO_TEST_MAP` in `utils/tests_fetcher.py` to add them. If your test file is triggered separately and is not supposed to be run by the regular CI, add it to the `EXPECTED_TEST_FILES_NEVER_TOUCHED` constant instead.

However, I think the naming is correct (test_tokenization_retribert.py tests tokenization_retribert.py) and adding it to neither constants makes sense to me. I'm also unable to reproduce this locally with pytest. Do you have any idea what triggered this?

[SaulLu]

The CI error seems to me to come from the fact that 3 days ago there was a re-organisation of the test folder (#17034).

To solves this, I suggest to1) merge the latest changes to main in your branch and 2) move the tests you added to conform to the new organisation (tests/retribert -> tests/models/retribert).

Keep me updated! 😄

[mpoemsl]

Hi @SaulLu, thank you very much for the reply. I think I've got it now! Can you take another look?

[SaulLu]

Look perfect to me! Thanks for the contribution @mpoemsl 🚀

[mpoemsl]

My pleasure! I would like to keep contributing and I was wondering if you could help me with a question related to that @SaulLu.

I noticed that the RetriBERT model itself is missing test files as well and I would like to write those. How do I make sure that no one else is writing them concurrently? Do I open an issue or perhaps a WIP pull request? I have already checked that there currently is no open issue or pull request related to this.