feat: Python Verifier implementation #1702
Conversation
Wendong-Fan
changed the title
|
Should be good now. |
There was a problem hiding this comment.
Thanks @hallerite , left some comments, we also need to add test code
camel/verifiers/python_verifier.py
Outdated
| self, | ||
| python_version: str = "python3", | ||
| timeout: Optional[float] = 30.0, | ||
| required_packages: Optional[List[str]] = None, |
There was a problem hiding this comment.
should we let user use interpreters integrated in camel for code execution? https://github.com/camel-ai/camel/tree/master/camel/interpreters
There was a problem hiding this comment.
I think this would be a good feature request for later. As of now, we can just leave it like this so we can accelerate the Loong project.
|
Tests added. |
There was a problem hiding this comment.
Thanks @hallerite , added one more commit af71fd4 to fix comment below, feel free to check
camel/verifiers/python_verifier.py
Outdated
| Defaults to an empty list. | ||
| """ | ||
| super().__init__(timeout=timeout) | ||
| self.python_version = python_version |
camel/verifiers/python_verifier.py
Outdated
| venv.create(self.venv_path, with_pip=True) | ||
| logger.info(f"Virtual environment created at {self.venv_path}") | ||
|
|
||
| venv_pip = os.path.join(self.venv_path, "bin", "pip") |
There was a problem hiding this comment.
path for different system could be different
camel/verifiers/python_verifier.py
Outdated
| if process.returncode == 0: | ||
| # If ground truth is provided, compare it with the result | ||
| if result.ground_truth is not None: | ||
| if output_result == str(result.ground_truth).strip(): |
There was a problem hiding this comment.
this could be problematic if there are whitespace differences or line ending differences
There was a problem hiding this comment.
This is an area that will have to be improved in general. We will need to use math-verify or extractors for more comprehensive semantic matching.
camel/verifiers/python_verifier.py
Outdated
|
|
||
| def __init__( | ||
| self, | ||
| python_version: str = "python3", |
|
Thank you @Wendong-Fan |
Description
This PR adds a first implementation of a PythonVerifier that is reusable for anything that is only dependent on Python packages.
This PR also includes a example usage, showing the verifier in action.
Checklist
Go over all the following points, and put an
xin all the boxes that apply.Fixes #issue-numberin the PR description (required)pyproject.tomlandpoetry.lock