Added Leaderboard Entry for PPO #580
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
Codecov Report
@@ Coverage Diff @@
## development #580 +/- ##
===============================================
- Coverage 88.09% 87.82% -0.27%
===============================================
Files 114 114
Lines 6693 6693
===============================================
- Hits 5896 5878 -18
- Misses 797 815 +18
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Woah, cool! I'd love to see a PPO entry added to the leaderboard, and I like your neat trick about running a guided random search after.
However, I've not been able to validate your results. The command lines in your results.csv file are malformed (you need to add input.bc -o output.bc to the end of each), and, after fixing that, I was then unable to replicate the rewards achieved on my local linux box running CompilerGym 0.2.1.
$ python -m compiler_gym.bin.validate --env=llvm-ic-v0 results.csv 2>/dev/null
❌ cbench-v1/adpcm Expected reward 1.0083798882681567 but received reward 0.994413407821229
❌ cbench-v1/stringsearch2 Expected reward 0.9962686567164181 but received reward 0.9440298507462687
❌ cbench-v1/rijndael Expected reward 1.1092372556535077 but received reward 1.0785741663472594
❌ cbench-v1/ispell Expected reward 1.028289936664321 but received reward 1.0216748768472907
❌ cbench-v1/qsort Expected reward 1.136222910216718 but received reward 1.0773993808049536
...
Did you use the compiler_gym.leader.llvm_instcount.eval_llvm_instcount_policy helper to generate your results?
Cheers,
Chris
| @@ -0,0 +1,98 @@ | |||
| <!-- To submit a leaderboard entry please fill in this document follow the | |||
There was a problem hiding this comment.
Nitpick: please strip the trailing whitespace in this file
README.md
Outdated
| | Patrick Hesse | DQN (N=4000, H=10) | [write-up](leaderboard/llvm_instcount/dqn/README.md), [results](leaderboard/llvm_instcount/dqn/results-instcountnorm-H10-N4000.csv) | 2021-06 | 91.018s | 1.029× | | ||
| | Jiadong Guo | Tabular Q (N=2000, H=5) | [write-up](leaderboard/llvm_instcount/tabular_q/README.md), [results](leaderboard/llvm_instcount/tabular_q/results-H5-N2000.csv) | 2021-04 | 694.105 | 0.988× | | ||
| | --- | --- | --- | --- |-----------------|------------------------------| | ||
| | Leibniz University| PPO + Guided Search | [write-up](leaderboard/llvm_instcount/ppo/README.md), [results](leaderboard/llvm_instcount/random_search/results.csv) | 2022-02 | 69.821s | **1.070×** | |
There was a problem hiding this comment.
It's entirely up to you, but if you'd prefer to list your three names here rather than the institution, that's totally fine :)
| <!-- Whether you have released the source code of your approach, yes/no. If | ||
| yes, please state the license. --> | ||
| The source code is available as Open-Source: | ||
| https://github.com/xtremey/ppo_compiler_gym |
There was a problem hiding this comment.
For us to accept this as open source your code must have a license (further reading here). Please add a LICENSE file to the linked repository, and state the name of the license here, e.g. "Is the approach Open Source? Yes, under Apache v2. Source code is here: ...".
Also, please link to an exact commit, such as: https://github.com/xtremey/ppo_compiler_gym/tree/99a677eaefdb33eddd01299aa683f8d0c14d8643
this enables you to make changes to your code without preventing people from reproducing your results here.
| **Is the policy deterministic?:** No | ||
| <!-- Whether the (state, action) policy is deterministic, yes/no. --> | ||
|
|
||
| ## Description |
There was a problem hiding this comment.
If I understand correctly, the PPO agent was trained on the same cbench programs that are used for the test set. Is that right? If so, this is an important distinction to make to make because it means that the agent is not being evaluated on its ability to generalize to an unseen program. Could you please add a few more details about the training process?
Since you are releasing your code, consider adding the hardware you used and commands needed to replicate the work. Take a look at this for an example: https://github.com/facebookresearch/CompilerGym/blob/development/leaderboard/llvm_instcount/dqn/README.md#experimental-setup
|
Thanks for the detailed feedback! The mistake was totally on our side. We used our own code to generate the results.csv file, but had a bug in the process. We updated the Readme file and our own repository according to your feedback. If there is anything still missing or if anything could be improved please let us know. Yannik |
ChrisCummins
approved these changes
Feb 23, 2022
There was a problem hiding this comment.
Hi Yannik,
Thanks for the quick fixes, and impressive results! I have reproduced your results on my end. Congrats on the top spot ⭐
$ python -m compiler_gym.bin.validate --env=llvm-ic-v0 results.csv
✅ cbench-v1/adpcm 1.0084
✅ cbench-v1/rijndael 1.1092
✅ cbench-v1/stringsearch2 0.9963
✅ cbench-v1/ispell 1.0283
✅ cbench-v1/lame 1.0789
✅ cbench-v1/crc32 1.0000
✅ cbench-v1/dijkstra 0.9949
✅ cbench-v1/sha 1.5251
✅ cbench-v1/qsort 1.1362
✅ cbench-v1/bitcount 1.0199
✅ cbench-v1/patricia 1.0111
✅ cbench-v1/ghostscript 1.0296
✅ cbench-v1/stringsearch 1.0164
✅ cbench-v1/blowfish 1.1018
✅ cbench-v1/gsm 1.1296
✅ cbench-v1/susan 1.0269
✅ cbench-v1/bzip2 1.2212
✅ cbench-v1/tiffdither 1.0489
✅ cbench-v1/tiffmedian 1.0459
✅ cbench-v1/tiff2rgba 1.0461
✅ cbench-v1/tiff2bw 1.0446
✅ cbench-v1/jpeg-d 1.0493
✅ cbench-v1/jpeg-c 1.0468
----------------------------------------------------
Number of validated results: 23 of 23
Mean walltime per benchmark: 69.821s (std: 11.741s)
Geometric mean IrInstructionCountOz: 1.070 (std: 0.110)
python -m compiler_gym.bin.validate --env=llvm-ic-v0 results.csv 2121.22s user 124.60s system 1446% cpu 2:35.21 total
Cheers,
Chris
ChrisCummins
added a commit
that referenced
this pull request
May 24, 2022
This release adds a new compiler environment, new APIs, and a suite of backend improvements to improve the flexibility of CompilerGym environments. Many thanks to code contributors: @sogartar, @KyleHerndon, @SoumyajitKarmakar, @uduse, and @anthony0727! Highlights of this release include: - [mlir] Began work on a new environment for matrix multiplication using MLIR ([#652](#652), thanks @KyleHerndon and @sogartar!). Note this environment is not yet included in the pypi package and must be [compiled from source](https://github.com/facebookresearch/CompilerGym/blob/development/INSTALL.md#building-from-source-with-cmake). - [llvm] Added a new `env.benchmark_from_clang_invocation()` method ([#577](#577)) that can be used for constructing LLVM environment automatically from C/C++ compiler invocations. This makes it much easier to integrate CompilerGym with your existing build scripts. - Added three new wrapper classes: `Counter`, that provides op counts for analysis ([#683](#683)); `SynchronousSqliteLogger`, that provides logging of environment interactions to a relational database ([#679](#679)), and `ForkOnStep` that provides an `undo()` operation ([#682](#682)). - Added `reward_space` and `observation_space` parameters to `env.reset()` ([#659](#659), thanks @SoumyajitKarmakar!) This release includes a number of improvements to the backend APIs that make it easier to write new CompilerGym environments: - Refactored the backend to make `CompilerEnv` an abstract interface, and `ClientServiceCompilerEnv` the concrete implementation of this interface. This enables new environments to be implemented without using gRPC ([#633](#633), thanks @sogartar!). - Extended the support for different types of action and observation spaces ([#641](#641), [#643](#643), thanks @sogartar!), including new `Permutation` and `SpaceSequence` spaces ([#645](#645), thanks @sogartar!).. - Added a new `disk/` subdirectory to compiler service's working directories, which is symlinked to an on-disk location for devices which support in-memory working directories. This fixes a bug with leftover temporary directories from LLVM ([#672](#672)). This release also includes numerous bug fixes and improvements, many of which were reported or fixed by the community. For example, fixing a bug in cache file locations ([#656](#656), thanks @uduse!), and a missing flag definition in example code ([#684](#684), thanks @anthony0727!). **Full Changelog**: v0.2.3...v0.2.4 This release brings in deprecating changes to the core `env.step()` routine, and lays the groundwork for enabling new types of compiler optimizations to be exposed through CompilerGym. Many thanks to code contributors: @mostafaelhoushi, @sogartar, @KyleHerndon, @uduse, @parthchadha, and @xtremey! Highlights of this release include: - Added a new `TextSizeInBytes` observation space for LLVM ([#575](#575)). * Added a new PPO leaderboard entry ([#580](#580). Thanks @xtremey! - Fixed a bug in which temporary directories created by the LLVM environment were not cleaned up ([#592](#592)). - **[Backend]** The function `createAndRunCompilerGymService` now returns an int, which is the exit return code ([#592](#592)). - Improvements to the examples documentation ([#548](#548)) and FAQ ([#586](#586)) Deprecations and breaking changes: - `CompilerEnv.step` no longer accepts a list of actions ([#627](#627)). A new method, `CompilerEnv.multistep` provides this functionality. This is to provide compatibility with environments whose action spaces are lists. To update your code, replace any calls to `env.step()` which take a list of actions to use `env.multistep()`. Thanks @sogartar! - The arguments `observations` and `rewards` to `step()` have been renamed `observation_spaces` and `reward_spaces`, respectively ([#627](#627)). - `Reward.id` has been renamed `Reward.name` ([#565](#565), [#612](#612)). Thanks @parthchadha! * The backend protocol buffer schema has been updated to natively support more types of observation and action, and to support nested spaces ([#531](#531)). Thanks @sogartar!
ChrisCummins
added a commit
that referenced
this pull request
May 24, 2022
This release adds a new compiler environment, new APIs, and a suite of backend improvements to improve the flexibility of CompilerGym environments. Many thanks to code contributors: @sogartar, @KyleHerndon, @SoumyajitKarmakar, @uduse, and @anthony0727! Highlights of this release include: - [mlir] Began work on a new environment for matrix multiplication using MLIR ([#652](#652), thanks @KyleHerndon and @sogartar!). Note this environment is not yet included in the pypi package and must be [compiled from source](https://github.com/facebookresearch/CompilerGym/blob/development/INSTALL.md#building-from-source-with-cmake). - [llvm] Added a new `env.benchmark_from_clang_invocation()` method ([#577](#577)) that can be used for constructing LLVM environment automatically from C/C++ compiler invocations. This makes it much easier to integrate CompilerGym with your existing build scripts. - Added three new wrapper classes: `Counter`, that provides op counts for analysis ([#683](#683)); `SynchronousSqliteLogger`, that provides logging of environment interactions to a relational database ([#679](#679)), and `ForkOnStep` that provides an `undo()` operation ([#682](#682)). - Added `reward_space` and `observation_space` parameters to `env.reset()` ([#659](#659), thanks @SoumyajitKarmakar!) This release includes a number of improvements to the backend APIs that make it easier to write new CompilerGym environments: - Refactored the backend to make `CompilerEnv` an abstract interface, and `ClientServiceCompilerEnv` the concrete implementation of this interface. This enables new environments to be implemented without using gRPC ([#633](#633), thanks @sogartar!). - Extended the support for different types of action and observation spaces ([#641](#641), [#643](#643), thanks @sogartar!), including new `Permutation` and `SpaceSequence` spaces ([#645](#645), thanks @sogartar!).. - Added a new `disk/` subdirectory to compiler service's working directories, which is symlinked to an on-disk location for devices which support in-memory working directories. This fixes a bug with leftover temporary directories from LLVM ([#672](#672)). This release also includes numerous bug fixes and improvements, many of which were reported or fixed by the community. For example, fixing a bug in cache file locations ([#656](#656), thanks @uduse!), and a missing flag definition in example code ([#684](#684), thanks @anthony0727!). **Full Changelog**: v0.2.3...v0.2.4 This release brings in deprecating changes to the core `env.step()` routine, and lays the groundwork for enabling new types of compiler optimizations to be exposed through CompilerGym. Many thanks to code contributors: @mostafaelhoushi, @sogartar, @KyleHerndon, @uduse, @parthchadha, and @xtremey! Highlights of this release include: - Added a new `TextSizeInBytes` observation space for LLVM ([#575](#575)). * Added a new PPO leaderboard entry ([#580](#580). Thanks @xtremey! - Fixed a bug in which temporary directories created by the LLVM environment were not cleaned up ([#592](#592)). - **[Backend]** The function `createAndRunCompilerGymService` now returns an int, which is the exit return code ([#592](#592)). - Improvements to the examples documentation ([#548](#548)) and FAQ ([#586](#586)) Deprecations and breaking changes: - `CompilerEnv.step` no longer accepts a list of actions ([#627](#627)). A new method, `CompilerEnv.multistep` provides this functionality. This is to provide compatibility with environments whose action spaces are lists. To update your code, replace any calls to `env.step()` which take a list of actions to use `env.multistep()`. Thanks @sogartar! - The arguments `observations` and `rewards` to `step()` have been renamed `observation_spaces` and `reward_spaces`, respectively ([#627](#627)). - `Reward.id` has been renamed `Reward.name` ([#565](#565), [#612](#612)). Thanks @parthchadha! * The backend protocol buffer schema has been updated to natively support more types of observation and action, and to support nested spaces ([#531](#531)). Thanks @sogartar!
ChrisCummins
added a commit
that referenced
this pull request
May 24, 2022
This release adds a new compiler environment, new APIs, and a suite of backend improvements to improve the flexibility of CompilerGym environments. Many thanks to code contributors: @sogartar, @KyleHerndon, @SoumyajitKarmakar, @uduse, and @anthony0727! Highlights of this release include: - [mlir] Began work on a new environment for matrix multiplication using MLIR ([#652](#652), thanks @KyleHerndon and @sogartar!). Note this environment is not yet included in the pypi package and must be [compiled from source](https://github.com/facebookresearch/CompilerGym/blob/development/INSTALL.md#building-from-source-with-cmake). - [llvm] Added a new `env.benchmark_from_clang_invocation()` method ([#577](#577)) that can be used for constructing LLVM environment automatically from C/C++ compiler invocations. This makes it much easier to integrate CompilerGym with your existing build scripts. - Added three new wrapper classes: `Counter`, that provides op counts for analysis ([#683](#683)); `SynchronousSqliteLogger`, that provides logging of environment interactions to a relational database ([#679](#679)), and `ForkOnStep` that provides an `undo()` operation ([#682](#682)). - Added `reward_space` and `observation_space` parameters to `env.reset()` ([#659](#659), thanks @SoumyajitKarmakar!) This release includes a number of improvements to the backend APIs that make it easier to write new CompilerGym environments: - Refactored the backend to make `CompilerEnv` an abstract interface, and `ClientServiceCompilerEnv` the concrete implementation of this interface. This enables new environments to be implemented without using gRPC ([#633](#633), thanks @sogartar!). - Extended the support for different types of action and observation spaces ([#641](#641), [#643](#643), thanks @sogartar!), including new `Permutation` and `SpaceSequence` spaces ([#645](#645), thanks @sogartar!).. - Added a new `disk/` subdirectory to compiler service's working directories, which is symlinked to an on-disk location for devices which support in-memory working directories. This fixes a bug with leftover temporary directories from LLVM ([#672](#672)). This release also includes numerous bug fixes and improvements, many of which were reported or fixed by the community. For example, fixing a bug in cache file locations ([#656](#656), thanks @uduse!), and a missing flag definition in example code ([#684](#684), thanks @anthony0727!). **Full Changelog**: v0.2.3...v0.2.4 This release brings in deprecating changes to the core `env.step()` routine, and lays the groundwork for enabling new types of compiler optimizations to be exposed through CompilerGym. Many thanks to code contributors: @mostafaelhoushi, @sogartar, @KyleHerndon, @uduse, @parthchadha, and @xtremey! Highlights of this release include: - Added a new `TextSizeInBytes` observation space for LLVM ([#575](#575)). * Added a new PPO leaderboard entry ([#580](#580). Thanks @xtremey! - Fixed a bug in which temporary directories created by the LLVM environment were not cleaned up ([#592](#592)). - **[Backend]** The function `createAndRunCompilerGymService` now returns an int, which is the exit return code ([#592](#592)). - Improvements to the examples documentation ([#548](#548)) and FAQ ([#586](#586)) Deprecations and breaking changes: - `CompilerEnv.step` no longer accepts a list of actions ([#627](#627)). A new method, `CompilerEnv.multistep` provides this functionality. This is to provide compatibility with environments whose action spaces are lists. To update your code, replace any calls to `env.step()` which take a list of actions to use `env.multistep()`. Thanks @sogartar! - The arguments `observations` and `rewards` to `step()` have been renamed `observation_spaces` and `reward_spaces`, respectively ([#627](#627)). - `Reward.id` has been renamed `Reward.name` ([#565](#565), [#612](#612)). Thanks @parthchadha! * The backend protocol buffer schema has been updated to natively support more types of observation and action, and to support nested spaces ([#531](#531)). Thanks @sogartar!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a leaderboard entry for guided search using the action probabilities of a Model trained by using Proximal Policy Optimization (PPO).
Our full implementation can be found here: https://github.com/xtremey/ppo_compiler_gym
This method achieved a geometric mean of 1.070, which scores first place on the current leaderboard. Additionally, the walltime of the search is much lower than of other search algorithms on the leaderboard.
What is in this pull request?