SAC doesn't work

[vadim0x60]

Problem Description

Soft Actor-Critic (sac_continuous_action.py) just doesn't work. I have tried different envs, it doesn't matter

Current Behavior

python cleanrl/sac_continuous_action.py --env-id 'MountainCarContinuous-v0'

Traceback (most recent call last):
  File "/home/320012346/cleanrl/cleanrl/sac_continuous_action.py", line 188, in <module>
    actor = Actor(envs).to(device)
  File "/home/320012346/cleanrl/cleanrl/sac_continuous_action.py", line 120, in __init__
    self.action_scale = torch.FloatTensor((env.action_space.high - env.action_space.low) / 2.0)
AttributeError: 'Tuple' object has no attribute 'high'

Expected Behavior

python cleanrl/sac_continuous_action.py --env-id 'MountainCarContinuous-v0'

global_step=1, episodic_return=-0.0020000000949949026
global_step=7, episodic_return=-0.0060000005178153515
global_step=8, episodic_return=-0.0010000000474974513
global_step=25, episodic_return=-0.017000000923871994

Possible Solution

Not sure. I think gym.vector.SyncVectorEnv handles action spaces incorrectly. The docstring says it chooses the action space of the first of the subenvironments, but in reality it's not a gym.space, it's a tuple.

[vwxyzjn]

Thanks for raising the issue. This seems to be a simple fix to just replace the action_space with single_action_space. @dosssman would you mind taking a look at this? Does this impact our benchmark results?

[vwxyzjn]

Wait, I just tested the code and it works just fine.

(cleanrl--brdzvln-py3.9) ➜  cleanrl git:(master) ✗ python cleanrl/sac_continuous_action.py --env-id 'MountainCarContinuous-v0'
/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/torch/utils/tensorboard/__init__.py:6: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if not hasattr(tensorboard, '__version__') or LooseVersion(tensorboard.__version__) < LooseVersion('1.15'):
/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/gym/core.py:172: DeprecationWarning: WARN: Function `env.seed(seed)` is marked as deprecated and will be removed in the future. Please use `env.reset(seed=seed) instead.
  deprecation(
global_step=998, episodic_return=-33.523799896240234
global_step=1997, episodic_return=-32.330013275146484
global_step=2996, episodic_return=-34.44511413574219
global_step=3995, episodic_return=-34.106590270996094
global_step=4994, episodic_return=-32.482479095458984
SPS: 1708
^CTraceback (most recent call last):
  File "/Users/costahuang/go/src/github.com/vwxyzjn/cleanrl/cleanrl/sac_continuous_action.py", line 277, in <module>
    actor_loss.backward()
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
KeyboardInterrupt

Have you installed dependencies via poetry install?

[dosssman]

Also working as expected on my side.
Maybe a dependency problem ? Or an older branch ?

[vwxyzjn]

Yeah I added a checklist item in the issue item saying please make sure to install dependencies correctly via poetry install. See #180

[vadim0x60]

I have run git pull and poetry install just to make sure I have the right versions of everything, but the problem persists. I will try to debug what it is on my system that causes the issue and will get back to you with my findings

[vadim0x60]

OK, I got it, I downgraded gym to 0.21 because there is a regression in gym 0.23. I don't understand the path from a wrong version of gym to this error, but with gym 0.23 everything works as expected. Sorry for not doing enough research prior to submitting the issue.

[vwxyzjn]

Not at all. Thanks for submitting the issue. Glad it’s working for you :)