Refactor value based methods (vwxyzjn#102)

* Refactor value based methods

* fix test cases

* fix test cases

* refactor ddpg

* quick fix

* refactor td3

* Update lock files

* Quick fix

* Fix learning rate and target-network-frequency

* fix learning rate

* Reproduce past results: epsilon and log clipping

* Fix learning rate

* Quick fix

* quick update

* Quick update

* Refactor value tweaks for TD3 and SAC (vwxyzjn#106)

* Fixed 'optimize the midel' typo in all files

* Fixed 'optimize the midel' typo in offline scripts too

* TD3: removed DDPG's update code from the training loop

* Refactored sac_continuous, with preliminary tests working

Co-authored-by: Rousslan F.J. Dossa <[email protected]>

Author: vwxyzjn

cleanrl/apex_dqn_atari.py

@@ -853,7 +853,7 @@ def learn(args, rb, global_step, data_process_queue, data_process_back_queues, s
 
         stats_queue.put(("losses/td_loss", loss.item(), update_step+args.learning_starts))
 
-        # optimize the midel
+        # optimize the model
         optimizer.zero_grad()
         loss.backward()
         nn.utils.clip_grad_norm_(list(learn_q_network.parameters()), args.max_grad_norm)