added support for Dict action space

[JaCoderX]

Epsilon greedy policy error when generating new action from Dict Action space. #276

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[JaCoderX]

@googlebot I signed it!

[googlebot]

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

[googlebot]

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

[tfboyd]

@oars I saw you did a +2 on the internal sync, so I added you as the reviewer here, you can assign someone else.

[ebrevdo]

We're trying to tighten up the contract that Agents present to the world, in particular we don't want to allow arbitrary nested structures "so long as its only got one entry", if the agent does not support more than one action in the action space. Better if you just add a wrapper that converts the tensor to a dict.

If we start allowing this kind of soft contract, it makes it much harder for future developers to know which agents really support multiple action spaces; and makes it harder for us to add pytype to tf-agents. Sorry; will have to reject this PR.

[ebrevdo]

(a good followup PR would be to disallow nested action_specs in this agent in the first place).

[JaCoderX]

We're trying to tighten up the contract that Agents present to the world, in particular we don't want to allow arbitrary nested structures "so long as its only got one entry", if the agent does not support more than one action in the action space.

@ebrevdo, I agree with you that the code need to be simple and clean for the agents, especially if a certain agent implementation (this case DQN) is meant for just one dimensional action space.

but this PR contain 2 commits, one for the DQN and the second is for the epsilon greedy policy.
As far as I understand, the epsilon greedy policy isn't bounded only for DQN and can be used by other agents that can have a structured action space? is that commit also too case specific?

personally I just wanted to test the tf-agent framework with an external and complex gym project and see they work well together. I use the DQN tutorial just for an easy entry to tf-agent framework.

anyway thanks for helping out :)

[ebrevdo]

Ah I didn't see the changes to epsilon_greedy. That change is fine! Can you revert the change to DQN agent?

[ebrevdo]

revert this file please.

[JaCoderX]

DQN changes had been reverted

[ebrevdo]

Is there a unit test?

[tfboyd]

@JacobHanouna If there is a unit test, please reference it. If not can you add one and we can merge this can get it closed. Not to be a drag, but without a test the chances of it getting broken by some other change increases greatly. Thank you for understanding. We are so close on this and appreciate you powering through to the end.

[JaCoderX]

Tutorial: Integration with TF-Agents RL Framework

[tfboyd]

So close to high fives all around.

[tfboyd]

MERGED