How to initialize structured sub-config (general case)?

[tbukic]

Hello,

I want to make my code modular because some parts can be reused without the whole project overhead. Here is the example:

my_package/main.py:

import hydra
from attrs import define
from hydra.core.config_store import ConfigStore
from omegaconf import MISSING, DictConfig, OmegaConf

@define
class DataSource:
    pass

@define
class SoundData(DataSource):
    name: str = "Sound"
    encoding: str = MISSING

@define
class ImageData(DataSource):
    name: str = "Image"
    size_x: int = MISSING
    size_y: int = MISSING

@define
class EnvConfig:
    name: str = "Environment"
    data_sources: dict[str, DataSource] = {}

@define
class TrainConfig:
    alpha: float = 0.1
    env: EnvConfig = MISSING

cs = ConfigStore.instance()
cs.store(name='base_config', node=TrainConfig)
cs.store(name='sound_config', node=SoundData, group='env/data_sources')
cs.store(name='image_config', node=ImageData, group='env/data_sources')

@hydra.main(config_path="../config", config_name="config", version_base=None)
def my_app(cfg : DictConfig) -> None:
    print(OmegaConf.to_yaml(cfg))

if __name__ == "__main__":
    my_app()

config/config.yaml:

defaults:
  - base_config
  - env: question
  - _self_

alpha: 0.01

config/env/question.yaml:

defaults:
  - data_sources:
    - sound1
    - image2
  - _self_

name: GitHub Example

config/env/data_sources/sound1.yaml:

# @package env.data_sources.sound1
defaults:
  - sound_config
  
name: Sound 1
encoding: wav

config/env/data_sources/image1.yaml:

# @package env.data_sources.image1
defaults:
  - image_config
  
name: instagram
size_x: 2560
size_y: 1440

config/env/data_sources/image2.yaml:

# @package env.data_sources.image2
defaults:
  - image_config

name: facebook
size_x: 1440
size_y: 900

The need is to be able to load config_path="../config/env", config_name="question" without the outer part of the config. In another scripts and notebooks, the whole config would be read.
I am aware that this can be done with reading a parent config and overriding groups manually from the code, but can it be done in a clean and direct way?
If there wasn't data_sources: dict[str, DataSource] dict in env, this would be trivial. But inheritance in dictionary value requires me to both

• register SoundData and ImageData at ConfigStore with their group specified: This is solvable by parametrizing the group.
• use package override at the start of image and sound configurations + flattened configuration. This is the problem.

Is it possible to get rid of hardcoded package override and absolute group dependencies? Is it maybe possible to tweak with the defaults list in config/env/question.yaml, so config/env/data_sources/image1.yaml can be instantiated in this form:

image1:
  name: instagram
  size_x: 2560
  size_y: 1440

Is there any other way to arrange configs above to be able to load both at the TrainConfig and EnvConfig level?