Skip to content

Source of validation accuracy in zero-cost case #133

Open
@jr2021

Description

@jr2021

In the zero-cost branch optimizers Npenas and Bananas, the validation accuracy of architectures is being queried from the zero-cost-benchmark as follows:

model.accuracy = self.zc_api[str(model.arch_hash)]['val_accuracy']

The question is, whether this supports the case where the user wants to use the ZeroCost predictor because their dataset or search space is not supported by the zero-cost benchmark.

If this is a case that we want to support, one option would be to introduce a parameter use_zc_api and use it as follows:

if self.use_zc_api:
    model.accuracy = self.zc_api[str(model.arch_hash)]['val_accuracy']
else:
    model.accuracy = model.arch.query(
        self.performance_metric, self.dataset, dataset_api=self.dataset_api
    )

Metadata

Metadata

Labels

zero cost mergeMerge of zerocost with Develop into Develop_copy

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions