Check YAHPO performance

[benjamc]

- +optimizer/smac20=multifidelity
- +problem/YAHPO/multifidelity/SO=cfg_lcbench_167168_epoch_mf
- +customcallback=expl2mf_hybrid
- seed=1

out/err

submitit INFO (2024-06-06 15:34:54,043) - Starting with JobEnvironment(job_id=3309128_15, hostname=ai-n008, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2024-06-06 15:34:54,043) - Loading pickle: /bigwork/nhwpklet/ba/BA_Klenke_Expl2MF/runs/.submitit/3309128_15/3309128_15_submitted.pkl
[�[36m2024-06-06 15:34:54,459�[0m][�[34m/bigwork/nhwpklet/ba/CARP-S/carps/run.py�[0m][�[32mINFO�[0m] - Runcommand: `python -m carps.run +optimizer/smac20=multifidelity +customcallback=expl2mf_hybrid +problem/YAHPO/multifidelity/SO=cfg_nb301_CIFAR10_epoch_mf +clusterconf=cpu seed=1`�[0m
{
    'seed': 1,
    'problem': {
        '_target_': 'carps.benchmarks.yahpo.YahpoProblem',
        'bench': 'nb301',
        'instance': 'CIFAR10',
        'budget_type': 'epoch',
        'metric': ['val_accuracy']
    },
    'task': {
        'n_trials': 250,
        'time_budget': None,
        'n_workers': 1,
        'n_objectives': 1,
        'objectives': ['val_accuracy'],
        'is_multifidelity': True,
        'fidelity_type': 'epoch',
        'min_budget': 1,
        'max_budget': 98,
        'has_constraints': False,
        'domain': 'ML',
        'objective_function_approximation': 'surrogate',
        'has_virtual_time': True,
        'deterministic': True,
        'dimensions': 34,
        'search_space_n_categoricals': 34,
        'search_space_n_ordinals': 0,
        'search_space_n_integers': 0,
        'search_space_n_floats': 0,
        'search_space_has_conditionals': True,
        'search_space_has_forbiddens': False,
        'search_space_has_priors': False
    },
    'benchmark_id': 'YAHPO',
    'problem_id': 'yahpo/multifidelity/so/epoch/nb301/CIFAR10/epoch',
    'optimizer': {
        '_target_': 'carps.optimizers.smac20.SMAC3Optimizer',
        '_partial_': True,
        'smac_cfg': {
            'smac_class': 
'smac.facade.multi_fidelity_facade.MultiFidelityFacade',
            'budget_variable': 'epoch',
            'scenario': {
                'seed': 1,
                'n_trials': 250,
                'deterministic': True,
                'n_workers': 1,
                'output_directory': 
'runs/SMAC-MF-Expl2MF-Hybrid/YAHPO/yahpo/multifidelity/so/epoch/nb301/CIFAR10/ep
och/1/1/smac3_output',
                'min_budget': 1,
                'max_budget': 98
            },
            'smac_kwargs': {
                'dask_client': None,
                'intensifier': {
                    '_target_': 'smac.intensifier.hyperband.Hyperband',
                    '_partial_': True,
                    'eta': 3
                },
                'overwrite': True,
                'callbacks': [
                    {
                        '_target_': 'expl2mf.callback.callback.Expl2MFCallback',
                        'log': True
                    }
                ],
                'acquisition_function': {
                    '_target_': 
'expl2mf.acquisition_function.hybrid_ei_pi.HybridEIPI'
                }
            }
        }
    },
    'optimizer_id': 'SMAC-MF-Expl2MF-Hybrid',
    'optimizer_container_id': 'SMAC3',
    'pyexperimenter_configuration_file_path': None,
    'database_credential_file_path': None,
    'use_special_conda_env': False,
    'conda_env_name': 'carps_YAHPO_SMAC3',
    'timestamp': '2024-06-06/15-34-06',
    'baserundir': 'runs',
    'outdir': 
'runs/SMAC-MF-Expl2MF-Hybrid/YAHPO/yahpo/multifidelity/so/epoch/nb301/CIFAR10/ep
och/1',
    'loggers': [
        {
            '_target_': 'carps.loggers.file_logger.FileLogger',
            '_partial_': True,
            'overwrite': True
        },
        {
            '_target_': 'carps.loggers.database_logger.DatabaseLogger',
            '_partial_': True
        }
    ]
}
[�[36m2024-06-06 15:34:54,632�[0m][�[34mDatabaseLogger�[0m][�[32mINFO�[0m] - Not logging to database (result processor is None).�[0m
submitit INFO (2024-06-06 15:35:15,523) - Job completed successfully
[�[36m2024-06-06 15:35:15,523�[0m][�[34msubmitit�[0m][�[32mINFO�[0m] - Job completed successfully�[0m
submitit INFO (2024-06-06 15:35:15,531) - Exiting after successful completion
[�[36m2024-06-06 15:35:15,531�[0m][�[34msubmitit�[0m][�[32mINFO�[0m] - Exiting after successful completion�[0m

================================================================================
========================== Job Accounting Information ==========================
=                                                                              =
=  To get more details about the job, execute: sacct -o all -j 3309144
=                                                                              =
================================================================================

JobID           JobName ReqNodes  ReqCPUS     ReqMem  Timelimit   TotalCPU  SystemCPU    UserCPU     MaxRSS   NNodes   NTasks    Elapsed      State ExitCode                       NodeList 
------------ ---------- -------- -------- ---------- ---------- ---------- ---------- ---------- ---------- -------- -------- ---------- ---------- -------- ------------------------------ 
3309128_15      expl2mf        1        1         4G   01:00:00  00:04.886  00:03.152  00:01.734                   1            00:00:36  COMPLETED      0:0                        ai-n008 
3309128_15.+      batch        1        2                        00:00.082  00:00.040  00:00.041          0        1        1   00:00:36  COMPLETED      0:0                        ai-n008 
3309128_15.+     extern        1        2                         00:00:00   00:00:00   00:00:00          0        1        1   00:00:36  COMPLETED      0:0                        ai-n008 
3309128_15.0     python        1        2                        00:04.803  00:03.110  00:01.692          0        1        1   00:00:29  COMPLETED      0:0                        ai-n008 

================================================================================
===================== Description of job accounting fields =====================
================================================================================

  JobID     - ID number of the job/step.
               <JobID>.batch: resources requested/used by all commands (including mpirun!) in the job script EXCEPT srun
               <JobID>.extern: resources used by external commands (e.g. session opened by SSH on the reserved node)
               <JobID>.<number>: resources requested/used by the job step (COMMAND srun!) of <number> in the job script

  ReqNodes  - Requested number of nodes for the job/step
  ReqCPUS   - Requested number of CPUs for the job/step
  ReqMem    - Requested amount of memory per CPU ('c' at the end) or 
              per node ('n' at the end) for the job (NOT the step)
  Timelimit - Wallclock time requested for the job

  TotalCPU  - Total CPU time (SystemCPU + UserCPU) used by the job/step
  MaxRSS    - Maximum memory(resident set size) among all tasks in the job/step
  NNode     - Number of nodes in the job/step
  NTasks    - Number of tasks in the job/step
  Elapsed   - Wallclock time spent on the job/step
  ExitCode  - Exit code returned by the job/step
  NodeList  - List of nodes used in job/step