Add ifbo

[Neeratyoy]

Adding ifBO and refactoring a bunch of freeze-thaw based multi-fidelity algorithms.
Working example here.

Existing issues or concerns:

• Adding parallelization support
  • Need resolving some issues
• Tblogger from the example does not seem to work
  • Only config_0_0 has the tbevent files
  • (did not reproduce) On restart, there was an error from the tbevent file-not-there
• Resolving Python dependency removing python 3.8 and 3.9 support #140

[Neeratyoy]

@eddiebergman @TarekAbouChakra is the tblogging supported currently?
If you can install this branch and run this example and view Tensorboard, it is no longer working.

[eddiebergman]

Just fyi, [PR](#140) didn't work as intended, you can just do PR #140 :)

[eddiebergman]

As for the PR about python dependancies, just get ifbo in first and we can do the python dep thing after

[eddiebergman]

Regarding tblogger, I can't initially see what the issue is. Seems that the tblogger at each iteration will get the trial that is currently being evaluated from the runtime.py. The runtime.py will ask the optimizer for the next config to evaluate (which returns the config dict as well as the ID), sets that as the currently running trial and then sends it to run_pipeline(). Inside run_pipeline() is where the tblogger.log() is called, at which point it asks the runtime.py, "hey, what config is currently being evaluated?". Should be the exact same as what's used for determining the config directory.

[eddiebergman]

Think I found the issue:

This line:

neps/neps/plot/tensorboard_eval.py

Lines 316 to 317 in 0025d3d

	if not tblogger._is_initialized():
	tblogger._initialize_writers()

Basically it only initializes once, where intialization is in charge of deciding where to write to:

neps/neps/plot/tensorboard_eval.py

Lines 109 to 135 in 0025d3d

	def _initialize_writers() -> None:
	# This code runs only once per config, to assign that config a config_writer.
	if (
	tblogger.config_previous_directory is None
	and tblogger.config_working_directory
	):
	# If no fidelities are there yet, define the writer via the config_id
	tblogger.config_id = str(tblogger.config_working_directory).rsplit(
	"/", maxsplit=1
	)[-1]
	tblogger.config_writer = SummaryWriter_(
	tblogger.config_working_directory / "tbevents"
	)
	return
	# Searching for the initial directory where tensorboard events are stored.
	if tblogger.config_working_directory:
	init_dir = get_initial_directory(
	pipeline_directory=tblogger.config_working_directory
	)
	tblogger.config_id = str(init_dir).rsplit("/", maxsplit=1)[-1]
	if (init_dir / "tbevents").exists():
	tblogger.config_writer = SummaryWriter_(init_dir / "tbevents")
	return
	raise FileNotFoundError(
	"'tbevents' was not found in the initial directory of the configuration."
	)

[Neeratyoy]

@eddiebergman how should we go about this PR?

Parallelization works and I happy for now with local testing.

Tblogger is the biggest bummer here, along with test cases failing.

[timurcarstensen]

Suggested change

	python-version: 3.10
	python-version: '3.10'