Do new rpointers actually work when running NINST>1 starting from a single instance case?

[mvdebolskiy]

I see this code:

CTSM/cime_config/buildnml

Lines 315 to 332 in 7ff6061

	ninst = int(ninst_lnd)
	for inst_counter in range(1, ninst + 1):
	# determine instance string
	inst_string = ""
	if ninst > 1:
	inst_string = "_" + "%04d" % inst_counter
	# If multi-instance case does not have restart file, use
	# single-case restart for each instance
	rpointer = "rpointer.lnd"
	if os.path.isfile(os.path.join(rundir, rpointer)) and (
	not os.path.isfile(os.path.join(rundir, rpointer + inst_string))
	):
	shutil.copy(
	os.path.join(rundir, rpointer),
	os.path.join(rundir, rpointer + inst_string),
	)

There is no mention of the rpointer timestamp, which means, if you start an NINST>1 case from a NINST=1 case with rpointers like rpointer.lnd.XXXX-XX-XX-XXXXX there will never be rpointers created for the multiple instances.

Also, right now there are no tests for multiple instances afaik. I do not run multiple instance cases, however, I assume calibration ppl do.
@adrifoster do you use NINST in you fates calibration?

[ekluzek]

@mvdebolskiy these are the multi-instance tests we have:

  <test name="ERI_C2_Ld9" grid="f10_f10_mg37" compset="I2000Clm60BgcCrop" testmods="clm/default">
  <test name="SMS_C2_D_Lh12" grid="f10_f10_mg37" compset="I2000Clm50Sp" testmods="clm/pauseResume">
  <test name="DAE_C2_D_Lh12" grid="f10_f10_mg37" compset="I2000Clm50BgcCrop" testmods="clm/DA_multidrv">

[ekluzek]

@mvdebolskiy your inspection of the code looks correct to me, and it will fail if you startup a multi-instance case from a single instance. We aren't testing for that -- should we?

[mvdebolskiy]

@ekluzek not sure how often that situation comes up.

[mvdebolskiy]

@ekluzek Also, looking in the buildnml: ctsm.inpudata_list will only get the last inputdata from ninst.

Here the inputdata_list is defined outside the loop:

CTSM/cime_config/buildnml

Lines 304 to 310 in 7ff6061

	# -----------------------------------------------------
	# Clear out old data
	# -----------------------------------------------------
	if os.path.exists(inputdata_file):
	os.remove(inputdata_file)

here it will get overwritten with every iteration over instances:

CTSM/cime_config/buildnml

Lines 387 to 422 in 7ff6061

	command = (
	'%s -cimeroot %s -infile %s -csmdata %s -inputdata %s %s -namelist "&clm_inparm start_ymd=%s %s/ " '
	"%s %s -res %s %s -clm_start_type %s -envxml_dir %s "
	"-configuration %s -structure %s "
	"%s -glc_nec %s %s -co2_ppmv %s -co2_type %s -config %s -driver %s "
	"%s %s %s %s"
	% (
	cmd,
	_CIMEROOT,
	infile,
	din_loc_root,
	inputdata_file,
	ignore,
	start_ymd,
	clm_namelist_opts,
	nomeg,
	usecase,
	lnd_grid,
	clmusr,
	start_type,
	caseroot,
	configuration,
	structure,
	lndfrac_setting,
	glc_nec,
	glc_use_antarctica_flag,
	ccsm_co2_ppmv,
	clm_co2_type,
	config_cache_file,
	driver,
	clm_bldnml_opts,
	spinup,
	tuning,
	gridmask,
	)
	)

This will lead for ./check_inputdata not to actually check properly if any of the input data-files are different between the instances.
This also probably a thing for other components.

[jedwards4b]

I think that starting a multinstance case from a single instance is not very common and this issue can easily be worked around with:
cp rpointer.clm rpointer_0001.clm
cp rpointer.clm rpointer_0002.clm etc.

[ekluzek]

Good point, @jedwards4b. Let's just leave this open as low priority in case someone has a chance to get to it.