LAT Mv11 Weird Heating Issue

[skhrg]

Mv11 (installed in LAT tube i6 and connected to slot 6 of the crate) sometimes has wild heating events:

which seems to be correlated with timing issues:

We were able to replicate this by faking a timing issue by switching slot 2 to ext_ref while leaving slot 6 on backplane:

Putting the FR in a couple fixed biases it seems like when timing drops out it puts it at about .5.

We then had Sam swap the DSUBs so that this UFM is now connected to slot 7 (and its associated cryocard). The issue stayed with the UFM:

It seems that there are two things happening here:

1. Bad timing makes the FR act up
2. This UFM heats up more than usual when the FR is high.

More information on 2:

• When setup normally this UFM runs warmer than others but is still in an acceptable range with (mostly) good noise
• There are some channels on the UFM that have some weird low freq power even when SC:
  
• The frac peak to peak on both AMCs is ~.35-.37 which seems typical compared to the others installed in the LAT

[dpdutcher]

Thanks for putting this summary together!

We have seen UFMs that get heated by the FR signal more than others, and it depends on flux ramp amplitude. Here is a particularly egregious example of a bad LF UFM, where I raised the flux ramp frac_pp from 0 to 1 in steps of 0.1, you can see the MC temp go from around 100mK to 450mK.

Even though this is obviously unusable, nothing was obviously wrong electrically: the flux ramp was not shorted to ground, another bias line, or itself (that we could measure).

A similar, but less egregious, thing also happened on UFM-Uv36r1. When the FR was powered, the thermometer on that UFM increased from 104mK to 121mK. When optimizing frac_pp, it determined 0.405 on the North and 0.360 on the South. This is a bigger difference than I would accept. Using the average frac_pp=0.382 for both, the superconducting white noise was around 150pA/rtHz instead of 130pA/rtHz. Again, everything seems fine electrically -- no shorts to gnd or another BL. We're debugging this one right now.

In your case, if each AMC is giving the same frac_pp, and the normal amplitude of flux ramp does not cause heating, and the noise looks good, then I would just leave it as is. Your noise does look mostly fine to me -- a spike at 50Hz is usually indicative of a short to gnd on a bias line, not a flux ramp. We almost (or maybe literally?) never see a flux ramp-ground short, and a flux ramp-BL short usually results in insanely high noise, like 10000 pA/rtHz.

So, I think the timing issue should be fixed, and once it is, then you're good.

[swh76]

Ok I understand this now from the SMuRF side; here's what I did on the bench ;

On a SMuRF server connected to timing and in ext_ref timing mode, I looked at flux ramp out of the RTM flux monitor output and ran S.tracking_setup(0,fraction_full_scale=1.0,reset_rate_khz=4) and saw a 4 kHz ramp, with a peak-to-peak voltage swing of 2.6V (this was to remind myself what full-scale is out of the flux ramp monitor output). Then I configured to accept timing (backplane here because the SMuRF blade is in slot 4 of a crate with a timing master in slot 2);

S.set timing_mode('backplane')
S.tracking_setup (0, fraction_full_scale=0.25, reset_ rate_khz=4)

and I see

here I've set fraction_full_scale to 0.25 to simulate typical values we use. If I then simulate timing dropout by changing timing mode to fiber, the flux ramp changes to look like this;

so it seems like the behavior is that if the system is expecting timing but doesn't get it, it ramps over the full flux ramp scale. I think this is because of the way the flux ramp is implemented; low level, the RTM DAC is a forever rail to rail ramp that gets reset by a trigger, which in this case is coming from the timing system. Without the timing trigger, it just ramps over the full scale over and over again. That's also why it's 1 kHz after timing loss ; since I had set fraction_full_scale to 0.25, or a 1/4 of the full flux ramp output range, since the ramp rate was configured to do that at 4kHz, it results in a full scale 1kHz ramp (since trigger is absent).

[swh76]

One question on this ; does the fraction_full_scale / lms_freq_hz the tuning methods compute for slot 6 agree for both N and S sides of UFM with the values for the other UFMs? If different, might point to current leaking somewhere it shouldn't in the flux ramp circuit (and possibly causing heating).

[skhrg]

For fraction_full_scale its ~.35-.36 on slot 6 which seems pretty typical looking at other slots.
lms_freq_hz is 20000.0 across the board for all slots.