Initial implementation of likelihood profiling #37

billdenney · 2022-12-13T04:05:01Z

fix #1

This is a draft implementation of likelihood profiling.

# Conflicts: # DESCRIPTION # R/bootstrap.R

lirui0321 · 2024-04-23T17:33:52Z

Hi Bill and Matt, has this log likelihood profiling feature been added to nlmixr? I feel it is a pretty helpful feature to quantify the uncertainty in estimated parameter values.

mattfidler · 2024-04-23T19:25:07Z

This is Bill's function. He would better understand the timelines.

lirui0321 · 2024-04-24T17:18:47Z

Thank you, Matt! Let's see what Bill says.

Other than variance matrix and bootstrap, is there any other approach currently included in nlmixr for uncertainty quantification? Is there any plan to incorporate an MCMC based approach like BAYES in NONMEM?

billdenney · 2024-04-24T19:09:46Z

I will likely complete this in a few weeks. The implementation is mostly complete, but it will take significant time to implement the tests.

lirui0321 · 2024-04-24T19:37:31Z

I will likely complete this in a few weeks. The implementation is mostly complete, but it will take significant time to implement the tests.

Thanks for the update, Bill! I am happy to try your beta version whenever that becomes available.

mattfidler · 2024-04-25T16:31:17Z

Other than variance matrix and bootstrap, is there any other approach currently included in nlmixr for uncertainty quantification?

Not clear what you mean, but vpc and the variances/standard errors of ETA values can also be used to test for uncertainty. Over-parameterized models can sometimes be checked for by converting the model to PopED (using the development versions as well as babelmixr2's poped "estimation" method).

Is there any plan to incorporate an MCMC based approach like BAYES in NONMEM?

Long term, perhaps adding a stan interface like stanette is on the road-map (though on the long-term road-map). Also prior values are something that is on the road-map.

lirui0321 · 2024-04-26T20:24:33Z

Other than variance matrix and bootstrap, is there any other approach currently included in nlmixr for uncertainty quantification?

Not clear what you mean, but vpc and the variances/standard errors of ETA values can also be used to test for uncertainty. Over-parameterized models can sometimes be checked for by converting the model to PopED (using the development versions as well as babelmixr2's poped "estimation" method).

Is there any plan to incorporate an MCMC based approach like BAYES in NONMEM?

Long term, perhaps adding a stan interface like stanette is on the road-map (though on the long-term road-map). Also prior values are something that is on the road-map.

Thanks, Matt! I am a "more" preclinical guy usually facing sparse data (e.g., mean digitized from literature) but relatively complicated model structure with multiple fitted parameters. Sometimes variance/covariance step fails. Sometimes I don't trust SE from covariance matrix because it tends to underestimate uncertainty. I like your plan of adding stan interface. NONMEM has BAYES and NUT, but again both approaches are not very friendly with data of limited N.

mattfidler · 2024-04-27T01:18:39Z

Nothing other than bootstrapping currently.

mattfidler · 2024-04-27T02:32:24Z

There is the preconditionFit function, too, if you are trying to get more reliable covariance. It does require some information about your covariance though... and it may not work with all scenarios.

lirui0321 · 2024-05-01T16:41:58Z

There is the preconditionFit function, too, if you are trying to get more reliable covariance. It does require some information about your covariance though... and it may not work with all scenarios.

Many thanks for your feedback! rxode2 and nlmixr are still very convenient tools for me even without these additional features.

simonbeyer1 · 2024-07-02T12:54:37Z

R/profile.R

Hi Bill (@billdenney),
Thanks for the initial implementation. Very interesting! I have a question to you: Your code only allows the profiling of fixed effect parameters. Why not profiling e.g. the variance-covariances?
Thanks in advance!
Simon

There's no specific reason for that right now. Generally, profiling fixed effects is easier than random effects because random effects can be correlated. So, I think that you would need to detect correlation and only allow profiling of the random effects without correlations (or remove the correlation). @mattfidler, can you fix a random effect or its correlation when it is correlated without fixing the other parts?

You have to fix the whole block

I was referring the omegas not the individual etas, since they are part of the marginal likelihood.

@simonbeyer1, yes, I understood that you were referring to omegas. The idea I was getting to is that if you specify an eta like etacl + etavc ~ c(0.2, 0.1, 0.2), we would have to fix or not fix the entirety of that, all or none of the c(0.2, 0.1, 0.2) part. I don't immediately have a good idea of how to profile correlated BSV, so that will remain out of scope for this initial version of profiling.

ahh okay! Now I got it! Thanks!

…s for at least one test problem.

lirui0321 · 2024-08-19T14:01:51Z

Hi Bill,

By reading the comments above, it sounds like you have implemented a (beta?) version of LLP. Is there a tutorial about how to use it with nlmixr2? I am very interested in trying it. Thanks!

billdenney · 2024-08-19T14:43:56Z

Hi Bill,

By reading the comments above, it sounds like you have implemented a (beta?) version of LLP. Is there a tutorial about how to use it with nlmixr2? I am very interested in trying it. Thanks!

I love the excitement! It isn't quite ready for use as I'm still working on the interface. I'm actively working on it, so I hope that it will be ready for testing soon. And, when it's ready for testing, I'll be sure to comment here and/or in the original issue. Alternatively, this pull request being merged will be an indication that it's ready for testing.

lirui0321 · 2024-08-19T15:04:20Z

Hi Bill,
By reading the comments above, it sounds like you have implemented a (beta?) version of LLP. Is there a tutorial about how to use it with nlmixr2? I am very interested in trying it. Thanks!

I love the excitement! It isn't quite ready for use as I'm still working on the interface. I'm actively working on it, so I hope that it will be ready for testing soon. And, when it's ready for testing, I'll be sure to comment here and/or in the original issue. Alternatively, this pull request being merged will be an indication that it's ready for testing.

Thank you, Bill! Looking forward to testing it when it is ready.

…rameter; Capture the bound estimate

…guments

…; ensure that new estimates have a small margin to the bounds

billdenney · 2024-08-20T17:47:18Z

@lirui0321 and @simonbeyer1 , This is ready for testing. I tried to make the documentation in ?profile.nlmixr2FitCore clear for how to use it. So, can you please take a look there and let me know if you have any questions?

@mattfidler, Can you please take a look and let me know if you see any issues?

Issues that I'm currently aware of are:

It doesn't handle fixing random effects, only fixed effects are supported.
If a new minimum OFV is found during profiling, the results may not be reliable.
If the OFV is not monotonic (an example is here:

nlmixr2extra/tests/testthat/test-profile.R

Line 152 in 8e42770

profadd.sd <- profile(fit, which = "add.sd")

), the results may not be reliable.
It should interface with confidence interval methods in the rest of nlmixr2 so that its output could be used in the fit$parFixed estimates.

mattfidler · 2024-09-17T22:54:16Z

R/profile.R

+#' @family Profiling
+#' @seealso [profileLlp()]
+#' @export
+llpControl <- function(ofvIncrease = qchisq(0.95, df = 1),


Because of shinyMixR this should now have a new method rxUiDeparse.llpControl()... I will write one quickly. I am trying to release this today and would like this included (I know I'm reviewing last minute...)

mattfidler · 2024-09-17T22:57:21Z

It would be nice it it would only use one function

nlmixr2(fit, est="llp")

But, it seems ok to me.

mattfidler · 2024-09-17T22:57:46Z

We may add that as a future feature.

mattfidler · 2024-09-17T23:03:30Z

Unless you have strong objections, I will submit this with the rest.

billdenney · 2024-09-18T00:36:29Z

Looks good to me. Thanks!

billdenney added 5 commits December 12, 2022 23:04

Initial implementation of likelihood profiling (fix #1)

4ca65f4

Additional feature updates

6c540b8

Intermediate work

f4503f4

Merge branch 'main' into fix-1

612ca73

# Conflicts: # DESCRIPTION # R/bootstrap.R

Update documentation

fc6e1b5

simonbeyer1 reviewed Jul 2, 2024

View reviewed changes

billdenney added 4 commits August 7, 2024 23:23

Merge branch 'main' into fix-1

b516473

Update documentation

c7a3a63

Fix check issues

725830e

A much-improved work-in-progress. profile.nlmixr2FitCore() now work…

56ac86e

…s for at least one test problem.

billdenney added 7 commits August 19, 2024 13:59

Stop estimation based on number of significant digits in estimated pa…

f8d87f5

…rameter; Capture the bound estimate

Use the default RSE when it is not available

589a622

Rename rse_theta to rseTheta

d3c5272

Update CI

114ccf2

Remove quiet argument

1a7b123

Add more development version packages for CI

26b965a

correct dparser repo location

c482d93

billdenney added 7 commits August 19, 2024 14:38

Remove unused ignoreBounds argument

4c1891d

Fix check issues

5159186

Remove unused maxpts argument

98805e1

Switch to a generalized profile method with method and control ar…

3bfb925

…guments

Ensure that initial estimates are within the limits for the parameter…

8e5f79f

…; ensure that new estimates have a small margin to the bounds

Test residual error profiling

cd720f6

Update documentation

8e42770

billdenney marked this pull request as ready for review August 20, 2024 17:47

billdenney added 2 commits August 20, 2024 13:49

Documentation fix

101d16a

Add omega values and covariances to profiling output

1b705ca

mattfidler reviewed Sep 17, 2024

View reviewed changes

Add rxUiDeparse method for llpControl

509db7b

mattfidler added 2 commits September 17, 2024 18:04

Version bump

64a4f43

Update check config

b17bbe1

mattfidler merged commit ea0cfc7 into main Sep 18, 2024
8 checks passed

mattfidler deleted the fix-1 branch September 18, 2024 02:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial implementation of likelihood profiling #37

Initial implementation of likelihood profiling #37

billdenney commented Dec 13, 2022

lirui0321 commented Apr 23, 2024

mattfidler commented Apr 23, 2024

lirui0321 commented Apr 24, 2024

billdenney commented Apr 24, 2024

lirui0321 commented Apr 24, 2024

mattfidler commented Apr 25, 2024 •

edited

Loading

lirui0321 commented Apr 26, 2024

mattfidler commented Apr 27, 2024

mattfidler commented Apr 27, 2024

lirui0321 commented May 1, 2024

simonbeyer1 Jul 2, 2024

billdenney Aug 18, 2024

mattfidler Aug 19, 2024

simonbeyer1 Aug 19, 2024

billdenney Aug 19, 2024

simonbeyer1 Aug 19, 2024

lirui0321 commented Aug 19, 2024

billdenney commented Aug 19, 2024

lirui0321 commented Aug 19, 2024

billdenney commented Aug 20, 2024 •

edited

Loading

mattfidler Sep 17, 2024

mattfidler commented Sep 17, 2024

mattfidler commented Sep 17, 2024 •

edited

Loading

mattfidler commented Sep 17, 2024

billdenney commented Sep 18, 2024

Initial implementation of likelihood profiling #37

Initial implementation of likelihood profiling #37

Conversation

billdenney commented Dec 13, 2022

lirui0321 commented Apr 23, 2024

mattfidler commented Apr 23, 2024

lirui0321 commented Apr 24, 2024

billdenney commented Apr 24, 2024

lirui0321 commented Apr 24, 2024

mattfidler commented Apr 25, 2024 • edited Loading

lirui0321 commented Apr 26, 2024

mattfidler commented Apr 27, 2024

mattfidler commented Apr 27, 2024

lirui0321 commented May 1, 2024

simonbeyer1 Jul 2, 2024

Choose a reason for hiding this comment

billdenney Aug 18, 2024

Choose a reason for hiding this comment

mattfidler Aug 19, 2024

Choose a reason for hiding this comment

simonbeyer1 Aug 19, 2024

Choose a reason for hiding this comment

billdenney Aug 19, 2024

Choose a reason for hiding this comment

simonbeyer1 Aug 19, 2024

Choose a reason for hiding this comment

lirui0321 commented Aug 19, 2024

billdenney commented Aug 19, 2024

lirui0321 commented Aug 19, 2024

billdenney commented Aug 20, 2024 • edited Loading

mattfidler Sep 17, 2024

Choose a reason for hiding this comment

mattfidler commented Sep 17, 2024

mattfidler commented Sep 17, 2024 • edited Loading

mattfidler commented Sep 17, 2024

billdenney commented Sep 18, 2024

mattfidler commented Apr 25, 2024 •

edited

Loading

billdenney commented Aug 20, 2024 •

edited

Loading

mattfidler commented Sep 17, 2024 •

edited

Loading