Skip to content

beta distributions with no variability fail #23

@RossBoylan

Description

@RossBoylan

After fixing #22, this further problem was revealed by a specification that set all variances in a row to zero.
This is on the justice branch.

Inputs

pcvd_sd.dat has 2 rows, for AGERG 4 and 5, with 0 standard deviation in all columns. The specification says this is legal, and it means to use the reported means as the simulated means. When montecarlo.py gets to process the first such row, it throws the error shown later. pcvd has a beta distribution.


               TABLE PCVD NHANES 2009-2018 standard error values for monte carlo simulations; aligns with pcJU25.dat

        AGERG     MALE      FEMALE

          0      0.0070     0.0042
          1      0.0145     0.0082
          2      0.0145     0.0099
          3      0.0155     0.0233
          4      0.0000     0.0000
          5      0.0000     0.0000
          7      0.0001     0.0001

For completeness, here is pcvd_mc0.dat, giving the corresponding means:


               TABLE PCVD NHANES 2009-2018 balaced for Probation/parole pop in NSDUH; 09-2018 for 55-64 (no drugs); assume US 05-18 for ages 65+; 10.1.2024 update

        AGERG     MALE      FEMALE

          0      0.0166     0.0107
          1      0.0578     0.0398
          2      0.0852     0.0669
          3      0.1800     0.1947
          4      0.2407     0.1947
          5      0.3608     0.2341
          7      0.0071     0.0007


Traceback

ERR montecarlo.py run failed
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\rdboylan\Documents\KBD\mccli-parallel\pyenv\Lib\site-packages\debugpy\__main__.py", line 71, in <module>
    cli.main()
  File "C:\Users\rdboylan\Documents\KBD\mccli-parallel\pyenv\Lib\site-packages\debugpy\server\cli.py", line 508, in main
    run()
  File "C:\Users\rdboylan\Documents\KBD\mccli-parallel\pyenv\Lib\site-packages\debugpy\server\cli.py", line 358, in run_file
    runpy.run_path(target, run_name="__main__")
  File "C:\Users\rdboylan\Documents\KBD\mccli-parallel\pyenv\Lib\site-packages\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_runpy.py", line 310, in run_path
    return _run_module_code(code, init_globals, run_name, pkg_name=pkg_name, script_name=fname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\rdboylan\Documents\KBD\mccli-parallel\pyenv\Lib\site-packages\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_runpy.py", line 127, in _run_module_code
    _run_code(code, mod_globals, init_globals, mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\rdboylan\Documents\KBD\mccli-parallel\pyenv\Lib\site-packages\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_runpy.py", line 118, in _run_code
    exec(code, run_globals)
  File "C:\Users\rdboylan\Documents\KBD\mccli-parallel\bin\js/../python/montecarlo.py", line 962, in <module>
    main()
  File "C:\Users\rdboylan\Documents\KBD\mccli-parallel\bin\js/../python/montecarlo.py", line 40, in main
    datfile.vary()
  File "C:\Users\rdboylan\Documents\KBD\mccli-parallel\bin\js/../python/montecarlo.py", line 305, in vary
    self.vary_line(line_num)
  File "C:\Users\rdboylan\Documents\KBD\mccli-parallel\bin\js/../python/montecarlo.py", line 372, in vary_line
    varied = self.sdfile.get_variation(line_num)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\rdboylan\Documents\KBD\mccli-parallel\bin\js/../python/montecarlo.py", line 608, in get_variation
    return self._do_line(line_num)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\rdboylan\Documents\KBD\mccli-parallel\bin\js/../python/montecarlo.py", line 614, in vary_individually
    return self._do_dist(means, sds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\rdboylan\Documents\KBD\mccli-parallel\bin\js/../python/montecarlo.py", line 603, in _rand_beta
    return self._correlated_beta(None, means, sds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\rdboylan\Documents\KBD\mccli-parallel\bin\js/../python/montecarlo.py", line 593, in _correlated_beta
    alpha, beta = mean_to_native("beta", ms, ss)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\rdboylan\Documents\KBD\mccli-parallel\bin\js/../python/montecarlo.py", line 161, in mean_to_native
    if sds.min() < 0:
       ^^^^^^^^^
  File "C:\Users\rdboylan\Documents\KBD\mccli-parallel\pyenv\Lib\site-packages\numpy\core\_methods.py", line 45, in _amin
    return umr_minimum(a, axis, None, out, keepdims, initial, where)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: zero-size array to reduction operation minimum which has no identity

Analysis

_correlated_beta operates on valid an invalid values separately. Standard deviations of 0 are considered invalid. When all values in a row of the _sd file are 0, there are no valid values left. The subsequent logic assumes there is at least one valid value, and throws an error when that is not true.

The logic appears somewhat specific to the beta distribution.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpythonissue in python code

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions