Merge pull request #566 from EducationalTestingService/release/v9.0

Release/v9.0

MANIFEST.in

@@ -6,6 +6,9 @@ recursive-exclude tests *
 recursive-exclude examples *
 recursive-include rsmtool/notebooks *
 prune test_outputs
+prune rsmtool/__pycache__
+prune rsmtool/utils/__pycache__
 prune rsmtool/rsmtool.egg-info
 prune rsmtool/notebooks/.ipynb_checkpoints
 prune rsmtool/notebooks/comparison/.ipynb_checkpoints
+prune rsmtool/notebooks/summary/.ipynb_checkpoints

README.rst

@@ -11,7 +11,7 @@ Rater Scoring Modeling Tool
 
 .. image:: https://codecov.io/gh/EducationalTestingService/rsmtool/branch/main/graph/badge.svg?token=wMCnQrB49J
    :target: https://codecov.io/gh/EducationalTestingService/rsmtool
-   :alt: Coverage status   
+   :alt: Coverage status
 
 .. image:: https://img.shields.io/conda/v/ets/rsmtool.svg
    :target: https://anaconda.org/ets/rsmtool
@@ -39,11 +39,11 @@ Rater Scoring Modeling Tool
 Introduction
 ------------
 
-Automated scoring of written and spoken test responses is a growing field in educational natural language processing. Automated scoring engines employ machine learning models to predict scores for such responses based on features extracted from the text/audio of these responses. Examples of automated scoring engines include `Project Essay Grade <http://pegwriting.com/about>`_ for written responses and `SpeechRater <https://www.ets.org/research/topics/as_nlp/speech/>`_ for spoken responses.
+Automated scoring of written and spoken test responses is a growing field in educational natural language processing. Automated scoring engines employ machine learning models to predict scores for such responses based on features extracted from the text/audio of these responses. Examples of automated scoring engines include `Project Essay Grade <http://pegwriting.com/about>`_ for written responses and `SpeechRater <https://www.ets.org/research/policy_research_reports/publications/report/2008/hukv>`_ for spoken responses.
 
 Rater Scoring Modeling Tool (RSMTool) is a python package which automates and combines in a single pipeline multiple analyses that are commonly conducted when building and evaluating such scoring models.  The output of RSMTool is a comprehensive, customizable HTML statistical report that contains the output of these multiple analyses. While RSMTool does make it really simple to run a set of standard analyses using a single command, it is also fully customizable and allows users to easily exclude unneeded analyses, modify the default analyses, and even include custom analyses in the report.
 
-We expect the primary users of RSMTool to be researchers working on developing new automated scoring engines or on improving existing ones. Note that RSMTool is not a scoring engine by itself but rather a tool for building and evaluating machine learning models that may be used in such engines. 
+We expect the primary users of RSMTool to be researchers working on developing new automated scoring engines or on improving existing ones. Note that RSMTool is not a scoring engine by itself but rather a tool for building and evaluating machine learning models that may be used in such engines.
 
 RSMTool is driven by a configuration file that users have to supply. Given the large number of available options, this can get complicated especially for new users. That's why RSMTool can help users generate configuration files interactively via guided prompts! The video below demonstrates this feature.
 
@@ -59,7 +59,7 @@ To get started with RSMTool, please see the extensive `official documentation <h
 Requirements
 ------------
 
-- Python >=3.7, <3.10
+- Python >=3.8, <3.11
 - ``numpy``
 - ``scipy``
 - ``scikit-learn``

azure-pipelines.yml

@@ -36,7 +36,7 @@ jobs:
       pip install -e .
     displayName: 'Install dependencies & code'
 
-  - powershell: ./DistributeTests.ps1 
+  - powershell: ./DistributeTests.ps1
     displayName: 'PowerShell Script to distribute tests'
 
   - script: |

conda-recipe/rsmtool/meta.yaml

@@ -1,14 +1,13 @@
-{% set data = load_setup_py_data() %}
-
 package:
   name: rsmtool
-  version: {{data.get('version')}}
+  version: 9.0
 
 source:
   path: ../../../rsmtool
 
 build:
   number: 0
+  noarch: python
   script:
     - cd $SRC_DIR
     - "{{ PYTHON }} -m pip install . --no-deps -vv"
@@ -18,36 +17,28 @@ build:
     - rsmeval = rsmtool.rsmeval:main
     - rsmpredict = rsmtool.rsmpredict:main
     - rsmsummarize = rsmtool.rsmsummarize:main
+    - rsmxval = rsmtool.rsmxval:main
     - render_notebook = rsmtool.reporter:main
     - convert_feature_json = rsmtool.convert_feature_json:main
 
 requirements:
   build:
-    - python
-    - jupyter
-    - ipython
-    - notebook
-    - numpy {{ numpy }}
-    - pandas
-    - seaborn
-    - skll==2.5.0
-    - statsmodels
-    - openpyxl
-    - xlrd
-    - xlwt
-
+    - python >=3.8
+    - pip
+    - setuptools
   run:
     - python
     - jupyter
     - ipython
     - nose
     - notebook
     - numpy
+    - openpyxl
     - pandas
     - seaborn
-    - skll==2.5.0
+    - skll==3.0.0
     - statsmodels
-    - openpyxl
+    - tqdm
     - xlrd
     - xlwt
 
@@ -60,6 +51,7 @@ test:
     - rsmtool.configuration_parser
     - rsmtool.container
     - rsmtool.convert_feature_json
+    - rsmtool.fairness_utils
     - rsmtool.modeler
     - rsmtool.preprocessor
     - rsmtool.reader
@@ -75,6 +67,7 @@ test:
     - rsmcompare --help
     - rsmpredict --help
     - rsmsummarize --help
+    - rsmxval --help
     - render_notebook --help
     - convert_feature_json --help
 

doc/automated_configuration.rst

@@ -2,7 +2,7 @@
 
 Auto-generating configuration files
 -----------------------------------
-Configuration files for :ref:`rsmtool <config_file_rsmtool>`, :ref:`rsmeval <config_file_rsmeval>`, :ref:`rsmcompare <config_file_rsmcompare>`, :ref:`rsmpredict <config_file_rsmpredict>`, :ref:`rsmsummarize <config_file_rsmsummarize>`, :ref:`rsmxval <config_file_rsmxval>` can be difficult to create manually due to the large number of configuration options supported by these tools. To make this easier for users, all of these tools support *automatic* creation of configuration files, both interactively and non-interactively.
+Configuration files for :ref:`rsmtool <config_file_rsmtool>`, :ref:`rsmeval <config_file_rsmeval>`, :ref:`rsmcompare <config_file_rsmcompare>`, :ref:`rsmpredict <config_file_rsmpredict>`, :ref:`rsmsummarize <config_file_rsmsummarize>`, and :ref:`rsmxval <config_file_rsmxval>` can be difficult to create manually due to the large number of configuration options supported by these tools. To make this easier for users, all of these tools support *automatic* creation of configuration files, both interactively and non-interactively.
 
 Interactive generation
 ~~~~~~~~~~~~~~~~~~~~~~
@@ -16,7 +16,7 @@ For example, to generate an ``rsmtool`` configuration file interactively, run th
 The following screencast shows an example interactive session after the above command is run (click to play):
 
 .. raw:: html
-   
+
    <script id="asciicast-313107" src="https://asciinema.org/a/313107.js" data-autoplay="false" async></script>
 
 
@@ -36,15 +36,15 @@ There are some configuration options that can accept multiple inputs. For exampl
 The following screencast shows the interactive session (click to play):
 
 .. raw:: html
-    
+
    <script id="asciicast-313149" src="https://asciinema.org/a/313149.js" data-autoplay="false" async></script>
 
 And here is the generated configuration file for ``rsmsummarize``:
 
 .. literalinclude:: assets/example_rsmsummarize.json
     :language: javascript
 
-.. important:: 
+.. important::
 
     If you want to include subgroup information in the reports for ``rsmtool``, ``rsmeval``, ``rsmcompare``, and ``rsmxval``, you should add ``--subgroups`` to the command. For example, when you run ``rsmeval generate --interactive --subgroups`` you would be prompted to enter the subgroup column names and the ``general_sections`` list (if shown [#f1]_) will also include subgroup-based sections.  Since the ``subgroups`` option can accept multiple inputs, it is handled in the same way as the ``experiment_dirs`` option for ``rsmsummarize`` above.
 
@@ -58,7 +58,7 @@ We end with a list of important things to note about interactive generation:
 
 - Required fields will *not* accept a blank input (just pressing enter) and will show an error message in the bottom left until a valid input is provided.
 
-- Optional fields will accept blank inputs since they have default values that will be used if no user input is provided. In some cases, default values are shown underlined in parentheses. 
+- Optional fields will accept blank inputs since they have default values that will be used if no user input is provided. In some cases, default values are shown underlined in parentheses.
 
 - You can also use ``-i`` as an alias for ``--interactive`` and ``-g`` as an alias for ``--subgroups``. So, for example, if you want to interactively generate a configuration file with subgroups for ``rsmtool``, just run ``rsmtool generate -ig`` instead of ``rsmtool generate --interactive --subgroups``.
 

doc/getting_started.rst

@@ -2,18 +2,18 @@
 
 Installation
 ============
-Note that RSMTool currently works with Python 3.7, 3.8, and 3.9.
+Note that RSMTool currently works with Python 3.8, 3.9, and 3.10.
 
 Installing with conda
 ----------------------
 
 Currently, the recommended way to install RSMTool is by using the ``conda`` package manager. If you have already installed ``conda``, you can skip straight to Step 2.
 
-1. To install ``conda``, follow the instructions on `this page <https://conda.io/projects/conda/en/latest/user-guide/install/index.html>`_. 
+1. To install ``conda``, follow the instructions on `this page <https://conda.io/projects/conda/en/latest/user-guide/install/index.html>`_.
 
-2. Create a new conda environment (say, ``rsmtool``) and install the RSMTool conda package for your preferred Python version. For example, for Python 3.7, run::
+2. Create a new conda environment (say, ``rsmtool``) and install the RSMTool conda package for your preferred Python version. For example, for Python 3.8, run::
 
-    conda create -n rsmtool -c conda-forge -c ets python=3.7 rsmtool
+    conda create -n rsmtool -c conda-forge -c ets python=3.8 rsmtool
 
 3. Activate this conda environment by running ``conda activate rsmtool``. You should now have all of the RSMTool command-line utilities in your path. [#]_
 

doc/index.rst

@@ -15,7 +15,7 @@ Rater Scoring Modeling Tool (RSMTool)
 
     .. image:: assets/spacer.png
 
-Automated scoring of written and spoken responses is a growing field in educational natural language processing. Automated scoring engines employ machine learning models to predict scores for such responses based on features extracted from the text/audio of these responses. Examples of automated scoring engines include `MI Write <https://measurementinc.com/miwrite>`_ for written responses and `SpeechRater <https://www.ets.org/research/topics/as_nlp/speech/>`_ for spoken responses.
+Automated scoring of written and spoken responses is a growing field in educational natural language processing. Automated scoring engines employ machine learning models to predict scores for such responses based on features extracted from the text/audio of these responses. Examples of automated scoring engines include `MI Write <https://measurementinc.com/miwrite>`_ for written responses and `SpeechRater <https://www.ets.org/research/policy_research_reports/publications/report/2008/hukv>`_ for spoken responses.
 
 RSMTool is a python package which automates and combines in a *single* :doc:`pipeline <pipeline>` multiple analyses that are commonly conducted when building and evaluating automated scoring models. The output of RSMTool is a comprehensive, customizable HTML statistical report that contains the outputs of these multiple analyses. While RSMTool does make it really simple to run this set of standard analyses using a single command, it is also fully customizable and allows users to easily exclude unneeded analyses, modify the standard analyses, and even include custom analyses in the report.
 
@@ -27,7 +27,7 @@ The primary means of using RSMTool is via the :doc:`command-line <usage_rsmtool>
 Documentation
 =============
 
-.. note:: 
+.. note::
 
    If you use the `Dash <https://kapeli.com/dash>`_ app on macOS, you can also download the complete RSMTool documentation for offline use. Go to the Dash preferences, click on "Downloads", then "User Contributed", and search for "RSMTool".
 

doc/internal/release_process.rst

@@ -9,14 +9,14 @@ This process is only meant for the project administrators, not users and develop
 
 #. Run the ``tests/update_files.py`` script with the appropriate arguments to make sure that all test data in the new release have correct experiment ids and filenames. If any (non-model) files need to be changed this should be investigated before the branch is released. Please see more details about running this `here <https://rsmtool.readthedocs.io/en/stable/contributing.html#writing-new-functional-tests>`__.
 
-    .. note:: 
+    .. note::
 
         Several files have been excluded from the repository due to their non-deterministic nature so please do not add them back to the repository. The following files are currently excluded:
 
             * Fairness test files for `lr-eval-system-score-constant` test
-            * Predictions and all evaluation files for `linearsvr` test. 
-     
-        Note that the full set of outputs from these test files are also used as input for `rsmcompare` and `rsmsummarize` tests. These *input* files need to be updated following the process under **Example 2** in `Writing new functional tests <https://rsmtool.readthedocs.io/en/stable/contributing.html#writing-new-functional-tests>`_. You can also see `this pull request <https://github.com/EducationalTestingService/rsmtool/pull/525>`_ for more information. 
+            * Predictions and all evaluation files for `linearsvr` test.
+
+        Note that the full set of outputs from these test files are also used as input for `rsmcompare` and `rsmsummarize` tests. These *input* files need to be updated following the process under **Example 2** in `Writing new functional tests <https://rsmtool.readthedocs.io/en/stable/contributing.html#writing-new-functional-tests>`_. You can also see `this pull request <https://github.com/EducationalTestingService/rsmtool/pull/525>`_ for more information.
 
 #. Create a release branch ``release/XX`` on GitHub.
 
@@ -34,23 +34,21 @@ This process is only meant for the project administrators, not users and develop
 
 #. Build the PyPI source and wheel distributions using ``python setup.py sdist build`` and ``python setup.py bdist_wheel build`` respectively.
 
-#. Upload the source and wheel distributions to TestPyPI  using ``twine upload --repository testpypi dist/*``. You will need to have the ``twine`` package installed and set up your ``$HOME/.pypirc`` correctly. See details `here <https://packaging.python.org/guides/using-testpypi/>`__. You will need to have the appropriate permissions for the ``ets`` organization
+#. Upload the source and wheel distributions to TestPyPI using ``twine upload --repository testpypi dist/*``. You will need to have the ``twine`` package installed and set up your ``$HOME/.pypirc`` correctly. See details `here <https://packaging.python.org/guides/using-testpypi/>`__. You will need to have the appropriate permissions for the ``ets`` organization on TestPyPI.
 
 #. Install the TestPyPI package as follows::
 
     pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple rsmtool
 
 #. Then run some tests from a RSMTool working copy. If the TestPyPI package works, then move on to the next step. If it doesn't, figure out why and rebuild and re-upload the package.
 
-#. Build the new conda package by running the following command in the ``conda-recipe`` directory (note that this assumes that you have cloned RSMTool in a directory named ``rsmtool``)::
+#. Build the new conda package by running the following command in the ``conda-recipe`` directory (note that this assumes that you have cloned RSMTool in a directory named ``rsmtool``). Note that you may need to comment out lines in your `$HOME/.condarc` file if you are using ETS Artifactory and you get conflicts::
 
     conda build -c conda-forge -c ets .
 
-#. This will create python 3.7, 3.8, and 3.9 packages for your native platform, e.g., ``osx-64``.
-
-#. Convert these built packages for the other two platforms. For example, if you ran the above command on macOS, run ``conda convert -p linux-64 -p win-64 <packages files>``, where `<packages_files>` are the package files that were created in step 10.
+#. This will create a noarch package with the path to the package printed out to the screen.
 
-#. Upload all 9 package files (3 Python versions x 3 platforms) to anaconda.org using ``anaconda upload --user ets <path_to_files>``. You will need to have the appropriate permissions for the ``ets`` organization. 
+#. Upload the package file to anaconda.org using ``anaconda upload --user ets <path_to_file>``. You will need to have the appropriate permissions for the ``ets`` organization.
 
 #. Create pull requests on the `rsmtool-conda-tester <https://github.com/EducationalTestingService/rsmtool-conda-tester/>`_ and `rsmtool-pip-tester <https://github.com/EducationalTestingService/rsmtool-pip-tester/>`_ repositories to test the conda and TestPyPI packages on Linux and Windows.
 
@@ -60,7 +58,7 @@ This process is only meant for the project administrators, not users and develop
 
 #. Once the build for the PR passes and the reviewers approve, merge the release branch into ``main``.
 
-#. Upload source and wheel packages to PyPI using ``python setup.py sdist upload`` and ``python setup.py bdist_wheel upload``
+#. Upload the already-built source and wheel packages to PyPI using ``twine upload dist/*``. You will need to have the ``twine`` package installed and set up your ``$HOME/.pypirc`` correctly. You will need to have the appropriate permissions for the ``ets`` organization on PyPI.
 
 #. Make sure that the ReadTheDocs build for ``main`` passes by examining the badge at this `URL <https://img.shields.io/readthedocs/rsmtool/main.svg>`__ - this should say "passing" in green.
 

rsmtool/version.py

@@ -3,5 +3,5 @@
 in one place. Based on the suggestion `here. <http://bit.ly/16LbuJF>`_
 """
 
-__version__ = '8.1.2'
+__version__ = '9.0.0'
 VERSION = tuple(int(x) for x in __version__.split('.'))

setup.py

@@ -49,8 +49,8 @@ def requirements():
                    'Operating System :: Unix',
                    'Operating System :: MacOS',
                    'Programming Language :: Python :: 3',
-                   'Programming Language :: Python :: 3.7',
                    'Programming Language :: Python :: 3.8',
                    'Programming Language :: Python :: 3.9',
+                   'Programming Language :: Python :: 3.10',
                    ],
       zip_safe=False)