WeBWorK 2.20 Release Candidate #2721
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large courses can cause a lot of CPU load when the javascript generates the menus to select users of the course. Also large courses will be hard to navigate with using the menus. By default courses with 200 or more users will not be listed in the drop down menus. If any course is not included, a warning will state which courses have been ignored and a button to show all courses will appear. The ignored courses are still listed in the list of destination courses when copying multiple secrets.
Use compute_unreduced_grade to determine if a problem is correct or not on the student's grade table to highlight correct problems.
Use reduced credit when checking if problem is correct in Grades table.
This has two components, a regular Webwork Authen module and a Mojolicious plugin for SAML2. The plugin implements a SAML2 SP using the Net::SAML2 library. Net::SAML2 claims to be compatible with many SAML2 implementations, including Shibboleth. The intent for this plugin is to replace the old Shibboleth auth that depended on the shibd service which requires Apache's mod_shib. The WeBWorK::Authen::Saml2 authen module's main purpose is to call the Saml2 plugin helper sendLoginRequest() that properly sends the user to the IdP to authenticate. There is a bypass option that allows skipping the Saml2 authen in favour of the authen module after it. So you can, for example, put Basic_TheLastOption after it allow access to the internal username/password auth. It seems to be standard for Mojolicous plugins to be located in lib/Mojolicious/Plugin, so I've put the Saml2 plugin there. Additional detail and configuration instructions can be found in lib/Mojolicious/Plugin/Saml2/README.md The Saml2 plugin will only be loaded if the corresponding conf file exists. Copy conf/authen_saml2.dist.yml to conf/authen_saml2.yml to enable it. 3 settings in the conf are crucial, the idp.metadata_url must be set to the IdP's xml metadata endpoint and a unique sp.cert & sp.signing_key pair should be generated. The example cert and signing_key must not be used for prod under any circumstances. I initially put the config in conf/mojolicious.webwork.yml under a saml2 section but seeing as how other authen modules have their own conf files, I figure it was better to follow that convention. And as there seems to be more work around the yaml config, I've made the saml2 authen conf also a yaml file. The NotYAMLConfig plugin for reading yaml conf files gave me some trouble though, as I didn't want to step on the main conf and had to read the code to figure out how to use just the load conf file functionality. The Saml2 plugin will generate its own xml metadata that can be used by the IdP for configuration. This is available at the /saml2/metadata URL under the default config. Note that endpoints are configurable as I wanted to be able to change them to match shibd endpoints. The Saml2 plugin has its own set of controllers and a router for them. The AcsPostController is the most important one as it handles the SAMLResponse that sends the user back to Webwork from the IdP. The errors aren't the most friendly unfortunately, should probably add a proper 401 error template so users don't see the more scary stacktrace one. Note that unlike shibd, attribute maps are not supported. So you probably have to replace user friendly attribute names like "studentNumber" with URN style names like "urn:mace:dir:attribute-def:studentNumber". You can check your IdP's attribute map xml for the official URN names. Some discussion about alternatives that I tried before settling on the Mojolicious plugin approach: The Saml2 as a plugin idea started after trying out Mojolicious::Plugin::SAML. Mojolicious::Plugin::SAML didn't work out in the end due to two downsides. The major one being a lack of RelayState support. RelayState is the defacto standard way for SPs to store where the user wants to go after returning from a successful auth from the IdP. This is a necessary feature for Webwork as auth to each course is handled separately and we need to know exactly what course the user wants to get into. The minor issue is that it doesn't parse the SAMLResponse attributes for you and I really didn't want to have to muck with xml parsing. Apache mod_proxy using mod_shib and shibd was another possibility. This requires passing shib params through HTTP headers via the use of the ShibUseHeaders setting. Shibboleth documentation says the ShibUseHeaders option should be avoided as they're not confident in the protections against spoofed HTTP header attacks. Running Webwork under mod_perl again so we don't need to use mod_proxy. This resulted in hard to debug issues with a few sections of async code. While I was able to 'unasync' those sections to get the thing running, it feels unmaintainable in the long run, especially if Webwork increases usage of async features.
WEBWORK_ROOT_URL definition in Dockerfiles used two colons before the port when it's supposed to be only 1. Also deleted version line from docker-compose.dist.yml, as I get a warning message that `version` is obsolete.
Mojo::Base's -strict option removed where -signature is present as it makes strict redundant. Use Mojo::JSON's functions instead of JSON. When using WeBWorK::Debug, specify import of debug() function. Remove import of WeBWorK::Debug and Data::Dumper where they're not actually being used. Fix README.md to pass markdownlint inspection.
Reformatted to follow .editorconfig rules.
This fixes some issues with #2511. This pull request is built on that one. That pull request is a nice start, but has some issues. Thanks to @ionparticle for starting this work. One of the most important things introduced in that pull request is the development identity provider via SimpleSAMLphp. That gave me a way to work with this. Although I liked the idea of using a Mojolicious plugin, that approach does not fit into the webwork2 course environment system. So, unfortunately, that had to be changed. So this rewrites the SAML2 authentication module to use the usual approach that all of the other authentication modules use. There is a `authen_saml2.conf.dist` file that should be copied to `authen_saml2.conf` to use SAML2 authentication. All settings for the authentication module are in that file. The primary limitation of the plugin approach was that it is not possible for different courses to use different identity providers. Or for one course to use SAML2 authentication and another only use the basic password authentication (without the need to add a URL parameter). This is something that is expected of the authentication modules, and is desirable for multi-institutional servers. Another problem with the implementation is that is does not work with `$session_management_via = "key"` set. The reason that doesn't work is because the implementation broke the rule of creating and accessing a session before authentication is completed. So this reimplementation uses the database via the "nonce" key hack much like LTI 1.1 authentication does. The previous implentation also used the `WEBWORK_ROOT_URL` environment variable in the code. That is not an environment variable that is defined for webwork2. It is only defined for the docker build, and can not be used in the code. The previous implementation also does not work with two factor authentication. If an identity provider does not provide two factor authentication, then webwork2's two factor authentication should be used. Of course, if the identity provider does provide two factor authentication, then the two factor authentication can be disabled in `localOverrides.conf`. An option to enable service provider initiated logout has also been added. It is off by default, but if enabled, when the user clicks the "Log Out" button, a request is sent to the identity provider to also end its session. The `README.md` file that was with the plugin code has been moved to the `docker-config/idp` directory, and is now purely for instructions on how to use the simpleSAMLphp development instance. The readme also includes instructions on how to set up a simpleSAMLphp development instance without docker. The documentation on how to configure and use the SAML2 authentication module is now all in the `authen_saml2.conf.dist` file. Note that the webwork2 service provider certificate files are no longer directly in the configuration file. Instead file locations are to be provided. It didn't really make sense to have the actual certificates in the configuration file, then write them to a file, which the Net::SAML2 module would read. Furthermore, it would be very easy to add the certificates incorrectly to the configuration file. With the YAML file approach if indentation were not correct, then the configuration file would fail to load.
Any route can now specify the methods that are allowed by adding a `methods` key to the route parameters. The value of the key should be a reference to an array containing the allowed methods. The ACS route is the only route that uses this at this point to restrict to the POST method only.
This option is for the case that the identity provider offers multi
factor authentication, and yet the $saml2{bypass_query} is also
allowed. In this case you would not want webwork2's two factor
authentication to be used when signing in via the identity provider.
However, two factor authentication should be used if the bypass query is
used. Setting $saml2{twoFAOnlyWithBypass} to 1 makes it so that
webwork2's two factor authentication is skipped for users signing in via
the identity provider, but still required for users signing in with a
username/password. If this is set to 0, then webwork2's two factor
authentication will always be required.
Really, the fix has been published in an update pg-codemirror-editor package (actually the bug was in the codemirror-lang-pg package). This just updates the version in the package.json file.
Add course admin tool to manage OTP secrets.
Updates to adding users to newly created courses.
Fix PGML Emphasis in the PG CodeMirror editor.
Hide the "Start New Test" button when acting as another user unless the user has permissions to do so. Fix some logic where the warning about creating new test version wouldn't show with the record_answers_when_acting_as_student permission, and instead the version would be created without warning. Translate the error messages that are created for an invalidSet. Remove the test for invalidProblem, which isn't used in tests. Update error messages to state 'test' instead of 'set'.
When acting as another user, disable the "Start New Test" button instead of hiding it if user doesn't have permission to start a test as a different user. Include a tooltip that states why the button is disabled.
Instead of telling user to click "Back Button", provide a "Cancel" button which returns them to the previous page. This also includes language updates as suggested during the PR review.
Tests, disable start button when acting as another user.
Fix invalid regex by moving hyphen to last character.
Improvements to SAML2 authentication.
This is a remnant from testing the CodeMirror 6 stuff locally, that seems to have stayed there even after uninstalling the local link. This was never supposed to have been committed.
Remove a local link in the `package-lock.json` file.
…user sets that were included in the scoring
Convert the conf/database.conf.dist file to a Perl module.
Database code clean up.
…r-release Finalize the PG CodeMirror Editor for the 2.20 release.
The `MOJO_TMPDIR` attempt was not working. The problem is that the `Mojo::Upload` object is already created before the `WeBWorK::Upload::store` method is called, and so the temporary file location is already determined and the temporrary file saved there. So when the upload is later retrieved it is not where it is expected to be. This does not happen with rather small files because a Mojo::Upload::Memory object is used instead of a Mojo::Upload::File object, and in that case the file is not saved to a temporary file to begin with.
Also fix extraction of zip or tar files that have files inside a directory, but not the directory itself.
… Secrets" page. This page loads quickly regardless of if these courses are filtered out. Furthermore the menus are generally responsive even when these large courses are included. The only case where something is a bit slow is in the case that on the "Copy Single Secret" tab the selected "Source Course ID" is one of these large courses. In that case if you click on the "Source User ID" it takes some time for the dropdown menu to appear and even then it has to be an astronomically large course for that to be slow (more than 20,000 users). However, even if the large courses are included as long as a large course is not selected the menus are quick. It is interesting that the multiple select elements on the "Copy Multiple Secrets" tab are still fast even with astronomically large courses. It is just the single select elements on the "Copy Single Secret" tab that experience a noticeable slowdown. In any case, the point is that there is no reason to filter these courses. Doing so just adds unnecessary steps for the user.
…r own OTP secret.
Fix file uploads.
…eed-tweak Speed up assignments of achievements.
First get all of the necessary achievement data from the database, then process it and print it to the file. Comparing scoring all course achievements for all users in a course with 5000 users shows a considerable speed improvement. It takes more than three minutes for this with the develop branch, and less than 3 seconds with this pull request. Obviously the generated scoring files are identical.
The usual thing (by now with this series of pull request) is done. That is removing database access from loops and reducing to single queries as much as possible. In addition instead of using the Mojolicious tag helpers to render the "earned" checkbox and "counter" text field, direct html is used. I discovered that this renders much faster. With the usual 5000 user test the rendering of the template alone takes 5 seconds after changing a checkbox or counter value and saving, while with the direct html it takes something like 0.2 seconds. I believe that it has something to do with their code to set the values of the inputs that is causing the slow down. I am not sure on that, but I know that the direct html is much faster. That database changes are still the biggest part of the speed improvement here in any case. This page was rendering quite slowly even on initial load before, and that part was fast with the tag helpers. After making the database changes things sped up considerably, but then timing parts of the code when saving revealed the tag helper issue.
Speed up achievement scoring.
…ements Speed up the achievement users page.
…-restrictions Remove the filtering of large courses on the course admin "Manage OTP Secrets" page and fix security vulnerability.
Importing the `default_achievements.axp` file into a course with no achievements and 5000 users and assigning to all users took about 8 minutes with the WeBWorK-2.20 branch. With this pull request the time decreased to about 20 seconds.
…-speed-improvement Speed up the assignment of achievments when importing achievments.
…HTML renderer. This is an alternate for #2734, and implements the workaround suggested by @dpvc in mathjax/MathJax#3370 (comment).
The script now uses the settings in the course environment for external programs and to check the database driver (DBD::mysql or DBD::MariaDB). In order for that to work, first the script checks that all of the needed modules are found, load, and are at the sufficient version. That is except for the database driver module. If any are not found, then the script exits with a message stating that it is unable to continue without the required modules. If all modules are good, then the script checks the database driver module next. For this it "runtime" loads the course environment and its dependencies (as well as a runtime detection of the webwork2 root and lib directories). This is to address issue #2740. In addition the following modules are removed that are no longer needed: * Array::Utils * Email::Sender::Simple - still used but is a dependency of Email::Stuffer and doesn't need to be checked separately * HTML::Tagset * HTML::Template * IO::Socket::SSL * Net::LDAPS * Net::SMPT * Net::SSLeay * PadWalker * Path::Class * Safe - we use our local version (WWSafe.pm) and still check Opcode which it uses * Statistics::R::IO * Template `npm` is added to the list of required applications. Most of these things were noted in issue #1917.
Update the pg modules to include Plots modules.
Implement suggested workaround for erroneous extension loading with CHTML renderer.
update package lock for caniuse-lite
Fix two missed tabs in package-lock.json.
Rework check_modules.pl.
Remove old files (notes and problems) from the `/doc` directory
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It is time for this. We have talked about just using the develop branch, but I think it is nice to have a separate branch for the release candidate. It just keeps things separate in a nice way.
So as usual, re-target pull requests for this branch that you want to get into the release, and I will handle synchronizing those to develop as they are merged.