LTI integration with Peer Instruction not quite right

[barbarer]

When you click on an LTI link for a Peer Instruction assignment it should take you to the student Peer Instruction assignment rather than the general assignment page. Also, it should autograde the assignment and pass the points back to canvas.

[bnmnetp]

We have never supported LTI and PI assignments. I'm not even sure it will work given that the assignment is "controlled" by the instructor.

In addition our LTI integration does not and has never supported automatically sending the grades back to canvas. It has always required the instructor to release the grades.

I can have the LTI launch code check for the assignment type and send the user to the correct page. I'm pretty sure the instructor will have to launch the assignment from Canvas as well, which is a different page than the students.

I did put in code --- very lightly tested -- for automatically scoring PI questions. But again, we don't and cannot at this time support sending the scores for individual questions back to canvas.

[bnmnetp]

Note to self:
Make sure that the assignment_id is embedded in the page for the student interface like it is for regular assignments.

[bnmnetp]

I've added the code to the lti controller to support launching a Peer Instruction assignment. The changes were simple, and I've looked at it carefully, but I have no way to test. So I'll put it out there on saturday morning. It would be great if someone could try it out on saturday to verify.

Additionally, this will only work for in class assignments. I cannot see a way to make it work for async. All I get from Canvas is the assignment id and some security stuff.

[barbarer]

Thanks for doing this. Margaret, can you test this on Saturday? Dr. Barbara Ericson Associate Professor, School of Information University of Michigan

…

On Thu, Aug 8, 2024 at 9:49 AM Bradley Miller ***@***.***> wrote: I've added the code to the lti controller to support launching a Peer Instruction assignment. The changes were simple, and I've looked at it carefully, but I have no way to test. So I'll put it out there on saturday morning. It would be great if someone could try it out on saturday to verify. Additionally, this will only work for *in class assignments.* I cannot see a way to make it work for async. All I get from Canvas is the assignment id and some security stuff. — Reply to this email directly, view it on GitHub <#524 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKOZ7M334EZEG5TIL2CVZU3ZQNZPDAVCNFSM6AAAAABMELL73SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZVHA4DGMJQGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[barbarer]

Yes, I think I can get some helpers to test it with me on Saturday. If that doesn't work out, then would it be fine to wait until Mon or Tues? thanks!

…

________________________________ From: Barbara Ericson ***@***.***> Sent: Thursday, August 8, 2024 10:16 AM To: RunestoneInteractive/rs ***@***.***>; Ellis, Margaret ***@***.***> Subject: Re: [RunestoneInteractive/rs] LTI integration with Peer Instruction not quite right (Issue #524) Thanks for doing this. Margaret, can you test this on Saturday? Dr. Barbara Ericson Associate Professor, School of Information University of Michigan On Thu, Aug 8, 2024 at 9:49 AM Bradley Miller ***@***.******@***.***>> wrote: I've added the code to the lti controller to support launching a Peer Instruction assignment. The changes were simple, and I've looked at it carefully, but I have no way to test. So I'll put it out there on saturday morning. It would be great if someone could try it out on saturday to verify. Additionally, this will only work for in class assignments. I cannot see a way to make it work for async. All I get from Canvas is the assignment id and some security stuff. — Reply to this email directly, view it on GitHub<#524 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKOZ7M334EZEG5TIL2CVZU3ZQNZPDAVCNFSM6AAAAABMELL73SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZVHA4DGMJQGA>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[bnmnetp]

Would strongly prefer to have it tested on the weekend in case there is a problem. I don't want to be impacting people when they are trying to teach their classes.

[maellis1]

Ok, makes sense. I found some students who can simulate using it in class with me via zoom on Saturday afternoon. I can do some trials on my own in the morning once you've made the update. thanks!

[bnmnetp]

OK, this is live now.

[maellis1]

Thanks, it now goes straight to the peer instruction instructor interface for the assignment for me when I pass through the canvas assignment. This afternoon, we will simulate an activity with some "students" and I will report back. Thank you!

[bnmnetp]

Ok, it should work fine. It does all the same processing as for a regular assignment, the only difference is that it has to redirect to two different places for the peer instruction pages. One for the instructor and the other for the students.

[maellis1]

We were able to simulate the class environment. I connected via LTI from canvas as a teacher and three others connected as students and we simulated an activity with "Week 2 day 2" but when I went to grade it they all had zeros.

We are working in book "vtcs2 _ peerplus _ fall24"

So we started fresh with an new assignment from canvas linked to "Week 3 day 1" and began recording. THe experience with that seemed ok but when I went to grade it, they did not receive credit and we did not see all of their votes in the Runestone Grading view. When I released the grades it did pass the 0 back to canvas. We then did a trial without starting over in canvas to double check the assignment settings and in that case they did earn credit. That is exercise "Week 3 day 2". So it seems like something didn't work with the grading when we initiated the peer instruction from canvas. In that book and the canvas course we were using, all three of the others are indeed students ( I cut this out of the video because student IDS were showing). Here are 2 videos of us working. It seems like something doesn't get initiated from grading when we enter via canvas. Can see around timestamp 6:14 in the second video the grading difference between the 2 we entered via canvas vs via the Runestone menu.
part 1:

part 2:

[bnmnetp]

There is nothing about LTI that impacts the grading. The scoring happens 100% as usual on the Runestone side, and then the totals are ultimately passed back.

Did you try running the "autograder" from the instructor interface after the fact to calculate the grades?

It may be that there is a bug in the real-time scoring changes I made that is not working quite right for peer instruction questions.

If you could privately send me the usernames of the students I can inspect the database to see where things might have gone wrong. Is "Week 3 day 1" the name of the assignment??

[bnmnetp]

Never mind, I can see the relevant activity in the database.
It looks to me like everything was captured, but I do see a couple of traceback reports that I'll need to investigate.

[bnmnetp]

I can see that for the Week 3 Day 1 assignment and the Week 2 Day 2 assignment the necessary bits got added to the grading table so we know it is an LTI assignment. -- I can also see that the scores are either null for your user, or 0 for the three students.

My preliminary conclusion is that this probably has nothing to do with LTI and everything to do with my real time grading. The good news (I know you won't call it good news) is that you can still run the old autograder and get the scores as barb has done.

I think I mentioned that the real time scoring of peer instruction was only very very lightly tested. So that is undoubtedly the problem, and I can debug that on my development machine this coming week.

[bnmnetp]

AHA!!!

You need to set which to grade to "all_answers" for peer instruction. You have it set to last_answer.

There is not way to have us grade peer instruction questions for correctness. Only on participation.

[maellis1]

Ok, thanks for looking and sorry about the delay. We were suspicious that we needed to change something in the configuration but then "last_answer" seemed to work for "Week 3 Day 2". It doesn't seem like I can change the configuration and regrade completed work to check this. So, I will create another canvas assignment with LTI to a Runestone PI assignment with "all answers" for the exercise grading and I'll see if we can test it out.

[maellis1]

We tested "Week 4 Day 1" with "all answers" with 1 student and it seemed to work. We will follow up if needed as we practice more this week. Thank you!

[barbarer]

You should be able to change the grading type on each question and regrade the assignment .  Make sure that it is a peer instruction assignment and you set the grading to peer or peer+chat and all answersSent from my iPhoneOn Aug 11, 2024, at 10:48 AM, Margaret Ellis ***@***.***> wrote: Ok, thanks for looking and sorry about the delay. We were suspicious that we needed to change something in the configuration but then "last_answer" seemed to work for "Week 3 Day 2". It doesn't seem like I can change the configuration and regrade completed work to check this. So, I will create another canvas assignment with LTI to a Runestone PI assignment with "all answers" for the exercise grading and I'll see if we can test it out. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[maellis1]

Got it. Changed it to "all answers", and remembered to save :-) . Regraded on Runestone, then Hide the grades and redisplayed them and it worked as expected, passing updated grades back. We will try to test some more to make sure instructors understand timing & cases where students can't answer so we are ready for class. I see the "all answers" option on slide 8 of the slides, so I'll make sure we look back at this when needed for creating PI assignments: https://docs.google.com/presentation/d/1uyTtNEcz3zCuHK8TmCJ9laBzXqy-G6zU/edit?usp=sharing&ouid=117491968079586052813&rtpof=true&sd=true. Thanks!

[bnmnetp]

What if I added a "Submit Scores" button to the PI interface so that at the end of a PI assignment you could click it and if LTI is active for the course we just send all the scores?

I could send the scores after every vote2 but this seems unnecessary and a lot of extra server work for something that is really not that hard for an instructor to press a button. Not to mention the confusion of what I should do when an instructor presses the "start over" button in the middle of a PI session.

[maellis1]

Thanks that would be an improvement. I had written some ideas in #527. I don't think we need it sent after every vote (I guess the issue would be if we ran out of time and wanted to give credit for what we did complete even if we don't make it through the last question?). Right now when we get to the last PI question it still has the "Next Question" button and there isn't a button to say we are finished, so I've just been leaving that page. Are there some use cases where it makes sense to go to the next question after you've gone through all the questions in the assignment? Perhaps if something didn't go correctly or you need to start over?

It would be more convenient to be able to signal that we want the Peer Instruction graded and ported/displayed right during the session vs. needing to go over to the grading tab, so this suggestion sounds like an improvement to me! So when the instructor goes through the last question of the assignment, what options would they have?
1)do nothing
2)hit "Next Question"
3)click a button to "Submit Scores" which would score the assignment and release the grades.

It seems like this would work well for our purposes. In most cases we would use option 3 but maybe sometimes we would purposefully do 1 or 2? I haven't used it enough to be sure. I can imagine there would be some circumstances where we might forget to click that "Submit Scores" button (for example having lines of students trying to talk with the instructor :-)).... would we then be able to return somehow to click it? Or, in that case maybe we'd need to go to the Grading tab and go through the drop downs and click the buttons to grade and release.

@barbarer @pearcej. what would work well for your circumstances based on your previous experiences?

[maellis1]

We experimented more with the grading to make sure we can explain it to our students:

1. When we select "peer" and "all answers" it appears to grade for participation on each question. Som, for example, if they answer the first time but not the second time it is 2.5/5.0. This seems fine. Just making sure we are all on the same page so we can tell students.

2. Also if the students start it from canvas LTI but the teacher has not then it does open for them in Runestone. And if they try to answer it doesn't seem to register (based on instructor view), but a student who answered early on the first one can answer the second time and end up with 5/5 on that exercise. This also seems fine.

3. And if students don't answer the first time they can answer the second time which is fine with us. They earn, for example, 2.5/5.0.

4. It also appears that students can't access the book via runestone and only through canvas under circumstances. This seems good but we aren't sure how it works.

[pearcej]

My classes are small, so while I do use LTI integration for readings, I have only used index cards for Peer Instruction. So, to lighten the workload on my TAs, this will be a big improvement. The individual and the group index cards had to be turned in to get credit, so it makes sense to me that each question must get an answer.

It is NOT actually fine with me if a student does not answer the first time because mistakes are exactly when we are primed for learning. Students are people who largely don't understand learning so not answering because they are distracted or bored and then engaging with only the discussion is not useful for PI to function as intended. I definitely want answers to each for credit, but I wonder if this could be instructor discretion? (Checkbox or something?)

I think if an instructor never launched a question due to lack of time then basically no students should have answered. Is this correct? (I teach solely in person, so this is a real question.) Depending on the answer to this, can we eliminate questions if the number of answers is below a certain threshold?