TryMyUI Proposal: Ensure Ratings Clearly Influenced by Blockstack Bugs Are Not Counted

[polluterofminds]

What is the problem you are seeing? Please describe.

I've seen, in multiple tests over the months, a bug in the Blockstack onboarding pages result in a tester giving Graphite low scores (usually 1's across the board). There is no amount of educating the user about the onboarding experience that will fix errors that fall squarely in Blockstack's control. Scores that are clearly tied to this should not be counted.

In an example from May's results, a user was unable to register a username. The Blockstack Browser just threw an error telling her the username couldn't be registered.

This user was incredibly frustrated. Ultimately, she gave Graphite all 1's and her reasoning was clearly because of the onboarding bug.

How is this problem misaligned with goals of app mining?

Unfairly penalizing apps for something beyond their control will surely drive developers away from Blockstack. As long as the onboarding is in control of Blockstack, issues tied to this should be thrown out of scoring.

What is the explicit recommendation you’re looking to propose?

An audit period to review TryMyUI videos to see if there were any clear low scores tied to Blockstack's infrastructure and not the app itself. Then allow for app devs to request a review from Blockstack.

** Describe your long term considerations in proposing this change. Please include the ways you can predict this recommendation could go wrong and possible ways mitigate.**

The obvious issue is that this will add time to the release of scores and payouts. However, I think it's worth it considering the impact low scores can have.

[GinaAbrams]

There has been some email back and forth on this. @timstackblock said:

I think this issue above is related to issue 1764

This was fixed in PR

The fix is currently merged into the develop branch and should go out in the next release.

[GinaAbrams]

Also tagging @markmhx who has reviewed the TryMyUI videos and onboarding in the past.

[polluterofminds]

Thanks, but I don’t think that’s the cause of this. Unless TryMyUI users are hitting the free name limit. I would expect that it would be next to impossible for that to happen.

[markmhendrickson]

@jehunter5811 Mind providing the exact TryMyUI video IDs here so I can audit?

I reviewed all the videos from March across the top 25% of App Mining participants (including Graphite) and filed the following issues based on the problems I saw with authentication at the time:

• Ensure that ID creation screen is fast across devices blockstack-browser#1900
• Redirect to app instead of homepage when user selects "back" on seed screen during onboarding blockstack-browser#1901
• Back button doesn't always work during onboarding flow depending on step blockstack-browser#1905
• Resolve confusion about email at end of onboarding flow blockstack-browser#1899

Those didn't relate, though, to username registration failures per se. So I'm not sure we're tracking the problem you've seen yet, and it doesn't seem any of the above issues are blocking a large number of users.

So perhaps we ought to track down just what blocking bugs may be live at the moment before deciding whether a recurring audit period is necessary in general. If we find that more are on the loose than we've been able to catch, then it may be a prudent move.

[polluterofminds]

@markmhx Here's the video from the user this past month who couldn't create an ID: https://www.trymyui.com/use_tests/43029

[GinaAbrams]

hey @trotolo we chatted about this earlier this week, can you comment on this? 🙏

[polluterofminds]

@markmhx I copied the wrong url for the tester in question. I gave you the full folder URL. This is the tester ID: 484112

[markmhendrickson]

Thanks @jehunter5811. I've confirmed this is the same bug currently being tracked and resolved with stacks-archive/blockstack-browser#1923.

[markmhendrickson]

I've confirmed this bug was also exhibited in the above video: stacks-archive/blockstack-browser#1900

[stackatron]

Spoke with Justin on this and we feel the best specific proposal is a review period, this specific proposal is captured here: #114

Closing this issue.