How to deal with already-failing espresso tests

[masclot]

Hi,

I'm fixing part of #632, in particular turning the class PromotedStoryListAdapter into BindableAdapter. I already have a modification ready that looks good on manual tests. However, when I run the espresso tests for RecentlyPlayedFragmentTest I get test errors. If I run the tests without my modifications, using the 'develop' branch, I also get the same errors. It is difficult to get a full run having the same errors in both branches because each run has seemingly random timing issues, but if I rerun the tests with timing issues until they pass, I get the same errors in both branches.

I looked for an existing issue for these tests, but I could not find any.

What's the policy for when you encounter such a scenario?

1. Since I get the same errors, I'm not making it worse (at least for what is tested successfully), go ahead with the PR
2. Create an issue about the failing tests, but not wait until it is fixed and go ahead with the PR
3. Create a blocking issue, fix the issue (myself or someone else) in a separate PR and then go ahead with my PR
4. Directly fix the tests in my PR. Note: this could be time consuming given my current knowledge

WDYT?
Thanks!

[BenHenning]

Thanks for this question @masclot.

The current team's policy is (1) with a modification: the PR description is meant to include screenshots of passing Espresso tests, so we generally expect an additional explanation for cases when these screenshots include failed images (where the explanation would say something like, "these tests are already failing on develop with the same error as shown here"). We don't want people to lose too much time to existing failures.

Now with regards to filing an issue, we've tried this in the past however since Espresso tests aren't running in CI they can very easily break (even with the PR description asking for proof of passing tests, it's still easy to miss stuff). I think anything more strongly enforced than the above would require enabling Espresso tests in CI, and that's not something we're going to do quite yet (though its primary blocker is currently being worked on).