Chrome extension

[ocram]

We should explore what a browser extension for Chrome could do:

• High priority
  • Playing media with filters applied (fetched via API (probably JSON over HTTP))
    • Netflix (if possible)
    • Amazon (Prime) Video (if possible)
• Low priority
  • Contributing and annotating media

[jacob-willden]

Thanks for maintaining this service. It has great potential, and I would love to help in any way I can. One place we could look to start exploring how we could create a Chrome Extension for MovieContentFilter could be the Github repository for the non-profit Play It My Way. The creator initially created the repository for a Desktop app called Sensible Cinema, but has since shifted focus to the website and Chrome Extension.

Here is the folder in the repository of interest, and the subfolders "chrome_extension" and "kemal_server" are of particular interest. We could look at the code provided in these folders to see how the software could be adapted for MovieContentFilter.

The service works with Amazon and Youtube, though if we wanted to expand the software to work with Netflix, we would need to address an issue with Netflix's terms of use that say "no injection of code," which is how the Play It My Way extension provides the filters. More info on the issue here.

The license for the repository is listed here (some parts of the license specifically refer to the older Desktop app).

What does everyone else think?

[ocram]

Thanks, @cambersome!

This sounds great. It seems this could be done without too much of a hassle by someone who has prior experience developing Chrome apps and who can also look at that repository for ideas.

Since any extensions should be developed and maintained in separate repositories anyway, and the code would not depend on other repositories from this project, except for its usage of public APIs that we’ll expose, the license being LGPL will certainly be acceptable and won’t cause any problems.

[jacob-willden]

I been looking through the Play It My Way extension code to gain some insights into how we could filter various streaming services. I also just started experimenting the other day to figure out how to turn control both the audio and video channels, and how to skip forward, for Amazon Video, Netflix, and Disney Plus (because I would love to filter Disney Plus with this extension as well). Thus far, I figured out a few things.

1. Amazon, Netflix, and Disney Plus videos can be muted with the following line of Javascript (entered in the browser console):
   document.querySelector("video").volume = 0;
   And the videos can be unmuted by setting the volume property back to what it originally was.

2. Amazon, Netflix, and Disney Plus videos can have their video channel turned off (resulting in a blank screen, but the audio continues and the video controls are still visible) with the following:
   document.querySelector("video").style.visibility = "hidden";
   Then by setting the visibility property to "visible", the video channel can be turned back on.

3. On Amazon and Disney Plus, we can skip an arbitrary amount of time forward (say 10 seconds for this example) using this:
   document.querySelector("video").currentTime = document.querySelector("video").currentTime + 10;
   This is much trickier to accomplish with Netflix, but after some searching, I found a longer snippet that should do the job (currentTime is in milliseconds):
   var videoPlayer = netflix.appContext.state.playerApp.getAPI().videoPlayer; var playerSessionId = videoPlayer.getAllPlayerSessionIds()[0]; var player = videoPlayer.getVideoPlayerBySessionId(playerSessionId); var currentTime = player.getCurrentTime(); player.seek(currentTime + 10000);

[ocram]

That’s awesome – thanks a lot, @cambersome!

These are indeed all the actions that we need. The other part is, of course, retrieving the necessary information that helps us know when to perform these actions. That’s basically two things: the title of the movie or TV show (and perhaps other information) that we need in order to identify it, and the current position, for which your solution from (3) can help as well. We would need to monitor the playback position, or repeatedly retrieve it, or schedule events.

[jacob-willden]

No problem, @ocram! After reading your response, I've done some more digging in the Play It My Way code. I'm not sure about how to fetch the work title yet, but there's a function that would allow us to retrieve the URL, named currentUrlNotIframe(), as a more reliable alternative to just using window.location.href because of iFrames.

One of the functions that runs when the extension loads runs the command setInterval(checkStatus, 5). This means the checkStatus() function will run every 5 milliseconds. This function includes getting the currentTime and checking it with the timings for the filters, among other commands. We could do the same thing.

I'll see if I can find out more about fetching other information for identifying movies or TV shows.

[ocram]

Thanks!

As for identifying the movie or TV show, we’ll probably need to fetch the title from <title>...</title> or <meta property="og:title" content="..."/> in the source HTML. Apart from that, I can only imagine mapping Netflix/Disney/Amazon URLs to internal entries, which would certainly be some hassle – although it’s perfectly precise then.

Calling setInterval(..., 5) to repeatedly check if a filter needs to be applied makes sense and will work well. I had just thought that there may have been a more elegant solution. We probably don’t need 5 – something like 10 or 25 will be sufficiently precise, I guess.

[jacob-willden]

Thanks for the suggestion. After looking through the HTML on some different videos on the streaming services, here's what I found.

For Amazon, the <title> tag in the <head> does contain the work title (e.g. "Watch (title) | Prime Video" or "Amazon.com: Watch (title) | Prime Video"). However, if it's a TV show, the individual episode name is not included with that attribute, but Play It My Way may have some functions to help with that. There's also a <meta name="title"> tag with a content value identical to the value for the title tag.

For Netflix, the <title> tag simply has the value "Netflix", and I couldn't find any <meta property="og:title"...> or <meta name="title"> tags. I did find a property in the Netflix API that would allow us to get the unique ID for a work, though that ID is also included in the URL, so that may not be very significant.

Likewise for Disney Plus, the <title> tag simply has the value "Disney+ | Video Player". It does have a <meta property="og:title"...> tag, but it's value is identical to the value of the <title> tag.

So fetching the title may work for Amazon, but mapping the URLs to the internal entries may be the best way for Netflix and Disney Plus.

---

As far as monitoring the playback position, I actually did find an alternative solution. There's another open-source filtering project called "VideoSkip", which has both a standalone filter player (for legal copies of movies/TV show episodes stored on one's computer) and a Chrome extension for streaming. The extension was created with the intention to work on virtually any streaming service (it's been tested on a handful of sites so far), and is also meant to be available in any language, provided there are enough community translators. I think its extension repository will come in real handy as the Movie Content Filter extension is developed.

The extension monitors the playback position using the following code:
if(myVideo) myVideo.ontimeupdate = function() {...}

The variable myVideo is the video element that we would get via querySelector. Within the anonymous function above, a for loop runs through all the work's filter tags to check if the current time is within the start time and end time of a tag. Then it checks the tag's action and sets it accordingly.

For reference, the function can be found here on line 189.

[ocram]

These are excellent findings again. Thank you!

Regarding work identification, it’s good to know that we’ll have to rely on URL/ID mapping. Title comparisons could have been a shortcut, but would have been unstable, fuzzy and likely to break, anyway. Adding a Netflix/Disney/Amazon URL to an (existing) entry is fine and quick to do.

As for monitoring the playback position, the timeupdate event is exactly what I had in mind. It’s a bit more elegant than scheduling a timer/interval and querying the position every time. The docs say that the “event frequency is dependant on the system load, but will be thrown between about 4Hz and 66Hz (assuming the event handlers don't take longer than 250ms to run)”. With 4Hz, we wouldn’t have the accuracy we want, but perhaps that will rarely happen. It’s questionable if we would get the alternative running every 5ms, as Play It My Way tries to do, anyway. Perhaps that’s just their way of expressing “as often as possible”.

[jacob-willden]

Sorry again for the late response. Very interesting information on the timeupdate event. Definitely will want to keep that in mind.

After doing some more looking through the VideoSkip code, I found that when the user loads a filter file, the file is parsed and converted to an array of objects. Each of the objects in the array is a filter tag, with the start and end time, the category, and the channel (whether audio, video, or both). The filter severities are based on the extension settings that are set by the user.

I think the hardest part with the Movie Content Filter extension will be figuring out how to allow it to access the filters on the MCF website. VideoSkip avoids this issue entirely by having users download filter files from the website and load the filters manually. Play It My Way appears to store the filters on an SQL server and uses the web framework Kemal, which is written with the language Crystal. This seems to be how Play It My Way automatically loads filters based on the URL. I'm not sure yet how the communication works between the SQL server, the web framework, and the Play It My Way extension, but I'm very curious about it.

What are your thoughts on this?

[ocram]

Thanks, and no problem.

When it comes to plugin development, you can really assume that you have all filter information readily available – as long as that plugin system allows you to fetch text from URLs to make API calls.

The MCF website would expose an API that offers all necessary filter information (probably in JSON format) for a given MCF ID or Netflix/Disney/Amazon identifier. That means, for example,

https://www.moviecontentfilter.com/api/v1/works/id/6e7458723b9b

would return all information you need for the entry with ID 6e7458723b9b from the MCF website, whereas

https://www.moviecontentfilter.com/api/v1/works/netflix/566536d6f6b1

would return all information you need for the movie or TV show identified by 566536d6f6b1 on Netflix – and so on.

That’s a given. During drafting and development, one can thus put some dummy data locally, and later fetch that data from the API instead. If you want to know what data will be available, just look at the downloadable filter for an existing entry and pick the MCF format.

[jacob-willden]

Sounds good!

I think I have enough information now that I could attempt to create a very basic version of the extension, with some dummy data in JSON format for the filters. Since the code from Play It My Way is under the LGPL license, and therefore compatible with the GPL license VideoSkip is under (provided the adopted Play It My Way code then falls under the GPL license instead), I can combine code from both to create the basic extension. I'll get started and keep you updated.

[ocram]

Awesome, sounds great!

As said earlier, you will get any assistance you need with this. But the API to fetch the filters for a specific movie or TV show from the backend will probably be the only point where the two applications need to talk to each other.

Looking forward to your updates.

[jacob-willden]

Awesome!

As you may have noticed, I changed my username recently. Just wanted to let you know.

I have now built a very basic version of the extension, with some dummy values. The filters work exactly as intended on Netflix and Disney Plus, and Amazon works except the currentTime at the beginning of the video isn't always 0. I think Play It My Way may have a solution for this, so I'll want to dive in deeper to the code.

In the meantime, the repository is available here. In order to comply with the terms of the GPL license, I included the full source code from both Play It My Way and VideoSkip, so the actual extension is in the "movie-content-filter-chromium-extension" folder.

[ocram]

That sounds great. What a promising start!

There is no GUI, however, right?

So would this just automatically apply filtering (when available) to any supported service and any supported movie or TV show? Basically, I would be fine with “no GUI”, but we might at least need one button to toggle the filtering on/off. What do you think?

(I know this is not finished yet, you have more ideas, and more things are to come. Just wanted to give some early feedback.)

It appears there are a lot of little workarounds (in the original projects) – but that’s fine, whatever we need to make this work. I was just surprised and would have expected the video controls to work in a more standard-compliant way across services.

Apart from that, we would need some settings menu, or at least one text input field to identify which user’s settings to load. With some ID or token, we could load a user’s settings from the website. Otherwise, you would have to implement the filter preferences (categories and severities) all in the extension, which I think doesn’t make sense, as it’s more work and duplicates the user’s settings.

Thanks for your work here!

[jacob-willden]

No problem, and thanks for the feedback! Yes, there’s no GUI yet, and I completely agree with your suggestions. I can certainly add a popup for when the extension icon is clicked, and that popup could include a checkbox to toggle filtering on/off and an input field for users to enter an ID or token to load the user's preferences. I will start to look into how to implement them.

[ocram]

That would be great, and sufficient to make this work with account-specific filtering. Thank you!

[jacob-willden]

I have now implemented a popup with a checkbox that enables/disables filters now, and a text field for entering an ID or token associated with the user preferences. In order to store the ID for the next time the extension is to be used, so the user doesn't need to enter the ID every time, I used the chrome.storage API (Firefox has an equivalent called storage.storageArea). This storage is unencrypted (as is Firefox's storage), so we'll want to make sure that the only the ID and associated filter information is stored in it, and not any confidential user information such as emails.

To allow for internationalization, I also added support for all the Amazon top-level domains (for domains such as amazon.de and amazon.co.uk) except for Amazon China (amazon.cn), because China doesn't have Amazon Video due to government regulations.

Finally, in order to comply with the United States Family Entertainment and Copyright Act of 2005, the extension now displays "a clear and conspicuous notice at the beginning of each performance that the performance of the motion picture is altered from the performance intended by the director or copyright holder of the motion picture" if filters are enabled. Once we can start detecting if filters are available for the specific video, I can then program the extension so it will only show the notice when filters are both available and enabled.

I'm still trying to figure out the weird timings that Amazon videos sometimes have. I think it has to do with advertisements being shown before the videos, so I'm looking into methods for detecting such advertisements and when they end.

I have also been thinking about my past concerns regarding Netflix's Terms of Use forbidding the insertion of "any code or product or manipulate the content of the Netflix service in any way," which has prevented Play It My Way from providing filtering to Netflix. Play It My Way has also backed down from providing Hulu support for similar reasons (on line 31-32 of the extension's content script, available here).

However, it says in the Family Entertainment Act that not only is "the making imperceptible, by or at the direction of a member of a private household, of limited portions of audio or video content of a motion picture, during a performance in or transmitted to that household for private home viewing, from an authorized copy of the motion picture" protected, but also "the creation or provision of a computer program or other technology that enables such making imperceptible and that is designed and marketed to be used, at the direction of a member of a private household, for such making imperceptible, if no fixed copy of the altered version of the motion picture is created by such computer program or other technology."

So it appears that Netflix can't really forbid the insertion of code that manipulates the content of their service, if that code is used for filtering purposes as described in the law above. Plus, Enjoy Movies Your Way has been providing filters to Netflix titles without issue, as has VidAngel (except for Disney and Sony-owned titles, but the court case appeared to have nothing to do with the insertion of code on Netflix), and ClearPlay is expanding to Netflix too now. So as long as we follow the requirements above, we should legally protected under United States law. I'm not certain if there are any other specific requirements from other countries though. I expect that would be worth investigating.

[ocram]

Thank you so much, @jacob-willden, this is excellent news again!

The popup with the checkbox and text field is absolutely the only configuration we need, as long as we can make API calls via HTTP GET when a movie or TV show starts (and the filter is active). That would allow us to use the stored token or ID to retrieve the user’s preferences and the necessary filters for the current title.

You could talk about multi-user functionality later. Perhaps someone wants a “Mom” and a “Son” account, or “Older child” and “Younger child”, but that can (and probably should) be done later. And perhaps separate user accounts of the operating system or separate browser profiles is the better solution.

The storage being unencrypted is not a problem as long as it’s protected against accesses by other apps, extensions or websites. Our threat model is not a thief running away with the computer and extracting secret information from our storage, but some software (e.g. another extension) running on the same computer and accessing the information stored by our extension. But I can’t imagine that would be possible, right?

Support for other Amazon TLDs is great, and important. Thanks! If that’s all you need to make it work, i.e. no script changes necessary – that’s perfect.

Your implementation (and later improvement) of how to comply with the law regarding that notice sounds perfect.

The exclusion of Netflix and Hulu would absolutely be a pity. I don’t know if it’s avoidable. Perhaps one can defer the decision of whether to continue (and thus breaking the ToS) to the user. But we would still be facilitating it then. Not sure. It definitely seems you wouldn’t be breaking any laws (as a user), but only the ToS of a single service that could ban you if detected at all. If the user is sure they want to do that and take the risk, and if other (large) services are doing it as well … I would not expect many other countries to have stricter rules than the US there. Ultimately, a legal analysis by professionals is certainly the best option, as soon as warranted.

[jacob-willden]

The popup with the checkbox and text field is absolutely the only configuration we need, as long as we can make API calls via HTTP GET when a movie or TV show starts (and the filter is active). That would allow us to use the stored token or ID to retrieve the user’s preferences and the necessary filters for the current title.

Perfect! Sounds great.

You could talk about multi-user functionality later. Perhaps someone wants a “Mom” and a “Son” account, or “Older child” and “Younger child”, but that can (and probably should) be done later. And perhaps separate user accounts of the operating system or separate browser profiles is the better solution.

Cool ideas!

The storage being unencrypted is not a problem as long as it’s protected against accesses by other apps, extensions or websites. Our threat model is not a thief running away with the computer and extracting secret information from our storage, but some software (e.g. another extension) running on the same computer and accessing the information stored by our extension. But I can’t imagine that would be possible, right?

Another Chrome extension on the same computer could definitely access any data in chrome.storage, and another Firefox add-on could likewise access any data in Firefox's storage area. Currently, what is stored is a boolean value representing if filters are turned on or off, and a string value associated with the user's filtering preferences. For example, if the user checks the box to turn filters on and enters "PmrqC" for the ID, then the extension will store the following JSON objects in chrome.storage:

{mcfFilterOn: true}
{mcfPrefsID: "PmrqC"}

These objects would be accessible to any other extension. If access to the ID only allows one to access the filter preferences themselves (and not other information such as usernames, emails, etc), then maybe it isn't a security issue. But if we do think this would be a security risk for the users, then maybe we can find a different way to store the information so the users don't need to reenter the information every time.

Support for other Amazon TLDs is great, and important. Thanks! If that’s all you need to make it work, i.e. no script changes necessary – that’s perfect.

I'll want to do some testing with the other TLDs to make sure, but I expect no other script changes will be needed once I figure out the Amazon timing issue. I'm actually talking with Roger Pack (the Play It My Way creator) in the repository discussions about how his extension detects advertisements, which could help with solving the timing issue.

The exclusion of Netflix and Hulu would absolutely be a pity. I don’t know if it’s avoidable. Perhaps one can defer the decision of whether to continue (and thus breaking the ToS) to the user. But we would still be facilitating it then. Not sure. It definitely seems you wouldn’t be breaking any laws (as a user), but only the ToS of a single service that could ban you if detected at all. If the user is sure they want to do that and take the risk, and if other (large) services are doing it as well …

It's strange that such statements are in their ToS, since they apparently go against consumer rights laid out in United States federal law, but maybe the authors of the ToS didn't know about such law. I expect that if Netflix or Hulu were to terminate user accounts because they were using our extension, they would only be hurting themselves financially since the users would no longer be paying for the subscription. All the same, I definitely agree that we should let users know about the risk of filtering titles on Netflix, Hulu, etc, so they can decide if they still want to use the extension, and let them know that larger filtering services (such as Clearplay, VidAngel, and Enjoy Movies Your Way) do provide filtering for one or more of these streaming services.

Another interesting perspective on the issue is a legal notice that is part of the VideoSkip Chrome extension description:

"Content copyright owners and distributors are hereby informed that users and developers of this software are exercising their right of free speech, guaranteed by law in many nations, by voluntarily refraining from seeing or hearing content without modifying said content in any way. Legal action that ignores this notice will be considered harassment and infringement of basic rights, and prosecuted according to the law" (linked here).

Food for thought.

I would not expect many other countries to have stricter rules than the US there.

I wouldn't expect so either. Copyright rules are generally very strict in the US compared to most other countries.

Ultimately, a legal analysis by professionals is certainly the best option, as soon as warranted.

Agreed.

[ocram]

Another Chrome extension on the same computer could definitely access any data in chrome.storage, and another Firefox add-on could likewise access any data in Firefox's storage area.

I believe (and hope) that is wrong.

{mcfFilterOn: true}
{mcfPrefsID: "PmrqC"}

Looks perfect.

These objects would be accessible to any other extension.

If that was the case, extensions would also have (naming) conflicts between their individual storage entries. I can’t imagine that’s how it is designed. Instead, I think chrome.storage works like localStorage or cookies on the web, which are private to each site.

And if extension storage is private to each extension, which I expect, it would be exactly what we need, and no changes would be required.

I'll want to do some testing with the other TLDs to make sure, but I expect no other script changes will be needed once I figure out the Amazon timing issue. I'm actually talking with Roger Pack (the Play It My Way creator) in the repository discussions about how his extension detects advertisements, which could help with solving the timing issue.

That’s great, thanks!

I expect that if Netflix or Hulu were to terminate user accounts because they were using our extension, they would only be hurting themselves financially since the users would no longer be paying for the subscription.

Agreed, so the practical relevance of these parts from the ToS may be limited.

All the same, I definitely agree that we should let users know about the risk of filtering titles on Netflix, Hulu, etc, so they can decide if they still want to use the extension

Yes, that would be the ethical thing to do, and put us on the safe(r) side.

Another interesting perspective on the issue is a legal notice that is part of the VideoSkip Chrome extension description

If experts agree that this is valid and provides value/safety, we should include it as well, yes.

[jacob-willden]

I now think I was incorrect when I said that other extensions could access the data, which I based off of only one post I found online. I created a test extension that tries to access mcfFilterOn and mcfPrefsID the same way the filter extension does, but it returned undefined rather than the objects. I have also found other posts that support what I discovered with the test extension, including this:

"The local storage [for an extension] can't be accessed or hijacked by another website/extension and thus is safe to use for storing session keys or access tokens. Of course a user can open the dev tools and read that token but that is not a security issue as he is the owner of that information anyways. A token does not have to be encrypted. What you should not store inside the local storage are e.g. passwords" (linked here).

So it looks like another extension can't access the data after all. What a relief!

Sorry about all the confusion, I'll be more thorough in my research next time. Thanks for all the feedback!

[jacob-willden]

Exciting news! I've figured out the Amazon timing issue, so now the timing for the filters are the same across all 3 services. The issue had to do with video advertisements being counted as part of the video duration, so I was able to create a way for the extension to get the video's actual duration, without the time from the advertisement(s). It appears that the ads are always a round number in seconds (usually exactly 16 seconds or 32 seconds), so I can determine the real duration with millisecond precision.

I have also been doing research about performing API calls via HTTP GET to get the user's preferences and the filters for the title, as you mentioned before, and it looks like we can definitely do that with Chrome/Firefox extensions, by using fetch() (though XMLHttpRequest() would also work too). This way, the preferences and filters can be stored on the website, and when the extension sends a fetch request for either, the website can respond with a JSON object (containing either the preferences or the filters for the title), which the extension can then use.

[ocram]

So it looks like another extension can't access the data after all. What a relief!

Great news, very happy to hear this – and thanks for testing this directly as well!

I've figured out the Amazon timing issue, so now the timing for the filters are the same across all 3 services. The issue had to do with video advertisements being counted as part of the video duration, so I was able to create a way for the extension to get the video's actual duration, without the time from the advertisement(s). It appears that the ads are always a round number in seconds (usually exactly 16 seconds or 32 seconds), so I can determine the real duration with millisecond precision.

That’s awesome. Thank you! Didn’t even know Amazon had this (ads being woven into the actual video stream). It’s great to see that the magic t - 10 for Amazon could be removed now :)

I have also been doing research about performing API calls via HTTP GET to get the user's preferences and the filters for the title, as you mentioned before, and it looks like we can definitely do that with Chrome/Firefox extensions

Sounds good. And personally, I wouldn’t have any preference for either XMLHttpRequest or fetch.

This way, the preferences and filters can be stored on the website, and when the extension sends a fetch request for either, the website can respond with a JSON object (containing either the preferences or the filters for the title), which the extension can then use.

Perfect!

It seems your extension is now becoming more and more complete :) You are exploring whether you want to add more video services, is that correct? Any candidates?

[jacob-willden]

Sorry for the late reply! Life's been busy the past few days.

Sounds good. And personally, I wouldn’t have any preference for either XMLHttpRequest or fetch.

Good to know. I'm thinking of using fetch, mostly because it's more developer-friendly, and is supported on all Desktop browsers except for Internet Explorer (and I wasn't planning on trying to develop for Internet Explorer anyway, especially especially since it's being phased out and has been replaced by Microsoft Edge as the Windows default browser).

It seems your extension is now becoming more and more complete :)

It's really exciting how far it's come!

You are exploring whether you want to add more video services, is that correct? Any candidates?

Yes, I am thinking about several different streaming services the extension could be expanded to, including IMDb TV, Apple TV+ (since I still have a free subscription to it for a while longer) and Hulu (once I can get a test account for it).

Edit: I was thinking of adding Vudu to the list too, but I found out that there's a weird stipulation in its ToS that prohibits linking "to any page on the Site other than the home page". So I'm not sure if I'm going to add support for it for now.

[jacob-willden]

By the way, here is how the JSON objects are stored in the extension.

An ID and the associated filter preferences are stored like this (0 means filter nothing, 1 means filter high severity, 2 means filter medium and high severity, and 3 means filter any severity):
{"id": "PmrqC", "advertBreak": 3, "consumerism": 1, "productPlacement": 0, ... "weapons": 2}

Each of the filter tags are stored like this (in an array):
{"startTime": 10.553, "endTime": 15.217, "category": "weapons", "severity": 1, "action": "skip"}

I also include an "enabled" value for each of the tags as well, but that's just to control whether the tag is active or not based on the preferences, so that value would not need to be included in the response to the fetch request.

If you have any other questions, feel free to let me know! I'll continue working on improvements for the extension as my schedule allows, and I'll keep posting about any new major features or updates. Looking forward to future developments!

P.S. I'm also considering adding support for Youtube (since both Play It My Way and VideoSkip work on the service) and maybe Vimeo too, in addition to the other candidates I mentioned above. Thanks again for all your amazing support!

[ocram]

Thank you!

Support for Apple TV+ and Hulu would be awesome. As for YouTube and Vimeo, I didn’t ever think about these two, because the project had always been focused on, well, “real” movies and TV. But anyway, both would certainly be nice to have as well, and their potential could be greater than anticipated.

Regarding the data structure, the website stores annotations of filterable content like this:

category = (nudity | violence | ...)
severity = (low | 1 | medium | 2 | high | 3)
channel = (video | audio | both)

And preferences for content that users would like to have filtered like this:

category = (nudity | violence | ...)
severity = (none | 4 | high | 3 | medium | 2 | low | 1)

So that’s essentially the same format that you have – only that the numeric values for the severity levels are reversed, with 1 = low and 3 = high. I think it would make sense to keep the same format between both projects for simplicity. And then none (which only exists for preferences, not for annotations) should be 4 as opposed to 0, or perhaps also null, so that we can continue to use simple numeric comparisons with annotation.severity >= preference.severity.

Making the preferences available to the extension is one thing the API has to do. But the other thing is making the annotations available. For that, we need one step in between, as we discussed earlier: Mapping movies and TV shows to releases on Netflix, Amazon, etc.

A movie or an episode of a TV show has a unique ID. With that ID, you can get all the annotations of filterable content available. I would propose we map that ID to the following new fields:

service = ("netflix", "disneyPlus", "amazonPrimeVideo", ...)
region = ("us", "uk", "br", ..., null)
url = e.g. "https://www.netflix.com/title/80057281"

It seems this would be sufficient to map any URL that you may encounter with a streaming provider to the list of annotations that we have, and that the extension needs.

[jacob-willden]

Sorry for the late reply again. I think your proposal for mapping the annotations will work great!

As for YouTube and Vimeo, I didn’t ever think about these two, because the project had always been focused on, well, “real” movies and TV. But anyway, both would certainly be nice to have as well, and their potential could be greater than anticipated.

To be honest, I actually wasn't thinking of them either, until I found out that Youtube was supported on Play It My Way (mostly for feature films that are available there), and that Vimeo is another place that many independent creators distribute their short films there. If I do end up adding support for them, I'll probably do so after adding support for the other streaming services.

So that’s essentially the same format that you have – only that the numeric values for the severity levels are reversed, with 1 = low and 3 = high. I think it would make sense to keep the same format between both projects for simplicity. And then none (which only exists for preferences, not for annotations) should be 4 as opposed to 0, or perhaps also null, so that we can continue to use simple numeric comparisons with annotation.severity >= preference.severity.

Thanks for the suggestion! That actually appears simpler to implement than what I currently have for the extension. So 1 = filter everything, 2 = filter medium and high, 3 = filter only high, and 4 = filter none, is that right?

[jacob-willden]

Update: I successfully changed the numeric values for the severity levels. The extension is working for Apple TV Plus as well now, and I'm testing for Hulu next.

I also discovered that when a user navigates from one episode to the next, the extension does not yet apply the annotations on the next episode like the first one. I am working on fixing that too.

[ocram]

Awesome, thank you!

So 1 = filter everything, 2 = filter medium and high, 3 = filter only high, and 4 = filter none, is that right?

Yes, sorry, that was indeed correct.

The extension is working for Apple TV Plus as well now, and I'm testing for Hulu next.

That’s great!

I also discovered that when a user navigates from one episode to the next, the extension does not yet apply the annotations on the next episode like the first one. I am working on fixing that too.

You are right, this is important. Thanks for looking into this!

I think it’s finally time for the server-side API to appear. I hope I can start working on this within the next 3 weeks.

[jacob-willden]

Update: I finally was able to fix the problem from before about the extension not applying the annotations after going to another episode.

As I have been figuring out Hulu, I found another section in the Terms of Service that we may possibly want to be aware of. It prohibits using "technology or other means to access, index, frame or link to the Services (including the Content) that is not authorized by Hulu" (section 3.11). However, I know about websites that do link to content on Hulu and other streaming services, like IMDb and The Movie Database (which Enjoy Movies Your Way uses). So it may not be an issue at all, but I wanted to mention it just in case.

I'm also going to start on internationalizing the extension so other languages can be added.

Besides connecting the extension with the website API for the annotations and user preferences, I think another thing we can discuss is how users can contribute filter annotations. The Clearplay extension connects with the user's account on their website via Auth0, so maybe we could implement something similar?

[jacob-willden]

Another Update: Hulu is fully supported now (except for turning off filters during ads, which I won't be able to test until my free trial ends). Sorry for such a long delay, life has been super busy!

[ocram]

Likewise – sorry!

Thanks for your work, also from May and before. And for looking at the ToS for Hulu.

The very first thing the website will need, in any case, is the mapping between streams and filters on the site. I mentioned that earlier (above), but looking at it now, it seems more fields are required:

# works_to_streams (maps records for movies and TV shows on the site to URLs from streaming services)
work_id = the internal ID of the movie or TV show
service = ("netflix", "disneyPlus", "amazonPrimeVideo", "hulu", "hboMax", "appleTvPlus", ...)
region = ("us", "uk", "br", ..., null)
url = e.g. "https://www.netflix.com/title/80057281"
start_time = time when actual content starts, for synchronization, as explained in "Download" section on website, and as required in "Contribute" section
end_time = time when actual content ends, for synchronization, as explained in "Download" section on website, and as required in "Contribute" section

From your perspective, is that enough to get what you need in the extension? With only the url, or with the tuple (service, region, url), when needed in ambiguous cases, you should be able to retrieve all information and filters for a specific movie or TV show.

The next part then is, of course, adding a way to contribute that data on the website. If I annotated a movie or TV show from a DVD version, but the Netflix version you’re about to watch has an intro that is 10 seconds longer, everything would be off. So we’d continue to need those times for synchronization. At least, when entered once, this should be valid for every single user watching from that source.

[jacob-willden]

Yes, those fields should be all that's needed. I'm glad you mentioned having a start_time and end_time for synchronization, as I just encountered another great example where they would be required (along with your DVD and Netflix example). I was testing some filters for an episode of a show on 2 different services, and I noticed that while the episodes start in sync, they slowly get out of sync until there's an 8 second difference by the end of the episode. So having those variables is definitely essential. I'll start working on trying to implement those.

Contributing the data is going to be interesting too. While most of the interface for the frontend for contributing is already in the VideoSkip code (which I can easily modify to suit our needs), there's also authenticating the users so only those who are logged in can contribute. I'm thinking of implementing this authenticated connection between the extension and the website by using the identity API for the extension, which would use OAuth2 to get user data from the Movie Content Filter website (more information about the API is here). This could also be how we fetch the filtering preferences for users, which I expect would be more intuitive for them than copying an ID (because they would only need to log in).

[ocram]

Thanks!

That’s another perfect example, you’re right. One service must be playing back the episode slightly faster, which easily adds up over the entire span, or have additional scenes or later cuts. I don’t think you have to implement this in the extension, however: The website (and its API soon) are already doing all that. We only need the start and end time stamps once, and that is part of the “contribute” experience, not the “view” experience.

To be honest, I think we’ll have to start with an ID/secret/key being copied to the extension. Sorry!

Adding the “contribute” experience as well would be awesome, of course. But the most important thing is viewing. If we have got this working with the API soon, that’s a great step forward already.

[jacob-willden]

I don’t think you have to implement this in the extension, however: The website (and its API soon) are already doing all that. We only need the start and end time stamps once, and that is part of the “contribute” experience, not the “view” experience.

Ah, perfect! Thanks for letting me know.

To be honest, I think we’ll have to start with an ID/secret/key being copied to the extension. Sorry!

No problem. It's easier to implement anyway.

Adding the “contribute” experience as well would be awesome, of course. But the most important thing is viewing. If we have got this working with the API soon, that’s a great step forward already.

Agreed. The extension having access to the website API should definitely be the next priority. Once it's working, it will be a huge milestone!

[jacob-willden]

Next update: Hulu is now 100% supported (including that it won't try to filter during ads). Also, thanks to an update from VideoSkip, the extension now supports overlapping annotations. This means that there can be an annotation to blank out the video, and then a skip annotation during it.

The update from VideoSkip even includes the ability to not only mute, blank, and skip video, but also to blur and fast-forward through the video (though we certainly don't need to implement those features too if we don't think there would be demand for them).

Thanks again for the support!

[ocram]

Thank you so much!

My understanding was that you combined parts of “Play It My Way” (most parts) and “VideoSkip” (a few small parts). Is this (still) correct?

Regarding the former, the license is fine (LGPL). As for the latter, while the main repository is GPL, it seems the repository for the extension is not marked as such, except for a single file. Could you check this? And do you think it would be a problem? Do you think it would make sense to contact the author and ask for clarification? Or are the parts you got from this so small and have not been copied verbatim anyway?

[jacob-willden]

No problem! The amount of code from both repositories that I used are actually quite similar (about 5 functions from VideoSkip and about 6 functions from Play It My Way in the content script, none of the other scripts use code from the other extensions except in code that's commented out).

Good catch on the VideoSkip extension repository not having a repository-wide license listed! Since he released the main repository and at least the one file in the extension under the GPL, I assume that he intended to release the whole extension under the license, but I'll try to find out for certain.

Edit: I checked all the functions from VideoSkip, and all but two of them are also in the main repository, and one of those two not in the main repository was copied from Stack Overflow. The other one is the goToTime function. In the VideoSkip extension, the original function checks if the streaming service is Netflix and takes the special function if it is (actually the same one that I discovered elsewhere on StackOverflow) and passes it into the function from Stack Overflow above. If the service isn't Netflix, the function just updates the currentTime. I added some checks for other streaming services so the currentTime matches the display time.

I'll still check for clarification about the license to be sure.

[jacob-willden]

Good news! The VideoSkip creator got back to me. He said he did intend for the whole extension repository to be released under the GPL, and he's going to clarify it here on Github. So we'll be good to go!

[jacob-willden]

Update: With Google Chrome deprecating manifest version 2 for extensions next year, I've created a new version of the browser extension that supports manifest version 3. I'm going to keep the old version of the extension in the same repository until manifest version 2 is deprecated on all major browsers.