Skip to content

yt: Detect Mix playlists robustly and avoid '-1 videos' overlay (Fixes #5454) #5477

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

yt: Detect Mix playlists robustly and avoid '-1 videos' overlay (Fixes #5454) #5477

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions locales/en-US.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,8 +122,6 @@
"Redirect homepage to feed: ": "Redirect homepage to feed: ",
"preferences_max_results_label": "Number of videos shown in feed: ",
"preferences_sort_label": "Sort videos by: ",
"preferences_default_playlist": "Default playlist: ",
"preferences_default_playlist_none": "No default playlist set",
Comment on lines -125 to -126
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't make unrelated changes.

"published": "published",
"published - reverse": "published - reverse",
"alphabetically": "alphabetically",
Expand Down Expand Up @@ -504,5 +502,6 @@
"carousel_go_to": "Go to slide `x`",
"timeline_parse_error_placeholder_heading": "Unable to parse item",
"timeline_parse_error_placeholder_message": "Invidious encountered an error while trying to parse this item. For more information see below:",
"timeline_parse_error_show_technical_details": "Show technical details"
"timeline_parse_error_show_technical_details": "Show technical details",
"label_mix": "Mix"
}
2 changes: 2 additions & 0 deletions src/invidious/helpers/serialized_yt_data.cr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,8 @@ struct SearchPlaylist
property videos : Array(SearchPlaylistVideo)
property thumbnail : String?
property author_verified : Bool
@[DB::Field(ignore: true)]
property is_mix : Bool = false

def to_json(locale : String?, json : JSON::Builder)
json.object do
Expand Down
50 changes: 31 additions & 19 deletions src/invidious/yt_backend/extractors.cr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,6 +321,7 @@ private module Parsers
video_count = HelperExtractors.get_video_count(item_contents)
playlist_thumbnail = HelperExtractors.get_thumbnails(item_contents)

is_mix = plid.starts_with? "RD"
SearchPlaylist.new({
title: title,
id: plid,
Expand All @@ -330,6 +331,7 @@ private module Parsers
videos: [] of SearchPlaylistVideo,
thumbnail: playlist_thumbnail,
author_verified: author_verified,
is_mix: is_mix,
})
end

Expand Down Expand Up @@ -382,6 +384,7 @@ private module Parsers

# TODO: item_contents["publishedTimeText"]?

is_mix = plid.starts_with? "RD"
SearchPlaylist.new({
title: title,
id: plid,
Expand All @@ -391,6 +394,7 @@ private module Parsers
videos: videos,
thumbnail: playlist_thumbnail,
author_verified: author_verified,
is_mix: is_mix,
})
end

Expand Down Expand Up @@ -656,27 +660,34 @@ private module Parsers

thumbnail = thumbnail_view_model.dig("image", "sources", 0, "url").as_s

# This complicated sequences tries to extract the following data structure:
# "overlays": [{
# "thumbnailOverlayBadgeViewModel": {
# "thumbnailBadges": [{
# "thumbnailBadgeViewModel": {
# "text": "430 episodes",
# "badgeStyle": "THUMBNAIL_OVERLAY_BADGE_STYLE_DEFAULT"
# }
# }]
# }
# }]
#
# NOTE: this simplistic `.to_i` conversion might not work on larger
# playlists and hasn't been tested.
Comment on lines -659 to -672
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please consider adapting the comment instead of entirely removing it

video_count = thumbnail_view_model.dig("overlays").as_a
# Parse overlays badges
overlays = thumbnail_view_model.dig("overlays").as_a
.compact_map(&.dig?("thumbnailOverlayBadgeViewModel", "thumbnailBadges").try &.as_a)
.flatten
.find(nil, &.dig?("thumbnailBadgeViewModel", "text").try { |node|
{"episodes", "videos"}.any? { |str| node.as_s.ends_with?(str) }
})
.try &.dig("thumbnailBadgeViewModel", "text").as_s.to_i(strict: false)

# Detect Mix playlists via icon imageName at any index or text case-insensitively
is_mix = overlays.any? { |badge|
sources = badge.dig?("thumbnailBadgeViewModel", "icon", "sources").try &.as_a || [] of JSON::Any
has_mix_icon = sources.any? { |s| s.dig?("clientResource", "imageName").try &.as_s == "MIX" }
text = badge.dig?("thumbnailBadgeViewModel", "text").try &.as_s || ""
has_mix_text = text.downcase == "mix"
has_mix_icon || has_mix_text
}

# Fallback: RD-prefixed playlist IDs are Mixes
is_mix ||= playlist_id.starts_with? "RD"
Comment on lines +668 to +678
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure to fully understand why you need all that extra logic to determine if the playlist is a mix, when that last check seems robust enough? (as far as I know, mixes have always began with RD).


# Robustly extract digits from any badge text for video_count; fallback to -1 if not found
video_count = nil.as(Int32?)
overlays.each do |badge|
t = badge.dig?("thumbnailBadgeViewModel", "text").try &.as_s
next unless t
digits = t.gsub(/\D/, "")
unless digits.empty?
video_count = digits.to_i
break
end
end

metadata = item_contents.dig("metadata", "lockupMetadataViewModel")
title = metadata.dig("title", "content").as_s
Expand All @@ -699,6 +710,7 @@ private module Parsers
videos: [] of SearchPlaylistVideo,
thumbnail: thumbnail,
author_verified: false,
is_mix: is_mix,
})
end

Expand Down
Loading