Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Store vector embeddings for talks, and display related talks #19

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
+181 −101
Open

Store vector embeddings for talks, and display related talks #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4d99696
Store vector embeddings for talks
crohr Jun 20, 2023
5807118
Working!
crohr Jun 20, 2023
5f4d277
Attempt to not save _vectors if title or description hasn't changed
crohr Jun 21, 2023
cde60c2
Merge remote-tracking branch 'upstream/main' into feature/similar-videos
crohr Sep 6, 2023
83197d1
Store embedding in SQLite to avoid recomputation unless title or desc…
crohr Sep 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion .env.sample
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
MEILI_MASTER_KEY=your local meili master key but in therory meilisearch can run without master key in development mode
YOUTUBE_API_KEY=useful for scraping youtube videos but the app can run without it
RUBYVIDEO_GITHUB_TOKEN=useful for the profile enhancement feature but the app can run without it

OPENAI_ACCESS_TOKEN="change_me"
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
*.local

/data.ms
/data_meili
/data_tmp
/data_preparation/**/*
!/data_preparation/organisations.yml
Expand Down
1 change: 1 addition & 0 deletions Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,5 +93,6 @@ gem "meta-tags", "~> 2.18"
gem "groupdate", "~> 6.2"

gem "appsignal", "~> 3.4"
gem "ruby-openai"

gem "chartkick", "~> 5.0"
146 changes: 63 additions & 83 deletions Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
GIT
remote: https://github.com/rails/rails.git
revision: 55412cd9257dc27a8a9175529857ce5f2d81f92f
revision: e10e35dd32aff48457a76b83e244e8baad34a9ec
branch: main
specs:
actioncable (7.1.0.alpha)
Expand Down Expand Up @@ -28,15 +28,15 @@ GIT
net-imap
net-pop
net-smtp
rails-dom-testing (~> 2.2)
rails-dom-testing (~> 2.0)
actionpack (7.1.0.alpha)
actionview (= 7.1.0.alpha)
activesupport (= 7.1.0.alpha)
nokogiri (>= 1.8.5)
rack (>= 2.2.4)
rack-session (>= 1.0.1)
rack-test (>= 0.6.3)
rails-dom-testing (~> 2.2)
rails-dom-testing (~> 2.0)
rails-html-sanitizer (~> 1.6)
actiontext (7.1.0.alpha)
actionpack (= 7.1.0.alpha)
Expand All @@ -49,7 +49,7 @@ GIT
activesupport (= 7.1.0.alpha)
builder (~> 3.1)
erubi (~> 1.11)
rails-dom-testing (~> 2.2)
rails-dom-testing (~> 2.0)
rails-html-sanitizer (~> 1.6)
activejob (7.1.0.alpha)
activesupport (= 7.1.0.alpha)
Expand All @@ -59,22 +59,17 @@ GIT
activerecord (7.1.0.alpha)
activemodel (= 7.1.0.alpha)
activesupport (= 7.1.0.alpha)
timeout (>= 0.4.0)
activestorage (7.1.0.alpha)
actionpack (= 7.1.0.alpha)
activejob (= 7.1.0.alpha)
activerecord (= 7.1.0.alpha)
activesupport (= 7.1.0.alpha)
marcel (~> 1.0)
activesupport (7.1.0.alpha)
base64
bigdecimal
concurrent-ruby (~> 1.0, >= 1.0.2)
connection_pool (>= 2.2.5)
drb
i18n (>= 1.6, < 2)
minitest (>= 5.1)
mutex_m
tzinfo (~> 2.0)
rails (7.1.0.alpha)
actioncable (= 7.1.0.alpha)
Expand Down Expand Up @@ -102,7 +97,7 @@ GIT
GEM
remote: https://rubygems.org/
specs:
addressable (2.8.5)
addressable (2.8.4)
public_suffix (>= 2.0.2, < 6.0)
ahoy_matey (4.2.1)
activesupport (>= 5.2)
Expand All @@ -111,26 +106,24 @@ GEM
annotate (3.2.0)
activerecord (>= 3.2, < 8.0)
rake (>= 10.4, < 14.0)
appsignal (3.4.5)
appsignal (3.4.4)
rack
ast (2.4.2)
authentication-zero (2.16.36)
base64 (0.1.1)
bcrypt (3.1.19)
authentication-zero (2.16.29)
bcrypt (3.1.18)
better_html (2.0.2)
actionview (>= 6.0)
activesupport (>= 6.0)
ast (~> 2.0)
erubi (~> 1.4)
parser (>= 2.4)
smart_properties
bigdecimal (3.1.4)
bindex (0.8.1)
bootsnap (1.16.0)
msgpack (~> 1.2)
builder (3.2.4)
byebug (11.1.3)
capybara (3.39.2)
capybara (3.39.0)
addressable
matrix
mini_mime (>= 0.1.3)
Expand All @@ -139,7 +132,7 @@ GEM
rack-test (>= 0.6.3)
regexp_parser (>= 1.5, < 3.0)
xpath (~> 3.2)
chartkick (5.0.2)
chartkick (5.0.4)
concurrent-ruby (1.2.2)
connection_pool (2.4.1)
crack (0.4.5)
Expand All @@ -150,14 +143,12 @@ GEM
irb (>= 1.5.0)
reline (>= 0.3.1)
device_detector (1.1.0)
dockerfile-rails (1.5.2)
dockerfile-rails (1.2.5)
rails
dotenv (2.8.1)
dotenv-rails (2.8.1)
dotenv (= 2.8.1)
railties (>= 3.2)
drb (2.1.1)
ruby2_keywords
dry-cli (1.0.0)
erb_lint (0.4.0)
activesupport
Expand All @@ -168,18 +159,17 @@ GEM
smart_properties
error_highlight (0.5.1)
erubi (1.12.0)
faraday (2.7.10)
faraday-net_http (>= 2.0, < 3.1)
ruby2_keywords (>= 0.0.4)
faraday-multipart (1.0.4)
multipart-post (~> 2)
faraday-net_http (3.0.2)
globalid (1.1.0)
activesupport (>= 5.0)
groupdate (6.2.1)
activesupport (>= 5.2)
hanami-router (0.6.2)
hanami-utils (~> 0.7)
http_router (~> 0.11)
hanami-utils (0.9.2)
hashdiff (1.0.1)
http_router (0.11.2)
rack (>= 1.0.0)
url_mount (~> 0.2.1)
httparty (0.21.0)
mini_mime (>= 1.0.0)
multi_xml (>= 0.5.2)
Expand All @@ -189,18 +179,14 @@ GEM
activesupport (>= 3.0)
nokogiri (>= 1.6)
io-console (0.6.0)
irb (1.7.4)
reline (>= 0.3.6)
irb (1.7.0)
reline (>= 0.3.0)
json (2.6.3)
language_server-protocol (3.17.0.3)
lint_roller (1.1.0)
litestack (0.3.0)
erubi
hanami-router
lint_roller (1.0.0)
litestack (0.2.3)
oj
rack
sqlite3
tilt
loofah (2.21.3)
crass (~> 1.0.2)
nokogiri (>= 1.12.0)
Expand All @@ -218,13 +204,13 @@ GEM
meta-tags (2.18.0)
actionpack (>= 3.2.0, < 7.1)
mini_mime (1.1.5)
minitest (5.19.0)
msgpack (1.7.1)
minitest (5.18.0)
msgpack (1.7.0)
multi_xml (0.6.0)
mutex_m (0.1.2)
multipart-post (2.3.0)
net-http (0.3.2)
uri
net-imap (0.3.7)
net-imap (0.3.6)
date
net-protocol
net-pop (0.1.2)
Expand All @@ -234,28 +220,25 @@ GEM
net-smtp (0.3.3)
net-protocol
nio4r (2.5.9)
nokogiri (1.15.4-aarch64-linux)
racc (~> 1.4)
nokogiri (1.15.4-arm64-darwin)
nokogiri (1.15.2-aarch64-linux)
racc (~> 1.4)
nokogiri (1.15.4-x86_64-darwin)
nokogiri (1.15.2-arm64-darwin)
racc (~> 1.4)
nokogiri (1.15.4-x86_64-linux)
nokogiri (1.15.2-x86_64-linux)
racc (~> 1.4)
oj (3.15.1)
oj (3.14.3)
pagy (6.0.4)
parallel (1.23.0)
parser (3.2.2.3)
parser (3.2.2.1)
ast (~> 2.4.1)
racc
prettier_print (1.2.1)
propshaft (0.7.0)
actionpack (>= 7.0.0)
activesupport (>= 7.0.0)
rack
railties (>= 7.0.0)
public_suffix (5.0.3)
puma (6.3.0)
public_suffix (5.0.1)
puma (6.2.2)
nio4r (~> 2.0)
racc (1.7.1)
rack (3.0.8)
Expand All @@ -274,71 +257,70 @@ GEM
actionpack (>= 5.0.1.rc1)
actionview (>= 5.0.1.rc1)
activesupport (>= 5.0.1.rc1)
rails-dom-testing (2.2.0)
activesupport (>= 5.0.0)
minitest
rails-dom-testing (2.0.3)
activesupport (>= 4.2.0)
nokogiri (>= 1.6)
rails-html-sanitizer (1.6.0)
loofah (~> 2.21)
nokogiri (~> 1.14)
rainbow (3.1.1)
rake (13.0.6)
regexp_parser (2.8.1)
reline (0.3.8)
regexp_parser (2.8.0)
reline (0.3.5)
io-console (~> 0.5)
rexml (3.2.6)
rubocop (1.52.1)
rexml (3.2.5)
rubocop (1.50.2)
json (~> 2.3)
parallel (~> 1.10)
parser (>= 3.2.2.3)
parser (>= 3.2.0.0)
rainbow (>= 2.2.2, < 4.0)
regexp_parser (>= 1.8, < 3.0)
rexml (>= 3.2.5, < 4.0)
rubocop-ast (>= 1.28.0, < 2.0)
ruby-progressbar (~> 1.7)
unicode-display_width (>= 2.4.0, < 3.0)
rubocop-ast (1.29.0)
rubocop-ast (1.28.1)
parser (>= 3.2.1.0)
rubocop-performance (1.18.0)
rubocop-performance (1.16.0)
rubocop (>= 1.7.0, < 2.0)
rubocop-ast (>= 0.4.0)
ruby-lsp (0.5.1)
language_server-protocol (~> 3.17.0)
sorbet-runtime
syntax_tree (>= 6.1.1, < 7)
ruby-openai (5.1.0)
faraday (>= 1)
faraday-multipart (>= 1)
ruby-progressbar (1.13.0)
ruby2_keywords (0.0.5)
rubyzip (2.3.2)
safely_block (0.4.0)
selenium-webdriver (4.11.0)
selenium-webdriver (4.9.1)
rexml (~> 3.2, >= 3.2.5)
rubyzip (>= 1.2.2, < 3.0)
websocket (~> 1.0)
smart_properties (1.17.0)
sorbet-runtime (0.5.10962)
sqlite3 (1.6.4-aarch64-linux)
sqlite3 (1.6.4-arm64-darwin)
sqlite3 (1.6.4-x86_64-darwin)
sqlite3 (1.6.4-x86_64-linux)
standard (1.30.1)
sorbet-runtime (0.5.10820)
sqlite3 (1.6.2-aarch64-linux)
sqlite3 (1.6.2-arm64-darwin)
sqlite3 (1.6.2-x86_64-linux)
standard (1.28.2)
language_server-protocol (~> 3.17.0.2)
lint_roller (~> 1.0)
rubocop (~> 1.52.0)
rubocop (~> 1.50.2)
standard-custom (~> 1.0.0)
standard-performance (~> 1.1.0)
standard-custom (1.0.2)
standard-performance (~> 1.0.1)
standard-custom (1.0.0)
lint_roller (~> 1.0)
rubocop (~> 1.50)
standard-performance (1.1.2)
lint_roller (~> 1.1)
rubocop-performance (~> 1.18.0)
standard-performance (1.0.1)
lint_roller (~> 1.0)
rubocop-performance (~> 1.16.0)
standardrb (1.0.1)
standard
syntax_tree (6.1.1)
prettier_print (>= 1.2.0)
thor (1.2.2)
tilt (2.2.0)
timeout (0.4.0)
timeout (0.3.2)
turbo-rails (1.4.0)
actionpack (>= 6.0.0)
activejob (>= 6.0.0)
Expand All @@ -347,13 +329,11 @@ GEM
concurrent-ruby (~> 1.0)
unicode-display_width (2.4.2)
uri (0.12.1)
url_mount (0.2.1)
rack
vcr (6.2.0)
vite_rails (3.0.15)
vcr (6.1.0)
vite_rails (3.0.14)
railties (>= 5.1, < 8)
vite_ruby (~> 3.0, >= 3.2.2)
vite_ruby (3.3.4)
vite_ruby (3.3.2)
dry-cli (>= 0.7, < 2)
rack-proxy (~> 0.6, >= 0.6.1)
zeitwerk (~> 2.2)
Expand All @@ -368,17 +348,16 @@ GEM
hashdiff (>= 0.4.0, < 2.0.0)
webrick (1.8.1)
websocket (1.2.9)
websocket-driver (0.7.6)
websocket-driver (0.7.5)
websocket-extensions (>= 0.1.0)
websocket-extensions (0.1.5)
xpath (3.2.0)
nokogiri (~> 1.8)
zeitwerk (2.6.11)
zeitwerk (2.6.8)

PLATFORMS
aarch64-linux
arm64-darwin-22
x86_64-darwin-19
x86_64-linux

DEPENDENCIES
Expand Down Expand Up @@ -409,6 +388,7 @@ DEPENDENCIES
rails!
rails-controller-testing
ruby-lsp (~> 0.5.1)
ruby-openai
selenium-webdriver
sqlite3 (~> 1.4)
standardrb (~> 1.0)
Expand Down
2 changes: 1 addition & 1 deletion Procfile.dev
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
web: bin/rails server -p 3000
vite: bin/vite dev
search: meilisearch
search: docker rm -f rubyvideo-meilisearch && docker run --name rubyvideo-meilisearch -p 7700:7700 -v $(pwd)/meili_data:/data_meili getmeili/meilisearch:v1.3.2 meilisearch --env development --no-analytics --log-level=INFO