PromptClip_spoken.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview\n",
    "This notebook walks your through the process of creating clips with LLM prompts using spoken content of the video. \n",
    "\n",
    "Pick a video, decide your prompt, generate a new clip ⚡️\n",
    "\n",
    "It's as simple as it sounds.\n",
    "\n",
    "If you want to go extra mile you can score and rank your results, add Image Overlays or Audio overlays on these clips."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# But first, let's install the dependecies.\n",
    "!pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choose the Video\n",
    "\n",
    "You can either use a fresh video from Youtube etc. or choose an exisitng one already uploaded on your VideoDB collection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import videodb\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "\n",
    "# TODO: setup .env file\n",
    "load_dotenv()\n",
    "\n",
    "# Connect to VideoDB\n",
    "conn = videodb.connect()\n",
    "coll = conn.get_collection()\n",
    "\n",
    "# NOTE: Please set video_id or SPOKEN_DEMO_VIDEO_ID in .env if video already exists in the collection\n",
    "video_id = os.getenv(\"SPOKEN_DEMO_VIDEO_ID\")\n",
    "video_url = \"https://www.youtube.com/watch?v=HpUR7-Oe1ss\"\n",
    "\n",
    "if not video_id:\n",
    "    video = coll.upload(url=video_url)\n",
    "else:\n",
    "    video = coll.get_video(video_id)\n",
    "\n",
    "print(f\"video_id: {video.id}, name: {video.name}\")\n",
    "video.play()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing Spoken Information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    transcript_text = video.get_transcript_text()\n",
    "except Exception:\n",
    "    video.index_spoken_words()\n",
    "    transcript_text = video.get_transcript_text()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run your Prompt\n",
    "\n",
    "To create a clip using the `text_prompter` function from a video, it's crucial to craft a specific prompt that will help identify the most relevant segments for your use case. This prompt should highlight the themes, topics, or specific phrases you're interested in. The function then analyzes the video's spoken content to find segments that match your criteria. \n",
    "\n",
    "Before you can use `text_prompter`, make sure the video's spoken content is indexed with the `video.index_spoken_words()` function. This prepares the video for analysis by making its spoken content searchable.\n",
    "\n",
    "The `text_prompter` will return sentences or segments from the video that match your prompt. Review these to ensure they align with your needs. You can then use these segments to create your clip, focusing on the content that's most relevant to your use case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from video_prompter import text_prompter\n",
    "\n",
    "# Choose a prompt to create create clip. \n",
    "user_prompt = \"find sentences where a deal is discussed\"\n",
    "result = text_prompter(transcript_text, user_prompt)\n",
    "print(f\"Found {len(result)} segments in the video.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate the Clip\n",
    "\n",
    "To generate a clip, we'll  use **VideoDB**'s `keyword search` feature. We already leveraged the power of the LLM (Large Language Model) to identify relevant sentences. We'll use the search results to create a `programmable video stream`. Here's how you can approach this process:\n",
    "\n",
    "We have the keywords in the `results` variable. Input these keywords into VideoDB's keyword search feature. This function will search through the indexed spoken content of your videos to find matches. \n",
    "\n",
    "The search will return a SearchResult object, which contains detailed information about the found segments, including their timestamps, the text of the spoken content, and possibly other metadata.\n",
    "\n",
    "**Create a Programmable Video Stream with Timeline**: With the specific segments identified, you can now use the Timeline to create a new programmable video stream. The Timeline tool allows you to stitch together video segments based on the timestamps provided in the SearchResult object. You can arrange, cut, or combine these segments to craft a fresh video stream that focuses on your topic of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from videodb import play_stream\n",
    "from videodb.timeline import Timeline\n",
    "from video_prompter import get_result_timestamps, build_video_timeline\n",
    "\n",
    "timeline = Timeline(conn)\n",
    "result_timestamps = get_result_timestamps(video, result, index_type=\"spoken_word\")\n",
    "timeline, duration  = build_video_timeline(video, result_timestamps, timeline)\n",
    "stream = timeline.generate_stream()\n",
    "print(stream)\n",
    "play_stream(stream)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modify the Timeline\n",
    "\n",
    "The programmable stream part of VideoDB allows you to not just watch the original clip but also modify and personalize the stream. Here we can add up the logo on each clip easily. You can read more about it here - https://docs.videodb.io/version-0-0-3-timeline-and-assets-44"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# upload Image\n",
    "from videodb import MediaType\n",
    "\n",
    "image_id = os.getenv(\"SPOKEN_DEMO_IMAGE_ID\")\n",
    "if not image_id:\n",
    "    image = conn.upload(url=\"https://www.freepnglogos.com/uploads/logo-ig-png/logo-ig-instagram-new-logo-vector-download-13.png\", media_type=MediaType.image)\n",
    "    image_id = image.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from videodb.asset import VideoAsset, AudioAsset, ImageAsset\n",
    "\n",
    "image_asset = ImageAsset(\n",
    "    asset_id=image_id,\n",
    "    width=40,\n",
    "    height=40,\n",
    "    x=20,\n",
    "    y=10,\n",
    "    duration=7\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "timeline.add_overlay(0, image_asset)\n",
    "stream = timeline.generate_stream()\n",
    "play_stream(stream)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonus : Ranking using LLM\n",
    "If you want to choose only a few top results and wodering how to do it, have LLM to rank your results and create a score that you can use to decide the order of segments. You can modify the ranking prompt creativiely to drive the outcome of it. We would love to see what you create 🙌🏼"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llm_agent import LLM\n",
    "import re\n",
    "import json\n",
    "from math import floor\n",
    "\n",
    "def ranking_prompt_llm(text, prompt):\n",
    "    ranking_prompt = \"\"\"Given the text provided below and a specific User Prompt, evaluate the relevance of the text\n",
    "    in relation to the user's prompt. Please assign a relevance score ranging from 0 to 10, where 0 indicates no relevance \n",
    "    and 10 signifies perfect alignment with the user's request.\n",
    "    The score quality also increases when the text is a complete senetence, making it perfect for a video clip result\"\"\"\n",
    "\n",
    "    # pass the data\n",
    "    ranking_prompt += f\"\"\"\n",
    "    text: {text}\n",
    "    User Prompt: {prompt}\n",
    "    \"\"\"\n",
    "\n",
    "    # Add instructions to always return JSON at the end of processing.\n",
    "    ranking_prompt += \"\"\"\n",
    "    Ensure the final output strictly adheres to the JSON format specified, without including additional text or explanations. \n",
    "    Use the following structure for your response:\n",
    "    {\n",
    "      \"score\": <relevance score>\n",
    "    }\n",
    "    \"\"\"\n",
    "    try:\n",
    "        response = LLM().chat(message=ranking_prompt)\n",
    "        print(response)\n",
    "        output = response[\"choices\"][0][\"message\"][\"content\"]\n",
    "        res = json.loads(output)\n",
    "        score = res.get('score')\n",
    "        return score\n",
    "    except Exception as e:\n",
    "        return 0 \n",
    "\n",
    "def rank_results(res, prompt, score_percentage=0.40):\n",
    "    \"\"\"\n",
    "    rank and give score to each result\n",
    "    \"\"\"\n",
    "    res_score = []\n",
    "    for text in res:\n",
    "        res_score.append((text, ranking_prompt_llm(text,prompt)))\n",
    "    \n",
    "    res_score_sorted = sorted(res_score, key=lambda x: x[1], reverse=True)\n",
    "    return res_score_sorted[0: floor(len(res_score_sorted)*score_percentage)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ranked_results = rank_results(result, user_prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ranked_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Search for matching video segments and watch the resulting stream\n",
    "from videodb import SearchType\n",
    "from videodb.timeline import Timeline, VideoAsset, AudioAsset\n",
    "\n",
    "timeline = Timeline(conn)\n",
    "for sentences, score in ranked_results:\n",
    "    search_res = video.search(sentences, search_type=SearchType.keyword)\n",
    "    matched_segments = search_res.get_shots()\n",
    "    \n",
    "    # No exact match found\n",
    "    if len(matched_segments) == 0:\n",
    "        continue\n",
    "\n",
    "    # Get the first matched video segment\n",
    "    video_shot = matched_segments[0]\n",
    "\n",
    "    # Create a new Video Asset and add it to a timeline.\n",
    "    timeline.add_inline(VideoAsset(asset_id=video.id, start=video_shot.start, end=video_shot.end))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stream = timeline.generate_stream()\n",
    "play_stream(stream)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add some sound effects to it 🎶\n",
    "\n",
    "Not just this we can jazz it up with audio overlays and create another stream with audio overlays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add music overlay, this can be laughter soundtrack\n",
    "audio_id = os.getenv(\"SPOKEN_DEMO_AUDIO_ID\")\n",
    "if not audio_id:\n",
    "    audio = conn.upload(url=\"https://www.youtube.com/watch?v=q3VVxbJa61Q\", media_type=MediaType.audio)\n",
    "    audio_id = audio.id\n",
    "    print(f\"Uploaded audio with id {audio_id}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1 sec background audio \n",
    "background = AudioAsset(asset_id=audio_id, start=3, end=4, disable_other_tracks=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "timeline = Timeline(conn)\n",
    "dur_so_far = 0\n",
    "for clip_sentences, score in ranked_results:\n",
    "    try:\n",
    "        search_res = video.search(clip_sentences, search_type=SearchType.keyword)\n",
    "        matched_segments = search_res.get_shots()\n",
    "        \n",
    "        # No exact match found\n",
    "        if len(matched_segments) == 0:\n",
    "            continue\n",
    "    \n",
    "        #video segment\n",
    "        video_shot = matched_segments[0]\n",
    "    \n",
    "        # Create a new Video Asset and add it to a timeline.\n",
    "        timeline.add_inline(VideoAsset(asset_id=video.id, start=video_shot.start, end=video_shot.end))\n",
    "        chunk_dur = (video_shot.end - video_shot.start)\n",
    "        dur_so_far += chunk_dur \n",
    "        if chunk_dur < 2:\n",
    "            print(\"Skipping since chunk duration is less then the overlay audio.\")\n",
    "            continue\n",
    "        timeline.add_overlay(dur_so_far-2, background)\n",
    "    except Exception as e:\n",
    "        print(f\"Error: skipping the segment {str(e)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# add music overlay in the last 2 sec of each supercut.\n",
    "stream = timeline.generate_stream()\n",
    "play_stream(stream)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have any questions or feedback. Feel free to reach out to us 🙌🏼\n",
    "\n",
    "* [Discord](https://colab.research.google.com/corgiredirector?site=https%3A%2F%2Fdiscord.gg%2Fpy9P639jGz)\n",
    "* [GitHub](https://github.com/video-db)\n",
    "* [Website](https://colab.research.google.com/corgiredirector?site=https%3A%2F%2Fvideodb.io)\n",
    "* [Email](ashu@videodb.io)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Playground For You To Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "video_url = \"\"\n",
    "video_id = \"\"\n",
    "\n",
    "if not video_id:\n",
    "    video = coll.upload(url=video_url)\n",
    "else:\n",
    "    video = coll.get_video(video_id)\n",
    "\n",
    "print(f\"video_id: {video.id}, name: {video.name}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    transcript = video.get_transcript()\n",
    "    transcript_text = video.get_transcript_text()\n",
    "except Exception:\n",
    "    video.index_spoken_words()\n",
    "    transcript = video.get_transcript()\n",
    "    transcript_text = video.get_transcript_text()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from video_prompter import text_prompter\n",
    "\n",
    "user_prompt = \"\"\n",
    "result = text_prompter(transcript_text, user_prompt)\n",
    "print(f\"Found {len(result)} segments in the video.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from videodb import play_stream\n",
    "from videodb.timeline import Timeline\n",
    "from video_prompter import get_result_timestamps, build_video_timeline\n",
    "\n",
    "timeline = Timeline(conn)\n",
    "result_timestamps = get_result_timestamps(video, result, index_type=\"spoken_word\")\n",
    "timeline, duration  = build_video_timeline(video, result_timestamps, timeline)\n",
    "stream = timeline.generate_stream()\n",
    "print(stream)\n",
    "play_stream(stream)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}