L4_predictions_prompts_safety.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d7a08969-6d4f-4896-bd26-26f8912e8adb",
   "metadata": {},
   "source": [
    "# L4: Predictions, Prompts and Safety"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90660a8a-e520-416d-b651-d15194d32c0a",
   "metadata": {},
   "source": [
    "- Load the Project ID and credentials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0ca3df3-cafa-4dbb-8201-a56087356eb9",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "from utils import authenticate\n",
    "credentials, PROJECT_ID = authenticate() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd225ca7-3753-402a-8c3b-2dbfc2a73e92",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "REGION = \"us-central1\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3f49abb-0bf6-45dc-ba84-921612e85bea",
   "metadata": {},
   "source": [
    "- Import the [Vertex AI](https://cloud.google.com/vertex-ai) SDK.\n",
    "- Import and load the model.\n",
    "- Initialize it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e52d8dd3-d24c-40eb-9d7c-623d10645f3d",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import vertexai\n",
    "from vertexai.language_models import TextGenerationModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9f7afcc-84ce-4eac-a754-47df31fc42bc",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "vertexai.init(project = PROJECT_ID,\n",
    "              location = REGION,\n",
    "              credentials = credentials)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11e9d660-9ad4-4a61-8c3a-99984e12b0a8",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deployment\n",
    "\n",
    "### Load Balancing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7adcd0b8-4f12-4723-a5e3-43e4ea0534c4",
   "metadata": {},
   "source": [
    "- Load from pre-trained `text-bison@001`\n",
    "- Retrieve the endpoints (deployed as REST API)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf246933-a0a1-4c1e-9475-8be2e1c82ba4",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "model = TextGenerationModel.from_pretrained(\"text-bison@001\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fce2a567-495e-4f38-a4ce-ab328da96e58",
   "metadata": {
    "tags": []
   },
   "source": [
    "- Get the list of multiple models executed and deployed.\n",
    "- This helps to rout the traffic to different endpoints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b41e10a8-b705-4ba1-9917-8db89164daa3",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "list_tuned_models = model.list_tuned_model_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cea9635e-77e1-4d12-92ae-7bf9f2b8f173",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "for i in list_tuned_models:\n",
    "    print (i)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e9bdfc9-7153-486a-a133-26421dbfc760",
   "metadata": {
    "tags": []
   },
   "source": [
    "- Randomly select from one of the endpoints to divide the prediction load."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d0931af-8f02-4e8b-baff-f14a476f5db4",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04a40f13-5612-4c3b-9e6d-374ba0c8d013",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "tuned_model_select = random.choice(list_tuned_models)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bc18f4d-4c0b-4fd5-b3a8-169a0f826de2",
   "metadata": {},
   "source": [
    "### Getting the Response\n",
    "\n",
    "- Load the endpoint of the randomly selected model to be called with a prompt.\n",
    "- The prompt needs to be as similar as possible as the one you trained your model on (python questions from stack overflow dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3f859f0-0ecb-4551-888c-3a84142c8ed2",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "deployed_model = TextGenerationModel.get_tuned_model\\\n",
    "(tuned_model_select)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0127d501-e300-495d-af10-908034bf7e6f",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "PROMPT = \"How can I load a csv file using Pandas?\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d2a3439-5ff5-4022-86b1-5cbc3eda7f33",
   "metadata": {
    "tags": []
   },
   "source": [
    "- Use `deployed_model.predit` to call the API using the prompt. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1bf6342-bd54-49fc-a683-183a5d8376fc",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "### depending on the latency of your prompt\n",
    "### it can take some time to load\n",
    "response = deployed_model.predict(PROMPT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38a07678-678d-4e2a-af02-89ce6616642a",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d9a61e7-4624-41f0-a97f-07ca1790d63f",
   "metadata": {
    "tags": []
   },
   "source": [
    "- `pprint` makes the response easily readable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9086ddf-f6b5-4004-b1a3-127bc42f7260",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "324230ac-5190-4048-97fd-b2302fd81926",
   "metadata": {
    "tags": []
   },
   "source": [
    "- Sending multiple prompts can return multiple responses `([0], [1], [2]...)`\n",
    "- In this example, only 1 prompt is being sent, and returning only 1 response `([0])`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6823d18-f88c-4433-90ae-448d59698671",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "### load the first object of the response\n",
    "output = response._prediction_response[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbe45174-44f8-44be-b9dd-e50222a80738",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "### print the first object of the response\n",
    "pprint(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40053c1d-691b-4090-8530-4fd63d0ebb9f",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "### load the second object of the response\n",
    "output = response._prediction_response[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56574410-b8bb-463b-b1ba-8d4fd9160e8f",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "### print the second object of the response\n",
    "pprint(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d871fa0-6943-4d48-b46a-ae8f43087c45",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "### retrieve the \"content\" key from the second object\n",
    "final_output = response._prediction_response[0][0][\"content\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbd53e48-f4c2-403a-8a4f-a5764f6ff149",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "### printing \"content\" key from the second object\n",
    "print(final_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffd68e51-9c59-44ae-acf1-0a8c9631c6ff",
   "metadata": {},
   "source": [
    "#### Prompt Management and Templates\n",
    "- Remember that the model was trained on data that had an `Instruction` and a `Question` as a `Prompt` (Lesson 2).\n",
    "- In the example above, *only*  a `Question` as a `Prompt` was used for a response.\n",
    "- It is important for the production data to be the same as the training data. Difference in data can effect the model performance.\n",
    "- Add the same `Instruction` as it was used for training data, and combine it with a `Question` to be used as a `Prompt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f84bc6fd-55a6-4162-a011-c92e9c209815",
   "metadata": {
    "height": 115,
    "tags": []
   },
   "outputs": [],
   "source": [
    "INSTRUCTION = \"\"\"\\\n",
    "Please answer the following Stackoverflow question on Python.\\\n",
    "Answer it like\\\n",
    "you are a developer answering Stackoverflow questions.\\\n",
    "Question:\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b08b9743-194c-49c1-88c4-4ddf56edeee1",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "QUESTION = \"How can I store my TensorFlow checkpoint on\\\n",
    "Google Cloud Storage? Python example?\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e7381e5-c564-496f-8271-eeb184ea3ce9",
   "metadata": {
    "tags": []
   },
   "source": [
    "- Combine the intruction and the question to create the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb37e786-7162-4244-8c5d-58c6ac847a1c",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "PROMPT = f\"\"\"\n",
    "{INSTRUCTION} {QUESTION}\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cea9cabe-539f-4a54-b8ce-00a772f7338c",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(PROMPT)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d443e7fd-ca01-4a67-9a97-72bfff21cfa0",
   "metadata": {
    "tags": []
   },
   "source": [
    "- Get the response using the new prompt, which is consistent with the prompt used for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "701056af-f9d1-4357-aca8-f6753db7f13d",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "final_response = deployed_model.predict(PROMPT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1f354d3-de03-4151-9be0-982e78e32716",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "output = final_response._prediction_response[0][0][\"content\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9512ba2d-d7f7-4a55-81a4-27e4e935d4e7",
   "metadata": {},
   "source": [
    "- Note how the response changed from earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21193021-0dcf-4f52-b7c1-7c752f09c64e",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cb6f518-e6d8-4fc2-a1e0-ea9c936a6a50",
   "metadata": {},
   "source": [
    "### Safety Attributes\n",
    "- The reponse also includes safety scores.\n",
    "- These scores can be used to make sure that the LLM's response is within the boundries of the expected behaviour.\n",
    "- The first layer for this check, `blocked`, is by the model itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10006430-0a06-4e62-88f3-3e67af39f54c",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [],
   "source": [
    "### retrieve the \"blocked\" key from the \n",
    "### \"safetyAttributes\" of the response\n",
    "blocked = response._prediction_response[0][0]\\\n",
    "['safetyAttributes']['blocked']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cd5e925-db25-45f0-9d44-e1adffa729f1",
   "metadata": {},
   "source": [
    "- Check to see if any response was blocked.\n",
    "- It returns `True` if there is, and `False` if there's none."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a6c563d-fe82-4c03-b0ed-bf0805cd4534",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(blocked)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e79306c-c207-4750-8cfc-349a7d536bbe",
   "metadata": {},
   "source": [
    "- The second layer of this check can be defined by you, as a practitioner, according to the thresholds you set.\n",
    "- The response returns probabilities for each safety score category which can be used to design the thresholds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf399ec9-5fe6-48fc-8348-7ff715e0dbaa",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "### retrieve the \"safetyAttributes\" of the response\n",
    "safety_attributes = response._prediction_response[0][0]\\\n",
    "['safetyAttributes']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97768701-f28e-4bf9-bfbd-8adfa798aba0",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "pprint(safety_attributes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "382a3db8-7d05-49c7-a5ba-841c9a04517f",
   "metadata": {},
   "source": [
    "### Citations\n",
    "- Ideally, a LLM should generate as much original cotent as possible.\n",
    "- The `citationMetadata` can be used to check and reduce the chances of a LLM generating a lot of existing content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0cf2018-df0f-4e9b-abcf-867a88652ae6",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [],
   "source": [
    "### retrieve the \"citations\" key from the \n",
    "### \"citationMetadata\" of the response\n",
    "citation = response._prediction_response[0][0]\\\n",
    "['citationMetadata']['citations']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72b6480a-3568-42de-8862-10c58116e1f4",
   "metadata": {},
   "source": [
    "- Returns a list with information if the response is cited, if not, then it retuns an empty list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afd1f20f-838a-4fa4-b344-66183771b223",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "pprint(citation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "531ce0aa",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Try it Yourself! - Return a Citation\n",
    "\n",
    "Now it is time for you to come up with an example, for which the model response should return citation infomration. The idea here is to see how that would look like, so keeping it basic, the prompt can be different than the coding examples used above. Below code is one such prompt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "173e6554-3a2b-44ae-b713-7bdf7feac684",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "PROMPT = \"Finish the sentence: To be, or not \""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12199f52",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "response = deployed_model.predict(PROMPT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44c89d2e-b2a0-460c-be41-92d6a6c640bf",
   "metadata": {
    "height": 64
   },
   "outputs": [],
   "source": [
    "### output of the model\n",
    "output = response._prediction_response[0][0][\"content\"]\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e24e165d",
   "metadata": {
    "height": 98
   },
   "outputs": [],
   "source": [
    "### check for citation\n",
    "citation = response._prediction_response[0][0]\\\n",
    "['citationMetadata']['citations']\n",
    "\n",
    "pprint(citation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "387d6242",
   "metadata": {},
   "source": [
    "**Your turn!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7952facb",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "946ce368",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e22c2871",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "276c14f0",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "63a095a5",
   "metadata": {},
   "source": [
    "### Optional Notebook\n",
    "\n",
    "[Tuning and deploy a foundation model](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/tuning/tuning_text_bison.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e3d3ddd",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "tf2-cpu.2-11.m113",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-cpu.2-11:m113"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}