diff --git a/parea/cookbook/img/dashboard.png b/parea/cookbook/img/dashboard.png index ff615de2..7fc39d9d 100644 Binary files a/parea/cookbook/img/dashboard.png and b/parea/cookbook/img/dashboard.png differ diff --git a/parea/cookbook/img/dashboard_detailed_view.png b/parea/cookbook/img/dashboard_detailed_view.png index a24e5899..129491a5 100644 Binary files a/parea/cookbook/img/dashboard_detailed_view.png and b/parea/cookbook/img/dashboard_detailed_view.png differ diff --git a/parea/cookbook/img/deployed_prompts.png b/parea/cookbook/img/deployed_prompts.png index 5f77e628..ed006c5a 100644 Binary files a/parea/cookbook/img/deployed_prompts.png and b/parea/cookbook/img/deployed_prompts.png differ diff --git a/parea/cookbook/img/feedback.png b/parea/cookbook/img/feedback.png index 51f9714a..3325e98f 100644 Binary files a/parea/cookbook/img/feedback.png and b/parea/cookbook/img/feedback.png differ diff --git a/parea/cookbook/img/logs.png b/parea/cookbook/img/logs.png new file mode 100644 index 00000000..22620a99 Binary files /dev/null and b/parea/cookbook/img/logs.png differ diff --git a/parea/cookbook/img/meta_data.png b/parea/cookbook/img/meta_data.png index c2435955..147a6645 100644 Binary files a/parea/cookbook/img/meta_data.png and b/parea/cookbook/img/meta_data.png differ diff --git a/parea/cookbook/tracing_with_Parea_sdk.ipynb b/parea/cookbook/tracing_with_Parea_sdk.ipynb index 4b58a9e2..25ff4436 100644 --- a/parea/cookbook/tracing_with_Parea_sdk.ipynb +++ b/parea/cookbook/tracing_with_Parea_sdk.ipynb @@ -13,13 +13,13 @@ "With the Parea SDK, you can gain visibility into **any LLM application**. Together with the web application, Parea speeds up your debugging, evaluating, and monitoring workflows.\n", "Parea is also framework and provider-agnostic. Parea traces your prompts and chains, whether deployed from Parea or within your codebase.\n", "\n", - "We will create a simple chat app and instrument logging with Parea. We will also add tags and other metadata to enrich our traces. The chat app uses three 'chained' components to generate a text argument on a provided subject:\n", + "For this example, ee will create a simple chat app and instrument trace logs and feedback with Parea. We will also add tags and other metadata to enrich our traces. The chat app uses three 'chained' components to generate a text argument on a provided subject:\n", "\n", "1. An argument generation function\n", "2. Critique function\n", "3. Refine function\n", "\n", - "Each function will call an LLM provider; in our case, we'll use OpenAI, but you could quickly call Anthropic or Azure. Parea's dashboard shows how the LLM calls are organized for further analysis and investigation.\n", + "Each function will call an LLM provider; in our case, we'll use OpenAI, but you could easily call any other provider. Parea's log dashboard provides a detailed trace of your LLM calls, so you can step into the details of each step for further analysis and investigation.\n", "\n", "![DashboardDetailedView](img/dashboard_detailed_view.png)\n", "\n", @@ -36,12 +36,12 @@ "execution_count": null, "id": "5615479e", "metadata": { - "id": "5615479e", - "is_executing": true + "id": "5615479e" }, "outputs": [], "source": [ - "# %pip install -U parea-ai > /dev/null" + "%pip install -U parea-ai > /dev/null\n", + "%pip install attrs > /dev/null" ] }, { @@ -56,7 +56,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 3, "id": "f665dbcd", "metadata": { "id": "f665dbcd", @@ -64,7 +64,10 @@ "base_uri": "https://localhost:8080/" }, "outputId": "3713f0ab-b90a-4751-ca0a-6d28ceebb385", - "is_executing": true + "ExecuteTime": { + "end_time": "2023-10-18T19:21:10.834353Z", + "start_time": "2023-10-18T19:21:10.828677Z" + } }, "outputs": [], "source": [ @@ -78,13 +81,13 @@ "source": [ "## Using the SDK\n", "\n", - "Next, define your chat application. Using the trace decorator will automatically generate a trace_id for each of your LLM calls. The decorated functions inputs, name, and other information are recorded and visible on Parea's dashboard. This is all done on a background thread to avoid blocking your app's execution.\n", + "Next, define your chat application. Using the trace decorator will automatically generate a trace_id for each of your LLM calls. Nested functions with a trace decorator will automatically get associated with a parent trace. The decorated functions inputs/outputs, name, and other information are recorded and visible on Parea's log dashboard. Note: Logging is executed on a background thread to avoid blocking your app's execution.\n", "\n", "We've created three prompts on Parea and have deployed them. Learn how to deploy a prompt [here](https://docs.parea.ai/deployments/deployments).\n", "\n", "![Deployed_Prompts](img/deployed_prompts.png)\n", "\n", - "Now we only need the deployment id for each prompt to get started. You can also do this without a deployed prompt; we'll revisit this example in another walkthrough." + "Now we only need the deployment id for each prompt to get started. You can also do this without a deployed prompt for the same experience (example [here](https://github.com/parea-ai/parea-sdk-py/blob/fc506a8fa9b5a118b15918cc00cdc5e323dcf9bb/parea/cookbook/tracing_without_deployed_prompt.py))." ], "metadata": { "collapsed": false @@ -93,20 +96,22 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 12, "id": "4ba8c359", "metadata": { "tags": [], "id": "4ba8c359", - "is_executing": true + "ExecuteTime": { + "end_time": "2023-10-18T19:42:58.458133Z", + "start_time": "2023-10-18T19:42:58.353216Z" + } }, "outputs": [], "source": [ "from parea.utils.trace_utils import trace\n", "from datetime import datetime\n", - "\n", - "from parea import Parea\n", "from parea.schemas.models import Completion\n", + "from parea import Parea\n", "\n", "p = Parea(api_key=os.getenv(\"PAREA_API_KEY\"))\n", "\n", @@ -114,7 +119,7 @@ "def argument_generator(query: str, additional_description: str = \"\") -> str:\n", " return p.completion(\n", " Completion(\n", - " deployment_id=\"p-XOh3kp8B0nIE82WgioPnr\",\n", + " deployment_id=\"p-tbFUZ5rRaXshj8o5Opfyr\",\n", " llm_inputs={\n", " \"additional_description\": additional_description,\n", " \"date\": f\"{datetime.now()}\",\n", @@ -127,27 +132,81 @@ "def critic(argument: str) -> str:\n", " return p.completion(\n", " Completion(\n", - " deployment_id=\"p-PSOwRyIPaQRq4xQW3MbpV\",\n", + " deployment_id=\"p-iAuVLFHy6VypfGZxwAWW0\",\n", " llm_inputs={\"argument\": argument},\n", " )\n", " ).content\n", "\n", "\n", - "def refiner(query: str, additional_description: str, argument: str, criticism: str) -> str:\n", + "def refiner(query: str, additional_description: str, current_arg: str, criticism: str) -> str:\n", " return p.completion(\n", " Completion(\n", - " deployment_id=\"p-bJ3-UKh9-ixapZafaRBsj\",\n", + " deployment_id=\"p-rEjM4X10rJomOD8Rj9gzJ\",\n", " llm_inputs={\n", " \"additional_description\": additional_description,\n", " \"date\": f\"{datetime.now()}\",\n", " \"query\": query,\n", - " \"argument\": argument,\n", + " \"argument\": current_arg,\n", " \"criticism\": criticism,\n", " },\n", " )\n", " ).content\n", "\n", "\n", + "# Non deployed version\n", + "# from parea.schemas.models import LLMInputs, Message, ModelParams, Role\n", + "\n", + "\n", + "# def argument_generator(query: str, additional_description: str = \"\") -> str:\n", + "# return p.completion(\n", + "# Completion(llm_configuration=LLMInputs(model=\"gpt-3.5-turbo\", model_params=ModelParams(temp=0),\n", + "# messages=[\n", + "# Message(role=Role.system,\n", + "# content=f\"You are a debater making an argument on a topic.\" f\"{additional_description}\" f\" The current time is {datetime.now()}\"),\n", + "# Message(role=Role.user, content=f\"The discussion topic is {query}\"),\n", + "# ],\n", + "# )\n", + "# )\n", + "# ).content\n", + "\n", + "\n", + "# def critic(argument: str) -> str:\n", + "# return p.completion(\n", + "# Completion(llm_configuration=LLMInputs(model=\"gpt-3.5-turbo\", model_params=ModelParams(temp=0),\n", + "# messages=[\n", + "# Message(\n", + "# role=Role.system,\n", + "# content=f\"You are a critic.\"\n", + "# \"\\nWhat unresolved questions or criticism do you have after reading the following argument?\"\n", + "# \"Provide a concise summary of your feedback.\",\n", + "# ),\n", + "# Message(role=Role.system, content=argument),\n", + "# ],\n", + "# )\n", + "# )\n", + "# ).content\n", + "\n", + "\n", + "# def refiner(query: str, additional_description: str, current_arg: str, criticism: str) -> str:\n", + "# return p.completion(\n", + "# Completion(llm_configuration=LLMInputs(model=\"gpt-3.5-turbo\", model_params=ModelParams(temp=0),\n", + "# messages=[\n", + "# Message(\n", + "# role=Role.system,\n", + "# content=f\"You are a debater making an argument on a topic. {additional_description}.\"\n", + "# f\"The current time is {datetime.now()}\",\n", + "# ),\n", + "# Message(role=Role.user, content=f\"The discussion topic is {query}\"),\n", + "# Message(role=Role.assistant, content=current_arg),\n", + "# Message(role=Role.user, content=criticism),\n", + "# Message(role=Role.system, content=\"Please generate a new argument that incorporates the feedback \"\n", + "# \"from the user.\"),\n", + "# ],\n", + "# )\n", + "# )\n", + "# ).content\n", + "\n", + "\n", "@trace\n", "def argument_chain(query: str, additional_description: str = \"\") -> str:\n", " argument = argument_generator(query, additional_description)\n", @@ -162,12 +221,12 @@ "id": "97fb9f73" }, "source": [ - "Now call the chain. If you set up your API key correctly at the start of this notebook, all the results should be traced to [Parea](https://www.optimusprompt.ai/dashboard). We will prompt the app to generate an argument that coffee is good for you." + "Now call the chain. If you set up your API key correctly at the start of this notebook, all the results should be traced to [Parea](https://www.optimusprompt.ai/logs). We will prompt the app to generate an argument that coffee is good for you." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 13, "id": "6f5798a7", "metadata": { "tags": [], @@ -175,17 +234,39 @@ "outputId": "f4c7b093-561f-493d-c7e2-8cde6352f28c", "colab": { "base_uri": "https://localhost:8080/" + }, + "ExecuteTime": { + "end_time": "2023-10-18T19:44:35.400997Z", + "start_time": "2023-10-18T19:43:09.514314Z" } }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Coffee is good for you because it contains several bioactive compounds, such as caffeine and chlorogenic acids, which have been extensively studied and shown to have various health benefits. Caffeine, in moderate amounts, can enhance alertness, improve cognitive function, and boost physical performance. Chlorogenic acids have antioxidant and anti-inflammatory properties, which can help protect against chronic diseases like heart disease and certain types of cancer. Numerous studies have also linked coffee consumption to a reduced risk of developing conditions such as type 2 diabetes, Parkinson's disease, and liver disease. However, it's important to note that individual responses to coffee can vary, and excessive consumption or added sugars and unhealthy additives can negate the potential benefits. As with any dietary choice, moderation and mindful consumption are key.\n" + ] + } + ], "source": [ - "result = argument_chain(\n", + "result1 = argument_chain(\n", " \"Whether coffee is good for you.\",\n", " additional_description=\"Provide a concise, few sentence argument on why coffee is good for you.\",\n", ")\n", - "print(result)" + "print(result1)" ] }, + { + "cell_type": "markdown", + "source": [ + "![Logs](./img/logs.png)" + ], + "metadata": { + "collapsed": false + }, + "id": "e3375e0356f76af7" + }, { "cell_type": "markdown", "id": "3cad8af8", @@ -195,7 +276,7 @@ "source": [ "## Recording feedback\n", "\n", - "The above is all you need to save your app's traces to Parea! You can try changing the functions or raising errors in the above code to see how it's visualized in [Parea](https://www.optimusprompt.ai/dashboard).\n", + "The above is all you need to save your app's traces to Parea! You can try changing the functions or raising errors in the above code to see how it's visualized in [Parea](https://www.optimusprompt.ai/logs).\n", "\n", "You can use the trace_id for other things like monitoring user feedback. You can use the get_current_trace_id() helper function to get the trace_id from within the function context.\n", "\n", @@ -204,12 +285,15 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 6, "id": "117dca62", "metadata": { "tags": [], "id": "117dca62", - "is_executing": true + "ExecuteTime": { + "end_time": "2023-10-18T19:21:50.019278Z", + "start_time": "2023-10-18T19:21:50.013231Z" + } }, "outputs": [], "source": [ @@ -218,7 +302,7 @@ "\n", "@trace\n", "def argument_chain2(query: str, additional_description: str = \"\") -> tuple[str, str]:\n", - " trace_id = get_current_trace_id()\n", + " trace_id = get_current_trace_id() # get parent's trace_id\n", " argument = argument_generator(query, additional_description)\n", " criticism = critic(argument)\n", " return refiner(query, additional_description, argument, criticism), trace_id" @@ -226,19 +310,31 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 7, "id": "2b339dd8", "metadata": { "tags": [], - "id": "2b339dd8" + "id": "2b339dd8", + "ExecuteTime": { + "end_time": "2023-10-18T19:22:14.190299Z", + "start_time": "2023-10-18T19:21:52.559703Z" + } }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "749c63be-74b4-4134-9a0e-a499c9d58f59\n" + ] + } + ], "source": [ "result, trace_id = argument_chain2(\n", " \"Whether coffee is good for you.\",\n", " additional_description=\"Provide a concise, few sentence argument on why coffee is good for you.\",\n", ")\n", - "print(result)" + "print(trace_id)" ] }, { @@ -253,11 +349,15 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 8, "id": "cf7b378c", "metadata": { "tags": [], - "id": "cf7b378c" + "id": "cf7b378c", + "ExecuteTime": { + "end_time": "2023-10-18T19:23:09.118002Z", + "start_time": "2023-10-18T19:23:05.536943Z" + } }, "outputs": [], "source": [ @@ -276,31 +376,87 @@ "![Feedback](./img/feedback.png)" ] }, + { + "cell_type": "markdown", + "source": [ + "The completion response from the SDK also has other useful information.You can get useful statistics such as tokens, latency, whether the call was cached and more.\n", + "\n", + "Let's return the CompletionResponse object and examine the response." + ], + "metadata": { + "collapsed": false + }, + "id": "cbf81b4a8bc1f69b" + }, { "cell_type": "code", - "execution_count": null, - "outputs": [], + "execution_count": 9, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"inference_id\": \"4a518e40-410c-4fc6-9521-d7df9295a50f\",\n", + " \"content\": \"Coffee is good for you because numerous studies have shown its potential health benefits. For example, research has indicated that coffee consumption may lower the risk of certain diseases, including Parkinson's disease, liver disease, and certain types of cancer. Additionally, coffee contains caffeine, which can improve focus, alertness, and cognitive performance. Moreover, the antioxidants in coffee can help protect against oxidative stress and inflammation, which are linked to various health conditions. Therefore, incorporating coffee into a balanced diet can contribute to overall well-being.\",\n", + " \"latency\": 2.87,\n", + " \"input_tokens\": 234,\n", + " \"output_tokens\": 102,\n", + " \"total_tokens\": 336,\n", + " \"cost\": 0.0008,\n", + " \"model\": \"gpt-3.5-turbo-0613\",\n", + " \"provider\": \"OpenAIProvider('gpt-3.5-turbo-0613')\",\n", + " \"cache_hit\": false,\n", + " \"status\": \"success\",\n", + " \"start_timestamp\": \"2023-10-18 19:23:32 UTC\",\n", + " \"end_timestamp\": \"2023-10-18 19:23:36 UTC\",\n", + " \"error\": null,\n", + " \"trace_id\": null\n", + "}\n" + ] + } + ], "source": [ + "import json\n", "from parea.schemas.models import CompletionResponse\n", + "from attrs import asdict\n", "\n", "\n", "# let's return the full CompletionResponse to see what other information is returned\n", - "def refiner2(query: str, additional_description: str, argument: str, criticism: str) -> CompletionResponse:\n", + "def refiner2(query: str, additional_description: str, current_arg: str, criticism: str) -> CompletionResponse:\n", " return p.completion(\n", " Completion(\n", - " deployment_id=\"p-bJ3-UKh9-ixapZafaRBsj\",\n", + " deployment_id=\"p-rEjM4X10rJomOD8Rj9gzJ\",\n", " llm_inputs={\n", " \"additional_description\": additional_description,\n", " \"date\": f\"{datetime.now()}\",\n", " \"query\": query,\n", - " \"argument\": argument,\n", + " \"argument\": current_arg,\n", " \"criticism\": criticism,\n", " },\n", " )\n", - " )" + " )\n", + "\n", + "\n", + "@trace\n", + "def argument_chain3(query: str, additional_description: str = \"\") -> CompletionResponse:\n", + " argument = argument_generator(query, additional_description)\n", + " criticism = critic(argument)\n", + " return refiner2(query, additional_description, argument, criticism)\n", + "\n", + "\n", + "result2 = argument_chain3(\n", + " \"Whether coffee is good for you.\",\n", + " additional_description=\"Provide a concise, few sentence argument on why coffee is good for you.\",\n", + ")\n", + "print(json.dumps(asdict(result2), indent=2))" ], "metadata": { - "collapsed": false + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-10-18T19:23:36.701964Z", + "start_time": "2023-10-18T19:23:22.185768Z" + } }, "id": "27da340acb87f252" }, @@ -320,19 +476,19 @@ "- target: str - a gold standard/expected output\n", "- end_user_identifier: str - unique identifier for your end user\n", "\n", - "Below is an example. Note: you can also define these properties on the Completion object itself." + "Below is an example. Note: you can also define these properties on the Completion object itself!" ] }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 10, "id": "ac98115b", "metadata": { "tags": [], "id": "ac98115b", "ExecuteTime": { - "end_time": "2023-08-18T03:35:01.978282Z", - "start_time": "2023-08-18T03:35:01.900898Z" + "end_time": "2023-10-18T19:23:39.268716Z", + "start_time": "2023-10-18T19:23:39.263973Z" } }, "outputs": [], @@ -355,13 +511,14 @@ { "cell_type": "code", "source": [ - "import json, attrs\n", + "import json\n", + "from attrs import asdict\n", "\n", "result2, trace_id = argument_chain_tags_metadata(\n", " \"Whether coffee is good for you.\",\n", " additional_description=\"Provide a concise, few sentence argument on why coffee is good for you.\",\n", ")\n", - "print(json.dumps(attrs.asdict(result2), indent=2))\n", + "print(json.dumps(asdict(result2), indent=2))\n", "\n", "p.record_feedback(\n", " FeedbackRequest(\n", @@ -376,11 +533,39 @@ "base_uri": "https://localhost:8080/" }, "id": "ggpS9JJ-dn4u", - "outputId": "6a324d73-8b6c-494d-fad3-f0ba0578cc4a" + "outputId": "6a324d73-8b6c-494d-fad3-f0ba0578cc4a", + "ExecuteTime": { + "end_time": "2023-10-18T19:24:06.692012Z", + "start_time": "2023-10-18T19:23:42.605033Z" + } }, "id": "ggpS9JJ-dn4u", - "execution_count": null, - "outputs": [] + "execution_count": 11, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"inference_id\": \"fb7910bd-0438-4f80-b128-a81b81f6e5a3\",\n", + " \"content\": \"Coffee is good for you because it contains a high amount of antioxidants, which have been linked to various health benefits. While the argument acknowledges that causation cannot be definitively proven, numerous studies have shown a consistent association between coffee consumption and a reduced risk of chronic diseases, such as heart disease, certain cancers, and neurodegenerative disorders like Parkinson's disease. Additionally, moderate coffee consumption has been associated with improved cognitive function, increased metabolism, and a lower risk of developing conditions like type 2 diabetes. While individual differences and potential negative effects should be considered, the overall body of research suggests that coffee can be a beneficial addition to a healthy lifestyle.\",\n", + " \"latency\": 3.98,\n", + " \"input_tokens\": 469,\n", + " \"output_tokens\": 131,\n", + " \"total_tokens\": 600,\n", + " \"cost\": 0.0012,\n", + " \"model\": \"gpt-3.5-turbo-0613\",\n", + " \"provider\": \"OpenAIProvider('gpt-3.5-turbo-0613')\",\n", + " \"cache_hit\": false,\n", + " \"status\": \"success\",\n", + " \"start_timestamp\": \"2023-10-18 19:23:57 UTC\",\n", + " \"end_timestamp\": \"2023-10-18 19:24:02 UTC\",\n", + " \"error\": null,\n", + " \"trace_id\": null\n", + "}\n" + ] + } + ] }, { "cell_type": "markdown", diff --git a/pyproject.toml b/pyproject.toml index 3b6f2487..a8a6817c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -6,7 +6,7 @@ build-backend = "poetry.core.masonry.api" [tool.poetry] name = "parea-ai" packages = [{ include = "parea" }] -version = "0.2.12" +version = "0.2.13a0" description = "Parea python sdk" readme = "README.md" authors = ["joel-parea-ai "]