add cost and token logging in openai online and batching with litellm completion_cost

[CharlieJCJ]

Changes:

• expose metrics (i.e. cost and tokens) on the top-level in GenericResponse

Resolving Issues:

• [Generic Response Expose Metrics]: token statistics #146
• [Generic Response Expose Metrics]: generation costs #147

Related Issues:

• Expose more metrics from the raw_response to cache generic request / response #57
• Add more model support with liteLLM #74

Next steps:

• Add cost info in viewer
• Progress bar with cost estimates

[CharlieJCJ]

Testing:

• examples/poem.py on GPT 4o, GPT 4o-mini
• examples/distill.py batch=True, on GPT 4o, GPT 4o-mini

Cost and token number look accurate.

[CharlieJCJ]

Example output from GPT 4o-mini on poem.py

{
  "response_message": {
    "poems_list": [
      "**A Day in the Life of a Wandering Cloud**\n\nUp in the sky, I drift and glide,  \nA wandering cloud, a nomad wide,  \nWith fluffy form and a heart so free,  \nI dance with the wind, a spirit at sea.\n\nAt dawn’s embrace, I take my flight,  \nPainted in hues of golden light,  \nI cradle the sun on my soft, white back,  \nAs shadows of night begin to slack.\n\nThrough fields of blue, I roam and muse,  \nOver mountains tall and rivers’ hues,  \nI wander afar, where the lilies sway,  \nAnd whisper secrets of the day.\n\nI coat the sky with a cottony brush,  \nKissing the earth in a gentle hush,  \nWith rain that drips like sweet lullabies,  \nGiving thirsting flowers a soft surprise.\n\nBy noon I’m lost in a sea of sun,  \nReflecting the joy of the world’s long run,  \nI scatter my shade o’er a sunbaked street,  \nWhile children look up, their laughter sweet.\n\nAs afternoon wanes, I’m painted in gray,  \nHeavy with thoughts of the end of the day,  \nI gather my rains and prepare to weep,  \nAs twilight whispers, I’m ready to sleep.\n\nRolling onwards, through dusk’s embrace,  \nI watch the stars in their velvet space,  \nMoonslight beckons, the night draws near,  \nA wandering cloud, with nothing to fear.  \n\nSo I drift through the night, a soft sight to spy,  \nA promise to rise with the day by and by,  \nAnd thus my tale ends, but only for now,  \nTomorrow’s adventure awaits this cloud!",
      "**The Journey of a Wandering Cloud**\n\nHigh above the world, where the eagles soar,  \nI’m a wandering cloud, forever wanting more,  \nA sailor of skies, with a shifting plan,  \nTracing the edges of nature’s span.\n\nIn the early light, I stretch and form,  \nA billowy shape, profoundly warm,  \nWith threads of silver and whispers of blue,  \nI sail the heavens, an ethereal view.\n\nThrough valleys and forests, o’er hills so green,  \nI peek down gently, the earth’s soft sheen,  \nI tickle the treetops, a playful caress,  \nWith a sprinkle of raindrops, I bless and press.\n\nBy noon’s bright blaze, I morph and swell,  \nAs children look up, they know me well,  \nI cast cool shadows on hot sandy shores,  \nA guardian spirit, the sky opens doors.\n\nBut storms will gather, a tempest brews,  \nWith rumbling thunder and darkened hues,  \nI gather my courage, embrace my design,  \nTo rain down blessings, a drenched divine.\n\nAs evening descends, I blush with the sun,  \nPainting the horizon—my day has been fun,  \nWith streaks of purple, and reds shining bright,  \nI whirl through the sky in the fading light.\n\nSo, when the stars emerge one by one,  \nI drift ever lightly, my journey not done,  \nFor tomorrow I’ll roam on a new summer's breeze,  \nAs a wandering cloud, I dance with such ease."
    ]
  },
  "response_errors": null,
  "raw_response": {
    "id": "chatcmpl-AVpkuxPgReKS8ZUfwyzc9WmtVxbZy",
    "object": "chat.completion",
    "created": 1732150480,
    "model": "gpt-4o-mini-2024-07-18",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "{\"poems_list\":[\"**A Day in the Life of a Wandering Cloud**\\n\\nUp in the sky, I drift and glide,  \\nA wandering cloud, a nomad wide,  \\nWith fluffy form and a heart so free,  \\nI dance with the wind, a spirit at sea.\\n\\nAt dawn’s embrace, I take my flight,  \\nPainted in hues of golden light,  \\nI cradle the sun on my soft, white back,  \\nAs shadows of night begin to slack.\\n\\nThrough fields of blue, I roam and muse,  \\nOver mountains tall and rivers’ hues,  \\nI wander afar, where the lilies sway,  \\nAnd whisper secrets of the day.\\n\\nI coat the sky with a cottony brush,  \\nKissing the earth in a gentle hush,  \\nWith rain that drips like sweet lullabies,  \\nGiving thirsting flowers a soft surprise.\\n\\nBy noon I’m lost in a sea of sun,  \\nReflecting the joy of the world’s long run,  \\nI scatter my shade o’er a sunbaked street,  \\nWhile children look up, their laughter sweet.\\n\\nAs afternoon wanes, I’m painted in gray,  \\nHeavy with thoughts of the end of the day,  \\nI gather my rains and prepare to weep,  \\nAs twilight whispers, I’m ready to sleep.\\n\\nRolling onwards, through dusk’s embrace,  \\nI watch the stars in their velvet space,  \\nMoonslight beckons, the night draws near,  \\nA wandering cloud, with nothing to fear.  \\n\\nSo I drift through the night, a soft sight to spy,  \\nA promise to rise with the day by and by,  \\nAnd thus my tale ends, but only for now,  \\nTomorrow’s adventure awaits this cloud!\",\"**The Journey of a Wandering Cloud**\\n\\nHigh above the world, where the eagles soar,  \\nI’m a wandering cloud, forever wanting more,  \\nA sailor of skies, with a shifting plan,  \\nTracing the edges of nature’s span.\\n\\nIn the early light, I stretch and form,  \\nA billowy shape, profoundly warm,  \\nWith threads of silver and whispers of blue,  \\nI sail the heavens, an ethereal view.\\n\\nThrough valleys and forests, o’er hills so green,  \\nI peek down gently, the earth’s soft sheen,  \\nI tickle the treetops, a playful caress,  \\nWith a sprinkle of raindrops, I bless and press.\\n\\nBy noon’s bright blaze, I morph and swell,  \\nAs children look up, they know me well,  \\nI cast cool shadows on hot sandy shores,  \\nA guardian spirit, the sky opens doors.\\n\\nBut storms will gather, a tempest brews,  \\nWith rumbling thunder and darkened hues,  \\nI gather my courage, embrace my design,  \\nTo rain down blessings, a drenched divine.\\n\\nAs evening descends, I blush with the sun,  \\nPainting the horizon—my day has been fun,  \\nWith streaks of purple, and reds shining bright,  \\nI whirl through the sky in the fading light.\\n\\nSo, when the stars emerge one by one,  \\nI drift ever lightly, my journey not done,  \\nFor tomorrow I’ll roam on a new summer's breeze,  \\nAs a wandering cloud, I dance with such ease.\"]}",
          "refusal": null
        },
        "logprobs": null,
        "finish_reason": "stop"
      }
    ],
    "usage": {
      "prompt_tokens": 79,
      "completion_tokens": 794,
      "total_tokens": 873,
      "prompt_tokens_details": {
        "cached_tokens": 0,
        "audio_tokens": 0
      },
      "completion_tokens_details": {
        "reasoning_tokens": 0,
        "audio_tokens": 0,
        "accepted_prediction_tokens": 0,
        "rejected_prediction_tokens": 0
      }
    },
    "system_fingerprint": "fp_0705bf87c0"
  },
  "raw_request": {
    "model": "gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": "Write two poems about A Day in the Life of a Wandering Cloud."
      }
    ],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "output_schema",
        "schema": {
          "properties": {
            "poems_list": {
              "description": "A list of poems.",
              "items": {
                "type": "string"
              },
              "title": "Poems List",
              "type": "array"
            }
          },
          "required": [
            "poems_list"
          ],
          "title": "Poems",
          "type": "object"
        }
      }
    }
  },
  "generic_request": {
    "model": "gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": "Write two poems about A Day in the Life of a Wandering Cloud."
      }
    ],
    "response_format": {
      "properties": {
        "poems_list": {
          "description": "A list of poems.",
          "items": {
            "type": "string"
          },
          "title": "Poems List",
          "type": "array"
        }
      },
      "required": [
        "poems_list"
      ],
      "title": "Poems",
      "type": "object"
    },
    "original_row": {
      "topic": "A Day in the Life of a Wandering Cloud"
    },
    "original_row_idx": 7
  },
  "created_at": "2024-11-21 00:54:40.638421",
  "finished_at": "2024-11-21 00:54:46.237762",
  "token_usage": {
    "prompt_tokens": 79,
    "completion_tokens": 794,
    "total_tokens": 873
  },
  "response_cost": 0.00048824999999999997
}

(79 * 0.150 + 794 * 0.6) / 1e6 = 0.00048824999999999997 ✅

Example output from GPT 4 on poem.py

{
  "response_message": {
    "poems_list": [
      "### The Sun-Kissed Wanderer\n\n**At Dawn's Gentle Embrace:**  \nSprung from misty dreams of night,  \nA cloud awakens, soft and white,  \nWith the sun's first gentle beams,  \nStretches o'er the earth and streams.\n\nBlown by breezes, light and free,  \nIt drifts towards the open sea,  \nOver hills with morning dew,  \nBathed in sky's eternal blue.\n\n**The Midday Dance:**  \nOver grasslands, vast and wide,  \nWith the sun now high in the sky,  \nThe cloud takes on a cotton shape,  \nCasting cool shadows, sweet escape.\n\nOn its journey, whispering tales,  \nTo eagles soaring, as it sails,  \nLeaves glisten in its gentle wake,  \nAs rivers with its soft shade quake.  \n\n**Towards Dusk's Gentle Farewell:**\nAs the sun sinks low, turning gold  \nThe cloud crests mountains, strong and bold,  \nDraped in colors of fiery hue,  \nA tapestry of pink, orange, blue.\n\nIn twilight's arms, it finds its rest,  \nOn a mountaintop's airy nest.  \nThe wider world hushes to sleep,  \nAs stars ascend and shadows creep.\n\n### Whispering Wanderer’s Lullaby\n\n**The Morning Call:**  \nFrom dreams of moonlit realms afar,  \nEmerges at dawn a wandering star  \nIn a sea of skies where the chambered mist  \nKisses the world in a sunlit tryst.\n\nBreezes greet the cloud with cool caress,  \nIt glides with a din of quietness,  \nFloating past trees, sharing secrets, small,  \nLeaves and branches nod to its call.\n\n**The Noon Whisper:**  \nUnleashed by warmth, the cloud runs free,  \nIts shadow a ship upon the sea,  \nWith sailors below wishing voyages bright  \nBeneath its alabaster flight.\n\nPeeking through forests dense and wide,  \nIt weaves between the sun’s pride,  \nTossing travels of a thousand miles  \nIn form of playful, wispy smiles.\n\n**The Evening Embrace:**\nSunset waltzing with colors, bold,  \nTurns a weary cloud to shades of gold,  \nAs heavens paint its curtain call  \nWith wondrous fervor over all.\n\nNestled within the twilight hush,  \nOn mountain snow with glowing blush,  \nSlumber speaks in airy tunes,  \nBeneath the gaze of silvery moons."
    ]
  },
  "response_errors": null,
  "raw_response": {
    "id": "chatcmpl-AVpo5spkMtbKWdDSqpU6M4Asaz1Ab",
    "object": "chat.completion",
    "created": 1732150677,
    "model": "gpt-4o-2024-08-06",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "{\"poems_list\":[\"### The Sun-Kissed Wanderer\\n\\n**At Dawn's Gentle Embrace:**  \\nSprung from misty dreams of night,  \\nA cloud awakens, soft and white,  \\nWith the sun's first gentle beams,  \\nStretches o'er the earth and streams.\\n\\nBlown by breezes, light and free,  \\nIt drifts towards the open sea,  \\nOver hills with morning dew,  \\nBathed in sky's eternal blue.\\n\\n**The Midday Dance:**  \\nOver grasslands, vast and wide,  \\nWith the sun now high in the sky,  \\nThe cloud takes on a cotton shape,  \\nCasting cool shadows, sweet escape.\\n\\nOn its journey, whispering tales,  \\nTo eagles soaring, as it sails,  \\nLeaves glisten in its gentle wake,  \\nAs rivers with its soft shade quake.  \\n\\n**Towards Dusk's Gentle Farewell:**\\nAs the sun sinks low, turning gold  \\nThe cloud crests mountains, strong and bold,  \\nDraped in colors of fiery hue,  \\nA tapestry of pink, orange, blue.\\n\\nIn twilight's arms, it finds its rest,  \\nOn a mountaintop's airy nest.  \\nThe wider world hushes to sleep,  \\nAs stars ascend and shadows creep.\\n\\n### Whispering Wanderer’s Lullaby\\n\\n**The Morning Call:**  \\nFrom dreams of moonlit realms afar,  \\nEmerges at dawn a wandering star  \\nIn a sea of skies where the chambered mist  \\nKisses the world in a sunlit tryst.\\n\\nBreezes greet the cloud with cool caress,  \\nIt glides with a din of quietness,  \\nFloating past trees, sharing secrets, small,  \\nLeaves and branches nod to its call.\\n\\n**The Noon Whisper:**  \\nUnleashed by warmth, the cloud runs free,  \\nIts shadow a ship upon the sea,  \\nWith sailors below wishing voyages bright  \\nBeneath its alabaster flight.\\n\\nPeeking through forests dense and wide,  \\nIt weaves between the sun’s pride,  \\nTossing travels of a thousand miles  \\nIn form of playful, wispy smiles.\\n\\n**The Evening Embrace:**\\nSunset waltzing with colors, bold,  \\nTurns a weary cloud to shades of gold,  \\nAs heavens paint its curtain call  \\nWith wondrous fervor over all.\\n\\nNestled within the twilight hush,  \\nOn mountain snow with glowing blush,  \\nSlumber speaks in airy tunes,  \\nBeneath the gaze of silvery moons.\"]}",
          "refusal": null
        },
        "logprobs": null,
        "finish_reason": "stop"
      }
    ],
    "usage": {
      "prompt_tokens": 79,
      "completion_tokens": 622,
      "total_tokens": 701,
      "prompt_tokens_details": {
        "cached_tokens": 0,
        "audio_tokens": 0
      },
      "completion_tokens_details": {
        "reasoning_tokens": 0,
        "audio_tokens": 0,
        "accepted_prediction_tokens": 0,
        "rejected_prediction_tokens": 0
      }
    },
    "system_fingerprint": "fp_7f6be3efb0"
  },
  "raw_request": {
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "Write two poems about A Day in the Life of a Wandering Cloud."
      }
    ],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "output_schema",
        "schema": {
          "properties": {
            "poems_list": {
              "description": "A list of poems.",
              "items": {
                "type": "string"
              },
              "title": "Poems List",
              "type": "array"
            }
          },
          "required": [
            "poems_list"
          ],
          "title": "Poems",
          "type": "object"
        }
      }
    }
  },
  "generic_request": {
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "Write two poems about A Day in the Life of a Wandering Cloud."
      }
    ],
    "response_format": {
      "properties": {
        "poems_list": {
          "description": "A list of poems.",
          "items": {
            "type": "string"
          },
          "title": "Poems List",
          "type": "array"
        }
      },
      "required": [
        "poems_list"
      ],
      "title": "Poems",
      "type": "object"
    },
    "original_row": {
      "topic": "A Day in the Life of a Wandering Cloud"
    },
    "original_row_idx": 7
  },
  "created_at": "2024-11-21 00:57:57.274561",
  "finished_at": "2024-11-21 00:58:13.849180",
  "token_usage": {
    "prompt_tokens": 79,
    "completion_tokens": 622,
    "total_tokens": 701
  },
  "response_cost": 0.0064175000000000005
}

(79 * 2.50 + 622 * 10.00) / 1e6 = 0.0064175000000000005 ✅

[vutrung96]

Thanks for verifying @CharlieJCJ