I use Stochastic Radiant to create a submission for ForecastBench. Both of these tools are intended for demonstration purposes. However, I plan to continue development of Stochastic Radiant, with associated evaluation in this repo.
The ForecastBench Wiki is available here.
The forecast routine requires a .env file in the project root:
ANTHROPIC_API_KEY={your_anthropic_api_key}
Use fb-forecast to generate forecasts on the latest question set (question_set below). Example usage:
make fb-forecast file_prefix="srfb_demo" model="claude-3-5-haiku-20241022"Arguments:
file_prefixcontrols the output file name (forecasts_file_prefix_question_set)modelspecifies the model (must be an Anthropic model, as Stochastic Radiant only supports Antrhopic at present)
Use fb-prepare to prepare a ForecastBench submission file. Example usage:
make fb-prepare file_prefix="srfb_demo" question_set="2025-08-03-llm" organization="Stochastic Radiant" model="claude-3-5-haiku-20241022" model_organization="Anthropic"Arguments:
file_prefix: Identifier for the forecast files (e.g., "srfb_demo")question_set: Name of the question set being forecasted (e.g., "2025-08-03-llm")organization: Your organization name (for the ForecastBench leaderboard)model: The model used to generate forecastsmodel_organization: Organization that created the model
Use fb-post to post a forecast set to the relevant GCS bucket. Example usage:
make fb-post file_prefix="srfb_demo" question_set="2025-08-03-llm" organization="Stochastic Radiant" N=1 upload_to_gcs=Truefile_prefix: Identifier for the forecast files (e.g., "srfb_demo")question_set: Name of the question set being forecasted (e.g., "2025-08-03-llm")organization: Your organization name (for the ForecastBench leaderboard)N: the submission number (integer)upload_to_gcs: Whether to upload the submission to Google Cloud Storage (True/False). If False, only creates a local copy.
Please contact me if you're interested in contributing. These submission tools and Stochastic Radiant are both in active development.