Feature/thetadata speed v2

.github/workflows/cicd.yaml

@@ -41,9 +41,18 @@ jobs:
 
     env:
       AIOHTTP_NO_EXTENSIONS: 1
+      # CRITICAL: Set this to "none" so tests use their explicit data sources.
+      # Tests that want ThetaData must explicitly request it.
+      # Without this, the default is "ThetaData" which overrides ALL backtests.
+      BACKTESTING_DATA_SOURCE: none
       POLYGON_API_KEY: ${{ secrets.POLYGON_API_KEY }}
       THETADATA_USERNAME: ${{ secrets.THETADATA_USERNAME }}
       THETADATA_PASSWORD: ${{ secrets.THETADATA_PASSWORD }}
+      # NOTE (2025-11-28): Data Downloader is a production proxy for ThetaData that allows
+      # shared access without requiring a local ThetaTerminal JAR. When these are set,
+      # ThetaData tests will use the remote downloader instead of spawning a local process.
+      DATADOWNLOADER_BASE_URL: ${{ secrets.DATADOWNLOADER_BASE_URL }}
+      DATADOWNLOADER_API_KEY: ${{ secrets.DATADOWNLOADER_API_KEY }}
       ALPACA_TEST_API_KEY: ${{secrets.ALPACA_TEST_API_KEY}} # Required for alpaca unit tests
       ALPACA_TEST_API_SECRET: ${{secrets.ALPACA_TEST_API_SECRET}} # Required for alpaca unit tests
       TRADIER_TEST_ACCESS_TOKEN: ${{secrets.TRADIER_TEST_ACCESS_TOKEN}} # Required for tradier unit tests

.gitignore

@@ -51,3 +51,7 @@ data/SRNE_Minute.csv
 .env
 token.json
 schwab_token.json
+
+# AI assistant instruction files (contain local paths)
+CLAUDE.md
+AGENTS.md

README.md

@@ -5,7 +5,16 @@
 
 Lumibot is a backtesting and trading library for stocks, options, crypto, futures and more. It is made so that the same code you use for backtesting can be used for live trading, making it easy to transition from backtesting to live trading. Lumibot is a highly flexible library that allows you to create your own strategies and indicators, and backtest them on historical data. It is also highly optimized for speed, so you can backtest your strategies quickly and efficiently.
 
-**IMPORTANT: This library requires data for backtesting. The recommended data source is [Polygon.io](https://polygon.io/?utm_source=affiliate&utm_campaign=lumi10) (a free tier is available too). Please click the link to give us credit for the sale, it helps support this project. You can use the coupon code 'LUMI10' for 10% off.**
+**IMPORTANT: This library requires data for backtesting. Our recommended data source is [ThetaData](https://www.thetadata.net/) because they provide the deepest historical coverage we’ve found and directly support BotSpot. Use the promo code `BotSpot10` at checkout for 10% off the first order (the code also tells ThetaData you were referred by us).**
+
+> **Contributor note:** Read `AGENTS.md` before running anything Theta-related. That file spells out the hard rules—never launch ThetaTerminal or the shared downloader locally, always point LumiBot at the AWS-hosted downloader, and wrap all long
+> commands with `/Users/robertgrzesik/bin/safe-timeout`. Breaking these rules kills the only licensed Theta session.
+
+## Architecture Documentation
+
+- `BACKTESTING_ARCHITECTURE.md` - Detailed documentation of the backtesting data flow (Yahoo, ThetaData, Polygon data sources, caching, and data flow diagrams)
+- `CLAUDE.md` - AI assistant instructions for working with the codebase
+- `AGENTS.md` - Critical rules for ThetaData and production safety
 
 ## Documentation - 👇 Start Here 👇
 

docsrc/backtesting.how_to_backtest.rst

@@ -1,7 +1,7 @@
 How To Backtest
 ===================================
 
-Backtesting is a vital step in validating your trading strategies using historical data. With LumiBot, you can backtest strategies across various data sources such as **Yahoo Finance**, **Polygon.io**, **ThetaData**, or even your own custom **CSV** files. This guide will walk you through each step of backtesting, explain the data sources, and introduce the files that LumiBot generates during backtesting.
+Backtesting is a vital step in validating your trading strategies using historical data. With LumiBot, you can backtest strategies across various data sources such as **ThetaData** (our recommended vendor), **Polygon.io**, **Yahoo Finance**, or even your own custom **CSV** files. This guide will walk you through each step of backtesting, explain the data sources, and introduce the files that LumiBot generates during backtesting.
 
 .. note::
 
@@ -41,38 +41,33 @@ Choosing a Data Source
 
 LumiBot supports several data sources for backtesting, each suited for different asset types and backtesting needs. Here's an overview of the available sources:
 
-**1. Yahoo Finance**
+**1. ThetaData (Recommended)**
 
-- Free stock and ETF data for daily trading backtests.
-- Suitable for longer-term strategies but not ideal for intraday backtesting.
-
-For more details, see the :ref:`Yahoo Backtesting <backtesting.yahoo>` section.
+- Deep historical coverage for U.S. equities and options with SIP-quality filtering.
+- Offers free tiers plus paid plans with higher rate limits and multi-year history.
 
-**2. Polygon.io**
+.. important::
 
-- Offers intraday and end-of-day data for stocks, options, forex, and cryptocurrency.
-- Provides up to two years of free data; paid plans offer more advanced features and faster data retrieval.
+   **Get Your ThetaData Account**
 
-.. important::
+   Sign up at `ThetaData <https://www.thetadata.net/>`_. Use the promo code ``BotSpot10`` for 10% off the first order—ThetaData uses this code to credit BotSpot for the referral.
 
-   **Get Your API Key from Polygon.io**
-
-   You can get an API key at `Polygon.io <https://polygon.io/?utm_source=affiliate&utm_campaign=lumi10>`_. **Please use the coupon code 'LUMI10' for 10% off!**
+For more details, see the :ref:`ThetaData Backtesting <backtesting.thetadata>` section.
 
-For more details, see the :ref:`Polygon.io Backtesting <backtesting.polygon>` section.
+**2. Yahoo Finance**
 
-**3. ThetaData**
+- Free stock and ETF data for daily trading backtests.
+- Suitable for longer-term strategies but not ideal for intraday backtesting.
 
-- Designed for users looking to backtest stock and options trading strategies.
-- Provides options pricing and other securities.
+For more details, see the :ref:`Yahoo Backtesting <backtesting.yahoo>` section.
 
-.. important::
+**3. Polygon.io**
 
-   **Get Your ThetaData Account**
-   
-   You can get a username and password at `thetadata.net <https://www.thetadata.net/>`_. **Please use the coupon code 'LUMI' for 10% off!**
+- Offers intraday and end-of-day data for stocks, options, forex, and cryptocurrency.
+- Provides up to two years of free data; paid plans offer more advanced features and faster data retrieval.
+- Best suited for existing workflows; new LumiBot users should consider ThetaData first for the BotSpot10 promo and deeper coverage.
 
-For more details, see the :ref:`ThetaData Backtesting <backtesting.thetadata>` section.
+For more details, see the :ref:`Polygon.io Backtesting <backtesting.polygon>` section.
 
 **4. Pandas (CSV or Other Custom Data)**
 

docsrc/backtesting.polygon.rst

@@ -5,7 +5,7 @@ Polygon.io Backtesting
 
 .. important::
 
-   **You can get an API key at** `Polygon.io <https://polygon.io/?utm_source=affiliate&utm_campaign=lumi10>`_. **Please use the full link to give us credit for the sale (https://polygon.io/?utm_source=affiliate&utm_campaign=lumi10), it helps support this project. You can use the coupon code 'LUMI10' for 10% off.**
+   **ThetaData is our preferred data partner and the service we recommend to most LumiBot users—sign up at** `ThetaData <https://www.thetadata.net/>`_ **and use the promo code ``BotSpot10`` for 10% off the first order.** This section remains for teams that still need Polygon.io. If you require Polygon access you can create an account at `polygon.io <https://polygon.io/>`_.
 
 Polygon.io backtester allows for flexible and robust backtesting. It uses the polygon.io API to fetch pricing data for stocks, options, forex, and cryptocurrencies. This backtester simplifies the process of getting pricing data; simply use the PolygonDataSource and it will automatically fetch pricing data when you call `get_last_price()` or `get_historical_prices()`.
 
@@ -104,7 +104,7 @@ Here's the full code (with explicit dates):
 
 .. important::
 
-   **You can get an API key at** `Polygon.io <https://polygon.io/?utm_source=affiliate&utm_campaign=lumi10>`_. **Please use the full link to give us credit for the sale (https://polygon.io/?utm_source=affiliate&utm_campaign=lumi10), it helps support this project. You can use the coupon code 'LUMI10' for 10% off.**
+   **ThetaData remains our recommended vendor (promo code ``BotSpot10`` at `thetadata.net <https://www.thetadata.net/>`_). These Polygon instructions are provided for existing workflows that still rely on Polygon’s API.**
 
 Optional: Environment Variables
 -------------------------------

docsrc/backtesting.thetadata.rst

@@ -5,7 +5,7 @@ ThetaData Backtesting
 
 .. important::
 
-   **You can get a username and password at** `thetadata.net <https://www.thetadata.net/>`_. **Please use the full link to give us credit for the sale (https://www.thetadata.net), it helps support this project. You can use the coupon code 'LUMI' for 10% off.**
+   **Sign up at** `ThetaData <https://www.thetadata.net/>`_. **Use the promo code ``BotSpot10`` at checkout for 10% off the first order—ThetaData tracks the code so they can credit BotSpot for the referral.**
 
 ThetaData backtester allows for flexible and robust backtesting. It uses the thetadata API to fetch pricing data for stocks, options, forex, and cryptocurrencies. This backtester simplifies the process of getting pricing data; simply use the thetadata DataSource and it will automatically fetch pricing data when you call `get_last_price()` or `get_historical_prices()`.
 

lumibot/backtesting/backtesting_broker.py

@@ -257,9 +257,25 @@ def get_historical_account_value(self):
 
     # =========Internal functions==================
 
-    def _update_datetime(self, update_dt, cash=None, portfolio_value=None):
+    def _update_datetime(self, update_dt, cash=None, portfolio_value=None, positions=None, initial_budget=None, orders=None):
         """Works with either timedelta or datetime input
-        and updates the datetime of the broker"""
+        and updates the datetime of the broker.
+
+        Parameters
+        ----------
+        update_dt : timedelta, int, float, or datetime
+            The time to advance by (if timedelta/int/float) or the new datetime
+        cash : float, optional
+            Current cash balance
+        portfolio_value : float, optional
+            Current portfolio value
+        positions : list, optional
+            List of minimal position dicts from Position.to_minimal_dict()
+        initial_budget : float, optional
+            Initial budget for calculating return percentage
+        orders : list, optional
+            List of minimal order dicts from Order.to_minimal_dict()
+        """
         tz = self.datetime.tzinfo
         is_pytz = isinstance(tz, (pytz.tzinfo.StaticTzInfo, pytz.tzinfo.DstTzInfo))
 
@@ -281,7 +297,14 @@ def _update_datetime(self, update_dt, cash=None, portfolio_value=None):
             if is_pytz:
                 new_datetime = tz.normalize(new_datetime)
 
-        self.data_source._update_datetime(new_datetime, cash=cash, portfolio_value=portfolio_value)
+        self.data_source._update_datetime(
+            new_datetime,
+            cash=cash,
+            portfolio_value=portfolio_value,
+            positions=positions,
+            initial_budget=initial_budget,
+            orders=orders
+        )
         if self.option_source:
             self.option_source._update_datetime(new_datetime, cash=cash, portfolio_value=portfolio_value)
 

lumibot/backtesting/polygon_backtesting.py

@@ -154,7 +154,8 @@ def _update_pandas_data(self, asset, quote, length, timestep, start_dt=None):
                 msg = (
                     "Polygon Access Denied: Your subscription does not allow you to backtest that far back in time. "
                     f"Requested {asset_separated} {ts_unit} bars from {formatted_start_datetime} to {formatted_end_datetime}. "
-                    "Consider starting later or upgrading your Polygon subscription (https://polygon.io/?utm_source=affiliate&utm_campaign=lumi10, code 'LUMI10')."
+                    "We strongly recommend switching to ThetaData (https://www.thetadata.net/ with promo code 'BotSpot10') for better coverage, speed, and LumiBot-native support. "
+                    "If you must stay on Polygon, consider starting later or upgrading your Polygon plan (https://polygon.io/?utm_source=affiliate&utm_campaign=lumi10, code 'LUMI10')."
                 )
                 logger.error(colored(msg, color="red"))
                 return
@@ -164,7 +165,8 @@ def _update_pandas_data(self, asset, quote, length, timestep, start_dt=None):
                     "Please check your API key and try again. "
                     "You can get an API key at https://polygon.io/?utm_source=affiliate&utm_campaign=lumi10 "
                     "Please use the full link to give us credit for the sale, it helps support this project. "
-                    "You can use the coupon code 'LUMI10' for 10% off. ",
+                    "You can use the coupon code 'LUMI10' for 10% off. "
+                    "We recommend switching to ThetaData (https://www.thetadata.net/ with promo code 'BotSpot10') for higher-quality, faster data and first-class support in LumiBot. ",
                     color="red")
                 raise Exception(error_message) from e
             else: