Dynamic Types Tests

[antoniosarosi]

Overview

Classes and enums marked as @@dynamic cannot be tested properly since there is no way to specify dynamic class fields or dynamic enum variants for return types. On the other hand, dynamic parameter types already work but only in the case of classes, dynamic enums are not handled properly.

Dynamic Return Types

Enums

Basic example of dynamic enum return type:

// Dynamic Enum
enum Category {
    Refund
    CancelOrder
    TechnicalSupport
    AccountIssue
    Question
    @@dynamic // Marked as @@dynamic
}

// Function that returns dynamic enum
function ClassifyMessage(message: string) -> Category { // Prompt ... }

// Test case
test Test {
  functions [ClassifyMessage]
  args {
    message "I want to cancel my order"
  }
}

There is no syntax for adding new variants to the Category enum in the test case, so the rendered prompt will only contain the statically defined variants of Category:

Answer with any of the categories:
Category
----
- Refund
- CancelOrder
- TechnicalSupport
- AccountIssue
- Question

Classes

We have the same problem when using dynamic classes as return types instead of enums:

// Dynamic class
class Resume {
  name string
  education Education[]
  skills string[]
  @@dynamic // Marked as @@dynamic
}

// Nested object
class Education {
  school string
  degree string
  year int
}

// Function that returns dynamic class
function ExtractResume(resume_text: string) -> Resume { // Prompt ... }

// Test case
test Test {
  functions [ExtractResume]
  args {
    resume_text #"
      John Doe
      
      Experience
      - Software Engineer, Boundary, Sep 2023 - Present
      - Software Engineer, Google, Sep 2022 - Sep 2023

      Education
      - University of California, Berkeley, B.S. in Computer Science, 2020

      Skills
      - Python
      - Java
      - C++
    "#
  }
}

Suppose we want to add a field called experience of this type:

class Experience {
  title string
  company string
  start_date string
  end_date string
}

There is no syntax that allows us to do that in the test, so the rendered prompt will only contain static fields defined in the Resume class:

Answer in JSON using this schema:
{
  name: string,
  education: [
    {
      school: string,
      degree: string,
      year: int,
    }
  ],
  skills: string[],
}

Dynamic Parameter Types

Enums

We cannot add dynamic variants to an enum because BAML reports them as errors:

enum Category {
    Refund
    CancelOrder
    TechnicalSupport
    AccountIssue
    Question
    @@dynamic
}

function WriteMessage(category: Category) -> string { // Prompt ... }

test Test {
  functions [WriteMessage]
  args {
    category FeatureRequest
  }
}

BAML Error:

Error: category: Invalid enum Category: expected one of (Refund | CancelOrder | TechnicalSupport | AccountIssue | Question), got `FeatureRequest`

Classes

Dynamic classes are rendered correctly since all the additional properties we write in the args block will get rendered in the prompt:

class Resume {
  name string
  education Education[]
  skills string[]
  @@dynamic
}

class Education {
  school string
  degree string
  year int
}

function WriteResume(resume: Resume) -> string { // Prompt ... }

test TestName {
  functions [WriteResume]
  args {
    resume {
      name "John Doe"
      skills ["C++", "Java"]
      education [
        {
          school "University of California, Berkeley"
          degree "BS in Computer Science"
          year 2020
        }
      ]
      experience [
        {
          title "Software Engineer"
          company "Boundary"
          start_date "2023-09-01"
          end_date "2024-09-01"
        }
      ]
    }  
  }
}

Resulting rendered resume in the prompt:

{
    "name": "John Doe",
    "education": [
        {
            "school": "University of California, Berkeley",
            "degree": "BS in Computer Science",
            "year": 2020,
        },
    ],
    "skills": [
        "C++",
        "Java",
    ],
    "experience": [
        {
            "title": "Software Engineer",
            "company": "Boundary",
            "start_date": "2023-09-01",
            "end_date": "2024-09-01",
        },
    ],
}

Proposed Syntax

The proposed syntax for specifying dynamic properties and variants in tests introduces the new type_builder and dynamic blocks.

Dynamic Classes

class Resume {
  name string
  education Education[]
  skills string[]
  @@dynamic // This class is marked as @@dynamic.
}

class Education {
  school string
  degree string
  year int
}

// This function returns the dynamic class defined above.
function ExtractResume(from_text: string) -> Resume { // Prompt ... }

test ReturnDynamicClassTest {
  functions [ExtractResume]
  // New type_builder block used to define types and inject dynamic props.
  type_builder {
    // Defines a new type available only within this test block.
    class Experience {
      title string
      company string
      start_date string
      end_date string
    }
 
    // This `dynamic` block is used to inject new properties into the
    // `@@dynamic` part of the Resume class.
    dynamic Resume {
      experience Experience[]
    }
  }
  args {
    from_text #"
      John Doe

      Education
      - University of California, Berkeley, B.S. in Computer Science, 2020

      Experience
      - Software Engineer, Boundary, Sep 2022 - Sep 2023
      
      Skills
      - Python
      - Java
    "#
  }
}

Resulting prompt during test execution:

Answer in JSON using this schema:
{
  name: string,
  skills: string[],
  education [
    {
      school: string,
      degree: string,
      year: int
    }
  ],
  experience: [
    {
      title: string,
      company: string,
      start_date: string,
      end_date: string,
    }
  ],
}

Dynamic Enums

enum Category {
  Refund
  CancelOrder
  AccountIssue
  @@dynamic // Marked as @@dynamic.
}

// Function that returns the dynamic enum.
function ClassifyMessage(message: string) -> Category { // Prompt ... }

test ReturnDynamicEnumTest {
  functions [ClassifyMessage]
  type_builder {
    // Injects new variants into the `@@dynamic` part of the Category enum.
    dynamic Category {
      Question
      Feedback
      TechnicalSupport
    }
  }
  args {
    message "I think the product is great!"
  }
}

Resulting prompt during test execution:

Answer with any of the categories:
Category
----
- Refund
- CancelOrder
- AccountIssue
- Question
- Feedback
- TechnicalSupport

Documentation Preview

• Testing Functions Guide
• BAML Advanced Dynamic Types Guide
• @@dynamic Attribute
• TypeBuilder Reference
• Language Reference