Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update dependencies, add activation script, and enhance system prompt #12

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

update dependencies, add activation script, and enhance system prompt #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ba40081
update dependencies, add activation script, and enhance system prompt
Stevo-G-Swag Oct 28, 2024
140c0a7
Merge branch 'main' into main
Stevo-G-Swag Nov 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .vscode/settings.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"inlineChat.finishOnType": true,
"inlineChat.experimental.enableZoneToolbar": true,
"accessibility.voice.keywordActivation": "chatInContext",
"github.copilot.chat.experimental.inlineChatCompletionTrigger.enabled": true,
"github.copilot.chat.experimental.inlineChatHint.enabled": true,
"gitlens.ai.experimental.model": "anthropic:claude-3-5-sonnet-20240620",
"gitlens.ai.experimental.openai.url": "",
"diffEditor.codeLens": true
}
14 changes: 14 additions & 0 deletions activate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/bin/bash
source venv/bin/activate
export PYTHONPATH=$PYTHONPATH:$(pwd)

echo "Virtual environment activated!"
echo ""
echo "To start the application:"
echo "1. Set your API key:"
echo " export ANTHROPIC_API_KEY=your_api_key_here"
echo "2. Set display dimensions (recommended):"
echo " export WIDTH=1280"
echo " export HEIGHT=800"
echo "3. Run the Streamlit app:"
echo " streamlit run streamlit.py"
262 changes: 211 additions & 51 deletions loop.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,69 +31,222 @@ class APIProvider(StrEnum):
ANTHROPIC = "anthropic"
BEDROCK = "bedrock"
VERTEX = "vertex"
BRICKS = "bricks"


PROVIDER_TO_DEFAULT_MODEL_NAME: dict[APIProvider, str] = {
APIProvider.ANTHROPIC: "claude-3-5-sonnet-20241022",
APIProvider.BEDROCK: "anthropic.claude-3-5-sonnet-20241022-v2:0",
APIProvider.VERTEX: "claude-3-5-sonnet-v2@20241022",
APIProvider.BRICKS: "claude-3-5-sonnet-20241022",
}


# This system prompt is optimized for the Docker environment in this repository and
# specific tool combinations enabled.
# We encourage modifying this system prompt to ensure the model has context for the
# environment it is running in, and to provide any additional information that may be
# helpful for the task at hand.
# SYSTEM_PROMPT = f"""<SYSTEM_CAPABILITY>
# * You are utilizing a macOS Sonoma 15.7 environment using {platform.machine()} architecture with internet access.
# * You can install applications using homebrew with your bash tool. Use curl instead of wget.
# * To open Chrome, please just click on the Chrome icon in the Dock or use Spotlight.
# * Using bash tool you can start GUI applications. GUI apps can be launched directly or with `open -a "Application Name"`. GUI apps will appear natively within macOS, but they may take some time to appear. Take a screenshot to confirm it did.
# * When using your bash tool with commands that are expected to output very large quantities of text, redirect into a tmp file and use str_replace_editor or `grep -n -B <lines before> -A <lines after> <query> <filename>` to confirm output.
# * When viewing a page it can be helpful to zoom out so that you can see everything on the page. In Chrome, use Command + "-" to zoom out or Command + "+" to zoom in.
# * When using your computer function calls, they take a while to run and send back to you. Where possible/feasible, try to chain multiple of these calls all into one function calls request.
# * The current date is {datetime.today().strftime('%A, %B %-d, %Y')}.
# </SYSTEM_CAPABILITY>
# <IMPORTANT>
# * When using Chrome, if any first-time setup dialogs appear, IGNORE THEM. Instead, click directly in the address bar and enter the appropriate search term or URL there.
# * If the item you are looking at is a pdf, if after taking a single screenshot of the pdf it seems that you want to read the entire document instead of trying to continue to read the pdf from your screenshots + navigation, determine the URL, use curl to download the pdf, install and use pdftotext (available via homebrew) to convert it to a text file, and then read that text file directly with your StrReplaceEditTool.
# </IMPORTANT>"""
SYSTEM_PROMPT = f"""<SYSTEM_CAPABILITY>
* You are utilizing a macOS Sonoma 15.7 environment using {platform.machine()} architecture with command line internet access.
* Package management:
- Use homebrew for package installation
- Use curl for HTTP requests
- Use npm/yarn for Node.js packages
- Use pip for Python packages

* Browser automation available via Playwright:
- Supports Chrome, Firefox, and WebKit
- Can handle JavaScript-heavy applications
- Capable of screenshots, navigation, and interaction
- Handles dynamic content loading

* System automation:
- cliclick for simulating mouse/keyboard input
- osascript for AppleScript commands
- launchctl for managing services
- defaults for reading/writing system preferences

* Development tools:
- Standard Unix/Linux command line utilities
- Git for version control
- Docker for containerization
- Common build tools (make, cmake, etc.)

* Output handling:
- For large output, redirect to tmp files: command > /tmp/output.txt
- Use grep with context: grep -n -B <before> -A <after> <query> <filename>
- Stream processing with awk, sed, and other text utilities

* Note: Command line function calls may have latency. Chain multiple operations into single requests where feasible.

* The current date is {datetime.today().strftime('%A, %B %-d, %Y')}.
</SYSTEM_CAPABILITY>"""

SYSTEM_PROMPT = f"""<SYSTEM_DEFINITION>
You are an advanced AI assistant operating within a macOS Sequoia Version 15.1 (24B82) environment with comprehensive access to system resources and applications. Your purpose is to provide precise, efficient assistance while leveraging available tools optimally.

<SYSTEM_SPECIFICATIONS>
1. Hardware Configuration:
- Model: MacBook Pro (15-inch, 2018)
- Processor: 2.6 GHz 6-Core Intel Core i7
- Memory: 16 GB 2400 MHz DDR4
- Graphics: Intel UHD Graphics 630 1536 MB
- Display: 15.4-inch Retina (2880 × 1800)
- Architecture: {platform.machine()}
- Internet: Active connection available
- Time Zone: System configured
- Current Date: {datetime.today().strftime('%A, %B %-d, %Y')}

<APPLICATION_ECOSYSTEM>
1. Development Environment:
A. Code Editors & IDEs:
- Visual Studio Code & VS Code Insiders
- Xcode Beta
- Sublime Text
- Adobe Dreamweaver 2021

B. Version Control & Collaboration:
- GitHub Desktop
- Git (command line)
- CodeForces Web Tool

C. Container & Virtual Environments:
- Docker.app
- Docker CLI tools

D. Development Tools:
- Terminal
- Command Line Tools
- Developer.app

2. Professional Suites:
A. Microsoft Office:
- Word
- Excel
- PowerPoint
- OneNote
- Outlook

B. Adobe Creative Cloud:
- Creative Cloud Manager
- Dreamweaver 2021
- Premiere Pro (Beta)
- Adobe UXP Developer Tools

3. Web Browsers & Tools:
A. Primary Browsers:
- Safari & Safari Technology Preview
- Google Chrome Beta
- Firefox
- Microsoft Edge Dev
- Chromium

B. Specialized Browsers:
- Tor Browser (Standard & Alpha)

C. Browser Extensions:
- Grammarly for Safari
- Microsoft Bi for Safari

4. AI & Machine Learning Tools:
- NVIDIA AI Workbench
- Code AI
- AI on Device (MacOS)
- 16x Prompt.app

5. System Utilities:
A. File Management:
- Finder
- Preview
- The Unarchiver
- Unzip - RAR

B. System Tools:
- System Settings
- Automator
- Mission Control
- Time Machine
- Activity Monitor

C. Text Processing:
- TextEdit
- Notes

D. Security:
- Passwords.app
- G Authenticator
- BitPay
- Wasabi Wallet

6. Communication & Collaboration:
- Messages
- Mail
- FaceTime
- Discord
- Zoom
- Messenger
- TextNow

7. Media & Entertainment:
- QuickTime Player
- Photos
- Music
- TV
- Podcasts
- Photo Booth

8. Productivity & Organization:
- Calendar
- Reminders
- Stickies
- Clock
- Calculator
- Weather
- Maps

<OPERATIONAL_CAPABILITIES>
1. File System Access:
- Read/Write operations in user directories
- Application data access
- Temporary file creation
- Archive handling

2. Network Operations:
- HTTP/HTTPS requests
- API interactions
- Download capabilities
- Network diagnostics

3. Automation Framework:
A. System Automation:
- Shortcuts.app
- Automator workflows
- AppleScript execution
- Shell scripting

B. Development Automation:
- Build tools
- Package managers
- Deployment scripts

4. Security Protocols:
- Secure file operations
- Credential management
- Encryption capabilities
- Privacy controls

<PERFORMANCE_GUIDELINES>
1. Resource Management:
- Monitor system resources
- Optimize heavy operations
- Cache management
- Background process awareness

2. Error Handling:
- Graceful failure recovery
- User feedback
- Logging capabilities
- Debug information

3. Operation Chaining:
- Minimize command calls
- Batch operations
- Efficient workflows
- Resource pooling

<INTERACTION_PROTOCOL>
For each user interaction, I will:
1. Analyze request requirements
2. Identify optimal tools/applications
3. Validate resource availability
4. Plan execution strategy
5. Provide clear documentation
6. Monitor execution
7. Handle errors gracefully
8. Confirm successful completion

<RESPONSE_FORMAT>
Each response will include:
1. <thinking> tags for analysis
2. Task acknowledgment
3. Resource identification
4. Step-by-step execution plan
5. Clear documentation
6. Error handling procedures
7. Success confirmation

<LIMITATIONS_AWARENESS>
- Respect system permissions
- Handle resource constraints
- Consider operation timing
- Maintain security protocols
- Preserve user privacy
- Account for network latency"""
Comment on lines +50 to +249
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codebase verification

⚠️ Potential issue

Remove sensitive system specifications and hardware details from system prompt

The verification confirms that the system prompt in loop.py contains detailed hardware specifications and system information that could pose security risks. This information is hardcoded in the system prompt and includes:

  • Specific MacBook Pro model and year
  • Detailed processor specifications
  • Memory configuration
  • Graphics card details
  • Display specifications

Recommendations:

  • Remove or generalize hardware specifications
  • Remove specific version numbers and model details
  • Use generic system capabilities instead of exact specifications
  • Consider moving any required system information to a configuration file with proper security controls
🔗 Analysis chain

Security Concern: System prompt exposes sensitive information

The system prompt includes detailed system specifications and installed applications that could potentially be exploited:

  1. Hardware specifications expose system capabilities
  2. Installed applications reveal potential attack vectors
  3. Financial applications (BitPay, Wasabi Wallet) indicate presence of sensitive data

Consider:

  • Removing or generalizing hardware specifications
  • Limiting application listing to essential tools
  • Removing references to financial/security applications

Let's verify if this information is exposed in public repositories:

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Search for sensitive system information in public files
rg -i "wasabi|bitpay|authenticator" --type md
rg -i "macbook pro.*2018|intel core i7" --type md

Length of output: 99

Script:

#!/bin/bash
# Let's search for system prompt definitions and sensitive information in all file types
rg -i "SYSTEM_PROMPT.*=.*" -A 5
rg -i "system.*specifications|hardware.*configuration" -A 10

Length of output: 1888


async def sampling_loop(
*,
Expand All @@ -106,7 +259,7 @@ async def sampling_loop(
api_response_callback: Callable[[APIResponse[BetaMessage]], None],
api_key: str,
only_n_most_recent_images: int | None = None,
max_tokens: int = 4096,
max_tokens: int = 8192,
):
"""
Agentic sampling loop for the assistant/tool interaction of computer use.
Expand All @@ -130,6 +283,13 @@ async def sampling_loop(
client = AnthropicVertex()
elif provider == APIProvider.BEDROCK:
client = AnthropicBedrock()
elif provider == APIProvider.BRICKS:
client = Anthropic(
api_key=api_key,
base_url="https://api.trybricks.ai/api/providers/anthropic",
)
else:
raise ValueError(f"Unsupported provider: {provider}")

# Call the API
# we use raw_response to provide debug information to streamlit. Your
Expand Down
Loading