A production-grade command-line tool for counting tokens in files and directories using Anthropic's Claude API.
- Single File or Directory Processing: Count tokens for individual files or entire directory trees
- Token Visualization: Web-based interactive viewer, HTML export, and colored terminal visualization
- Token Analysis: Comprehensive token optimization analysis with recommendations (files only)
- Smart Caching: Local file-based cache to avoid redundant API calls
- Cost Estimation: Automatic cost calculation based on current Claude API pricing
- Gitignore Support: Respects
.gitignorefiles to skip unwanted files - Extension Filtering: Filter files by extension (e.g., only
.goand.mdfiles) - Parallel Processing: Concurrent API requests with configurable concurrency
- Multiple Output Formats: Human-readable tree view or JSON output
- Stdin Support: Pipe content directly from stdin
- File Size Limits: Configurable maximum file size to avoid processing huge files
- Production-Grade: Proper error handling, timeouts, and verbose logging
go install github.com/iota-uz/cc-token@latestThis installs the cc-token binary to your $GOPATH/bin directory.
git clone https://github.com/iota-uz/cc-token.git
cd cc-token
go build -o cc-token .You need an Anthropic API key. Get one from console.anthropic.com.
Set it as an environment variable:
export ANTHROPIC_API_KEY="your-api-key-here"Add this to your ~/.bashrc, ~/.zshrc, or equivalent to persist across sessions.
cc-token uses a subcommand-based interface:
cc-token <command> [flags] [arguments]| Command | Description |
|---|---|
count |
Count tokens in files or directories |
visualize |
Visualize individual tokens in a file |
cache |
Manage the token count cache |
Available for all commands:
| Flag | Short | Type | Default | Description |
|---|---|---|---|---|
--model |
-m |
string | claude-sonnet-4-5 |
Model to use for token counting |
--ext |
-e |
strings | [] |
File extensions to include (e.g., .go,.txt,.md) |
--max-size |
int64 | 2097152 |
Maximum file size in bytes (2MB) | |
--concurrency |
-c |
int | 5 |
Number of concurrent API requests |
--show-cost |
bool | true |
Show estimated API cost | |
--json |
-j |
bool | false |
Output results in JSON format |
--verbose |
-v |
bool | false |
Enable verbose output (shows cache hits) |
--no-cache |
bool | false |
Disable caching | |
--yes |
-y |
bool | false |
Skip confirmation prompts (for automation) |
--plain |
bool | false |
Use plain text output (no ANSI colors) | |
--output |
-o |
string | "" |
Output file path for HTML export |
--no-browser |
bool | false |
Skip auto-opening browser for web visualization | |
--analyze |
bool | false |
Perform token optimization analysis (files only) |
Count tokens in a single file using the default model:
cc-token count document.txtOutput:
document.txt: 1234 tokens
Estimated cost: $0.003702
Count tokens in all files within a directory:
cc-token count .Output:
./
├─ main.go: 5432 tokens
├─ README.md: 987 tokens
└─ go.mod: 45 tokens
--------------------------------------------------
Total: 6464 tokens across 3 files
Estimated cost: $0.019392
Count only Go and Markdown files:
cc-token count --ext .go,.md src/Use Claude Opus 4.1 with the full model name:
cc-token count --model claude-opus-4-1 document.txtOr use convenient aliases for the latest Claude 4.x models:
cc-token count --model haiku document.txt # Uses claude-haiku-4-5 (fastest, cheapest)
cc-token count --model opus document.txt # Uses claude-opus-4-1 (most capable)
cc-token count --model sonnet document.txt # Uses claude-sonnet-4-5 (default, best balance)Get results in JSON format (useful for scripting):
cc-token count --json . > tokens.jsonOutput:
[
{
"path": ".",
"tokens": 6464,
"type": "directory",
"files": 3,
"estimated_cost": 0.019392
}
]Pipe content directly:
cat large-file.txt | cc-token count -
echo "Hello, Claude!" | cc-token count -Process multiple files in one command:
cc-token count file1.txt file2.txt file3.txtSee which files are served from cache:
cc-token count --verbose .Output:
./
├─ main.go: 5432 tokens (cached)
├─ README.md: 987 tokens
└─ go.mod: 45 tokens (cached)
Force fresh API calls:
cc-token count --no-cache document.txtProcess large directories faster:
cc-token count --concurrency 10 ./large-projectIncrease max file size to 50MB:
cc-token count --max-size 52428800 ./dataRemove all cached token counts:
cc-token cache clearcc-token can analyze individual files to identify token optimization opportunities and provide actionable recommendations.
The --analyze flag performs a comprehensive analysis of a single file's token usage patterns, including:
- Efficiency Score: Overall assessment of token usage (0-100 scale)
- Token Density Heatmap: Visual representation of token distribution across the file
- Category Breakdown: Distribution by content type (prose, code, URLs, formatting, whitespace)
- Line-by-Line Insights: Detailed analysis of the 25 most token-expensive lines
- Pattern Detection: Identifies optimization opportunities like repeated URLs, excessive whitespace, and inefficient formatting
- Actionable Recommendations: Prioritized suggestions with estimated token savings
Analyze a single file to identify optimization opportunities:
cc-token count --analyze document.txtConstraints:
- Works with single files only (no directories or multiple files)
- Does not support stdin input (use
-argument) - Automatically skips the cost confirmation prompt for analysis mode
The analysis output includes:
📊 Token Analysis Report
================================================================================
File: document.txt
Total Tokens: 1,234 | Lines: 45 | Efficiency Score: 72/100
================================================================================
📈 Token Distribution (Heatmap)
Low ░░░ Moderate ▒▒▒ High ███
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📋 Content Categories
📝 Prose: 456 tokens (37%)
💻 Code: 234 tokens (19%)
🔗 URLs: 145 tokens (12%)
✨ Formatting: 198 tokens (16%)
⬜ Whitespace: 201 tokens (16%)
🎯 Quick Wins
• Consolidate 12 consecutive empty lines → Save 48 tokens
• URL appears 5 times, use link references → Save 87 tokens
💡 Recommendations
[Priority 1] Remove excessive blank lines (Lines 5-8, 12-15, 23-26)
Before: "...\n\n\n\n..." → After: "...\n..."
Estimated savings: 35 tokens
[Priority 2] Use reference-style links for repeated URLs (Lines 3, 7, 14, 19, 31)
Before: "See [link](https://...)" → After: "See [link][1]"
Estimated savings: 42 tokens
[Priority 3] Remove trailing whitespace from long lines
Before: "content " → After: "content"
Estimated savings: 18 tokens
📊 Summary
Total potential savings: 95 tokens (7.7% reduction)
The analysis respects the --plain flag for plain text output without ANSI colors or emoji:
cc-token count --analyze --plain document.txtThis is useful for automation and piping output to other tools.
Content Optimization:
# Identify redundant content in documentation
cc-token count --analyze README.mdCode Review:
# Find inefficient formatting in code files
cc-token count --analyze main.goLLM Prompt Optimization:
# Reduce token usage for API requests
cc-token count --analyze prompt.txtBatch Analysis:
# Analyze multiple files (run separately)
for file in *.md; do
echo "=== $file ==="
cc-token count --analyze "$file" --plain
doneEfficiency Score Calculation: The efficiency score considers:
- Total tokens vs. total characters
- Whitespace and empty line usage
- Average token-to-character ratio
- Deviation from optimal density
Higher scores indicate more efficient token usage.
Pattern Detection: The analyzer identifies:
- Consecutive empty lines (consolidation opportunity)
- Repeated URLs and phrases (reference-style linking)
- Long lines exceeding typical width (reformatting opportunity)
- Unicode characters (potential high token cost)
- Inefficient markdown formatting
Recommendation Prioritization: Recommendations are sorted by:
- Priority 1 (High Impact): Significant token savings, easy to implement
- Priority 2 (Medium Impact): Moderate savings, may require minimal changes
- Priority 3 (Low Impact): Small savings, consider for comprehensive cleanup
cc-token supports visualizing individual tokens using Claude's streaming API. This feature helps you understand
exactly how text is tokenized.
Displays tokens with colored borders in your terminal:
cc-token visualize basic document.txtOutput shows each token with alternating colors for easy identification:
Token Visualization
================================================================================
Tokens: 42 Characters: 156 Model: claude-sonnet-4-5
Estimated Cost: $0.000126
================================================================================
⎡Hello⎦⎡,⎦ ⎡world⎦⎡!⎦ ⎡This⎦ ⎡is⎦ ⎡a⎦ ⎡test⎦⎡.⎦...
Launches a modern web server with an interactive UI that automatically opens in your browser:
cc-token visualize interactive document.txtFeatures:
- Modern Web UI: Beautiful, responsive interface with dark/light theme
- Two View Modes: Text visualization with colored tokens + detailed table view
- Search & Filter: Real-time token search with match highlighting
- Statistics Panel: Token count, avg/max/min length analysis
- Copy to Clipboard: Click any token to copy it
- Keyboard Shortcuts: Full keyboard navigation support
- Mobile-Friendly: Responsive design works on all devices
Keyboard Shortcuts:
Tab- Switch between text and table view/- Focus search boxEsc- Clear search and deselect?- Show help dialogt- Toggle dark/light theme↑/↓orj/k- Navigate tokensCtrl+C(in terminal) - Stop server
Server Options:
# Launch without auto-opening browser
cc-token visualize interactive --no-browser document.txt
# Server starts on a random available port (8080+)
# Press Ctrl+C to stop the serverExport token visualization to a self-contained HTML file that can be opened in any browser:
cc-token visualize html --output tokens.html document.txtFeatures:
- Self-Contained: Single HTML file with inline CSS and JavaScript
- Portable: Share and view offline without running a server
- Same UI: Identical features to interactive web mode
- No Dependencies: Works in any modern browser
- Automatic Export: No cost confirmation required
Examples:
# Export to HTML file
cc-token visualize html --output report.html README.md
# Export and open in browser (macOS)
cc-token visualize html --output viz.html code.py && open viz.html
# Export and open in browser (Linux)
cc-token visualize html --output viz.html code.py && xdg-open viz.html
# Export with custom model
cc-token visualize html --output tokens.html --model haiku document.txtOutputs structured JSON data for programmatic use and LLM consumption:
cc-token visualize json document.txtOutput format:
{
"content": "Hello, world!",
"model": "claude-sonnet-4-5",
"total_tokens": 5,
"total_chars": 13,
"total_bytes": 13,
"cost": 0.000015,
"tokens": [
{"index": 0, "text": "Hello", "position": 0, "length": 5, "byte_size": 5},
{"index": 1, "text": ",", "position": 5, "length": 1, "byte_size": 1},
{"index": 2, "text": " world", "position": 6, "length": 6, "byte_size": 6},
{"index": 3, "text": "!", "position": 12, "length": 1, "byte_size": 1}
]
}Benefits:
- Machine-readable and parseable by LLMs
- Includes detailed token metadata (position, length, byte size)
- Can be piped to
jqfor filtering and analysis - Scriptable and automatable
- No interactive confirmation (auto-skips cost warning)
Outputs plain text without ANSI colors, perfect for pipes and simple viewing:
cc-token visualize plain document.txtOutput format:
Token Visualization (Plain Text)
================================================================================
Tokens: 5 Characters: 13 Model: claude-sonnet-4-5
Estimated Cost: $0.000015
================================================================================
Tokenized Text:
--------------------------------------------------------------------------------
Hello|,| world|!
--------------------------------------------------------------------------------
Token Details:
--------------------------------------------------------------------------------
[0] "Hello" (pos: 0, len: 5)
[1] "," (pos: 5, len: 1)
[2] " world" (pos: 6, len: 6)
[3] "!" (pos: 12, len: 1)
--------------------------------------------------------------------------------
Total: 5 tokens
Benefits:
- No ANSI color codes (works in all environments)
- Human-readable token boundaries with pipe delimiters
- Detailed token information in structured format
- LLM-friendly plain text format
- No interactive confirmation (auto-skips cost warning)
Before visualization runs, you'll see a cost comparison:
⚠️ Token Visualization Cost Warning
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Token counting (current): $0.000126
Token visualization: $0.000504
Cost difference: $0.000378 (4.0x more expensive)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Proceed with visualization? [Y/n]:
Basic visualization with Haiku (cheapest):
cc-token visualize basic --model haiku code.pyInteractive exploration:
cc-token visualize interactive README.mdJSON output for LLMs and automation:
# Output to stdout
cc-token visualize json document.txt
# Save to file
cc-token visualize json document.txt > tokens.json
# Use with jq for filtering
cc-token visualize json document.txt | jq '.tokens[] | select(.length > 5)'Plain text output for pipes:
cc-token visualize plain document.txtUse global --json flag:
# Override basic mode with JSON output
cc-token visualize basic --json document.txtSkip confirmation prompt (for automation):
# Skip cost warning with --yes flag
cc-token visualize basic --yes document.txt
# Or use shorthand -y
cc-token visualize basic -y document.txtVisualize piped content:
echo "How are tokens split?" | cc-token visualize json -- Single files only: Visualization doesn't work with directories
- No caching: Token boundaries aren't cached (yet)
- Requires API key: Uses full streaming API, not just token counting endpoint
cc-token automatically caches token counts to avoid redundant API calls. The cache is stored in
~/.cc-token/cache.json.
Cache Invalidation: The cache is invalidated when:
- File content changes (detected via SHA-256 hash)
- File modification time changes
Clear Cache:
cc-token cache clearDisable Cache:
cc-token count --no-cache <path>cc-token is designed to be LLM-friendly and easily integrated into automated workflows. The JSON and plain text output modes make it ideal for use with AI agents, scripts, and pipelines.
JSON Output for Structured Data:
# Get structured token data
cc-token visualize json document.txt
# Get token count in JSON format
cc-token count --json document.txtThe JSON output includes:
- Complete token list with positions and lengths
- Byte size information for each token
- Model information and cost estimates
- Fully parseable by LLMs and scripts
Skip Confirmation for Non-Interactive Use:
# Auto-skip cost warnings
cc-token visualize json --yes document.txt
# Or use shorthand
cc-token visualize json -y document.txtJSON and plain text modes automatically skip confirmation prompts.
Example: Token Analysis Script
#!/bin/bash
# Analyze token distribution
cc-token visualize json document.txt | jq '{
total_tokens: .total_tokens,
avg_length: (.tokens | map(.length) | add / length),
long_tokens: (.tokens | map(select(.length > 10)) | length)
}'Example: CI/CD Integration
# Check if file fits in context window
TOKENS=$(cc-token count --json main.go | jq '.[0].tokens')
if [ $TOKENS -gt 100000 ]; then
echo "File too large for context window"
exit 1
fiExample: Batch Processing
# Process multiple files
for file in *.txt; do
cc-token visualize json "$file" > "tokens_${file%.txt}.json"
done# View token boundaries without colors
cat document.txt | cc-token visualize plain -
# Save plain text visualization
cc-token visualize plain document.txt > tokens_analysis.txt- No Interactive Prompts: JSON and plain modes skip confirmation automatically
- Machine-Readable Output: Structured data that's easy to parse
- Exit Codes: Proper error handling with meaningful exit codes
- Stdin Support: Pipe content directly without temporary files
- Concurrent Processing: Built-in support for parallel API requests
Cost estimates are based on Claude API input pricing (per 1M tokens).
| Model | Input (per 1M) | Output (per 1M) | Context Window | Alias |
|---|---|---|---|---|
| Claude Sonnet 4.5 | $3.00 | $15.00 | 200K | sonnet |
| Claude Haiku 4.5 | $1.00 | $5.00 | 200K | haiku |
| Claude Opus 4.1 | $15.00 | $75.00 | 200K | opus |
| Claude Sonnet 4 | $3.00 | $15.00 | 200K | - |
| Model | Input (per 1M) | Output (per 1M) | Context Window |
|---|---|---|---|
| Claude Haiku 3.5 | $0.80 | $4.00 | 200K |
| Claude Sonnet 3.7 | $3.00 | $15.00 | 200K |
| Claude Sonnet 3.5 | $3.00 | $15.00 | 200K |
| Claude Opus 3 | $15.00 | $75.00 | 200K |
| Claude Haiku 3 | $0.25 | $1.25 | 200K |
Notes:
cc-tokenonly counts input tokens (the content you're analyzing)- Output pricing is shown for reference but not calculated by this tool
- Pricing as of 2025-11-01 - check Anthropic's pricing page for latest rates
- Disable cost estimation with
-show-cost=false
When processing directories, cc-token automatically respects .gitignore files in the root directory being scanned.
This means:
node_modules/,.git/, and other ignored directories are skipped- Ignored file patterns are excluded
- Saves API costs by not processing unnecessary files
Important Notes:
- Only the
.gitignorefile in the root directory being scanned is used - Nested
.gitignorefiles in subdirectories are ignored .git/directory is always ignored (even without .gitignore)
All Claude models are supported. The tool accepts multiple naming formats for flexibility.
claude-sonnet-4-5(default) - Best balance of performance and costclaude-haiku-4-5- Fastest, most cost-effectiveclaude-opus-4-1- Most capable, highest costclaude-sonnet-4- Previous generation Sonnet
claude-haiku-3-5- Previous generation Haikuclaude-sonnet-3-7- Previous generation Sonnetclaude-sonnet-3-5- Older Sonnet variantclaude-opus-3- Previous generation Opusclaude-haiku-3- Oldest Haiku version
For convenience, you can use short aliases that automatically resolve to the latest Claude 4.x models:
| Alias | Resolves To | Description |
|---|---|---|
sonnet |
claude-sonnet-4-5 |
Latest Sonnet - best balance of performance and cost |
haiku |
claude-haiku-4-5 |
Latest Haiku - fastest and most cost-effective |
opus |
claude-opus-4-1 |
Latest Opus - most capable model |
Examples:
cc-token -model haiku . # Use Haiku 4.5 (fast and cheap)
cc-token -model opus file.txt # Use Opus 4.1 (most capable)
cc-token -model sonnet dir/ # Use Sonnet 4.5 (default)The tool supports multiple naming conventions for full model names:
- Hyphen format:
claude-sonnet-4-5,claude-haiku-3-5 - Dot format:
claude-sonnet-4.5,claude-haiku-3.5 - Prefix format:
claude-4-sonnet,claude-3-5-haiku
Note: Newer models will work automatically as Anthropic releases them, defaulting to Sonnet 4.5 pricing if not in the pricing table.
-
Use Extension Filtering: Only count relevant files
cc-token count --ext .go,.py src/
-
Increase Concurrency: Process more files in parallel
cc-token count --concurrency 10 . -
Leverage Cache: Run twice - second run will be instant
cc-token count . # First run: ~30s cc-token count . # Second run: ~0.1s
-
Use Gitignore: Ensure
.gitignoreis up to date to skip build artifacts
- Cache is especially useful for repeated checks during development
- Use
--verboseto confirm cache hits
Set your API key:
export ANTHROPIC_API_KEY="sk-ant-..."Your API key is invalid or expired. Check:
- Key is correctly set:
echo $ANTHROPIC_API_KEY - Key has correct permissions at console.anthropic.com
You've hit the rate limit. Solutions:
- Reduce concurrency:
--concurrency 3 - Add delays between runs
- Contact Anthropic to increase limits
Increase max size:
cc-token count --max-size 52428800 large-file.txt # 50MBClear the cache:
cc-token cache clearThe default timeout is 30 seconds. For slow connections, this is currently hardcoded but can be modified in the source.
cc-token is designed as a simple, maintainable CLI tool:
- Single File: All code in
main.go(~1200 lines) - Minimal Dependencies: Core counting uses stdlib only; visualization uses fatih/color, charm/lipgloss, and charm/bubbletea
- Simple Installation:
go installjust works
For rapid UI iteration during development, use Air for automatic rebuilding:
Installation:
# Install Air globally
go install github.com/cosmtrek/air@latestUsage:
# Start Air with the default configuration
air
# Air will:
# 1. Build the project to tmp/main
# 2. Start the visualizer in interactive mode
# 3. Watch for changes to Go, HTML, CSS, and JS files
# 4. Automatically rebuild and restart on file changesDevelopment Workflow:
- Run
airin your terminal - Open
http://localhost:8080(or the port shown) in your browser - Edit HTML, CSS, or JS files in
internal/server/orinternal/visualizer/ - Air rebuilds automatically in ~1-2 seconds
- Refresh your browser to see changes
- Press
Ctrl+Cto stop
What's Watched:
- All
.gofiles (includingcmd/andinternal/packages) - HTML templates (
internal/server/templates/*.html,internal/visualizer/templates/*.html) - CSS files (
internal/server/static/style.css) - JavaScript files (
internal/server/static/app.js)
Why Rebuild for HTML/CSS/JS?
The web server uses go:embed to embed static assets into the binary. Changes to these files require a full rebuild
to be reflected in the running server. Air handles this automatically.
Configuration:
The project includes a .air.toml configuration file with optimized settings. You can customize it by editing .air.toml.
Contributions are welcome! Please:
- Fork the repository
- Create a feature branch
- Make your changes
- Test thoroughly
- Submit a pull request
Testing Changes:
go build .
./cc-token test-file.txtCurrent version: 1.0.0
Check version:
cc-token -versionMIT License - see LICENSE file for details.
Q: Does this count output tokens? A: No, only input tokens. Output tokens depend on Claude's response, which this tool doesn't generate.
Q: Can I count tokens for multiple models at once?
A: No, but you can run the command multiple times with different -model flags. Results will be cached, so only the
first run hits the API.
Q: Does the cache work across different models? A: No, cache is model-specific. Changing models will result in new API calls.
Q: How accurate is the cost estimation? A: Very accurate for input tokens. Prices are current as of November 2025 but may change. Check Anthropic's pricing page for latest rates. Note: Cost estimation uses input token pricing only.
Q: Can I use this in CI/CD pipelines?
A: Yes! Use --json for structured output and check the token count programmatically.
Example:
TOKENS=$(cc-token count --json main.go | jq '.[0].tokens')
if [ $TOKENS -gt 100000 ]; then
echo "File too large for context window"
exit 1
fiQ: What happens if a file can't be read?
A: The error is reported but processing continues for other files. Use --verbose to see all errors.
Q: Can I exclude specific files or directories?
A: Yes, add them to .gitignore in the target directory. Alternatively, use --ext to include only specific
extensions.