Getting Started¶

This guide walks you through installing and configuring MAGIC for your development workflow.

Prerequisites¶

Rust toolchain (for building from source)
zsh or bash shell
~100MB disk space for the database

Installation¶

From Source¶

# Clone the repository
git clone https://github.com/yourorg/magic
cd magic

# Build and install shq
cargo install --path shq

# Verify installation
shq --version

Initialize BIRD¶

Before using shq, initialize the BIRD database:

shq init

Storage Mode Options:

# Default: Parquet mode (multi-writer safe, for concurrent shells)
shq init

# DuckDB mode (single-writer, simpler, no compaction needed)
shq init --mode duckdb

This creates the directory structure at ~/.local/share/bird/:

~/.local/share/bird/
├── db/
│   ├── bird.duckdb           # Main DuckDB database
│   └── pending/              # In-flight invocation markers (crash recovery)
├── data/
│   ├── recent/               # Hot tier (0-14 days)
│   │   ├── invocations/
│   │   │   └── status=<status>/  # pending, completed, orphaned
│   │   ├── outputs/
│   │   └── sessions/
│   └── archive/              # Cold tier (>14 days)
├── blobs/
│   └── content/              # Content-addressed storage
├── extensions/               # DuckDB extensions
├── sql/                      # Custom SQL files
└── config.toml               # Configuration

Shell Integration¶

Zsh¶

Add to your ~/.zshrc:

eval "$(shq hook init)"

Bash¶

Add to your ~/.bashrc:

eval "$(shq hook init --shell bash)"

Reload Your Shell¶

# Zsh
source ~/.zshrc

# Bash
source ~/.bashrc

Verify Installation¶

After reloading your shell, run a few commands:

# Run some commands
echo "Hello, MAGIC!"
ls -la
make --version

# Check that they were captured
shq i              # or: shq invocations

# View output from last command
shq o              # or: shq output

Quick Reference¶

Run shq ? for a quick reference card showing all commands and query syntax.

Basic Usage¶

View Command History¶

shq i              # Last 20 commands (default)
shq i 50           # Last 50 commands
shq i %exit<>0~10  # Last 10 failed commands
shq i %/cargo/~20  # Last 20 cargo commands

View Command Output¶

shq o              # Output from last command
shq o 3            # Output from 3rd-last command
shq o -E 1         # Only stderr
shq o -A 1         # Both streams combined
shq o %/make/~1    # Output of last make command

Show Invocation Details¶

shq I              # Details about last command (alias: info)
shq I 3            # Details about 3rd-last command
shq I -f json 1    # As JSON

Re-run Previous Commands¶

shq R              # Re-run last command (alias: rerun)
shq R 3            # Re-run 3rd-last command
shq R %/make/~1    # Re-run last make command
shq R -n %/test/~1 # Dry-run: show what would run

Run with Capture¶

shq r make test           # Run and capture (alias: run)
shq r -c "echo hello"     # Run shell command

SQL Queries¶

shq q "SELECT cmd, exit_code FROM invocations LIMIT 10"
shq q "SELECT * FROM failed_invocations LIMIT 5"
shq q "SELECT * FROM invocations_today"

Statistics¶

# Show database statistics
shq stats

# JSON output (for scripting)
shq stats --json

Events (Parsed Errors/Warnings)¶

BIRD automatically parses build output for structured events:

shq e              # Show events from recent commands
shq e %/cargo/~5   # Events from last 5 cargo commands
shq e -s error     # Only errors (filter by severity)
shq e --count      # Count events by severity

Extract Events¶

Manually extract/re-extract events from outputs:

shq extract-events         # Extract from last command
shq extract-events --all   # Backfill all commands without events
shq extract-events -f gcc  # Force specific format

Format Hints¶

Configure format detection for build tools:

shq format-hints list              # Show configured hints
shq format-hints add "make*" gcc   # Use gcc parser for make commands
shq format-hints remove "make*"    # Remove a hint
shq format-hints set-default cargo # Set default format for unknown commands

Retrospective Buffer¶

The buffer provides "retroactive capture" - capture commands you didn't explicitly save:

# Enable buffer mode (captures all commands to rotating buffer)
shq buffer enable --on

# Check status
shq buffer status

# List buffered commands
shq buffer list
shq buffer list -n 20       # Show last 20 entries

# View buffer entry output
shq buffer show ~1          # Most recent
shq buffer show ~3          # 3rd most recent

# Promote buffer entry to permanent storage
shq save ~1                 # Promote most recent
shq save ~3                 # Promote 3rd most recent

# Clear buffer
shq buffer clear

# Disable buffer mode
shq buffer enable --off

Buffer mode is security-focused: - Disabled by default - Extensive exclude patterns for sensitive commands (passwords, tokens, SSH, etc.) - Secure file permissions (0600) - Automatic rotation based on age/size limits

Data Lifecycle¶

Archive Old Data¶

Move data older than 14 days to archive tier:

# Archive with default settings (14 days)
shq archive

# Archive data older than 30 days
shq archive --days 30

# Preview what would be archived
shq archive --dry-run

Compact Files¶

Merge small parquet files for better performance:

# Compact all sessions
shq compact

# Compact specific session
shq compact -s shell-12345

# Preview what would be compacted
shq compact --dry-run

Automatic Compaction

Shell hooks automatically run background compaction after each command, so manual compaction is rarely needed.

Clean and Recover¶

Recover orphaned invocations (from crashes, SIGKILL, etc.) and clean up stale data:

# Recover orphaned invocations
shq clean

# Mark as orphaned after 12 hours (default: 24)
shq clean --max-age 12

# Also prune old archive data
shq clean --prune

# Prune data older than 90 days
shq clean --prune --older-than 90d

# Preview what would be cleaned
shq clean --dry-run

What is an orphaned invocation?

When a command crashes, is killed with SIGKILL, or the system reboots unexpectedly, BIRD may have pending invocation records that were never completed. The shq clean command detects these by checking if the process (PID) is still alive and marks them as "orphaned" so they appear in your history with the correct status.

Remote Storage¶

Sync your command history across machines using remote DuckDB databases:

Configure Remotes¶

# Add a remote (S3, local file, MotherDuck, or PostgreSQL)
shq remote add team --type s3 --uri s3://bucket/bird.duckdb
shq remote add backup --type file --uri /mnt/backup/bird.duckdb --read-only

# List configured remotes
shq remote list

# Test remote connectivity
shq remote test team

Push/Pull Data¶

# Push local data to remote
shq push --remote team

# Push only recent data
shq push --remote team --since 7d

# Preview what would be pushed
shq push --remote team --dry-run

# Pull data from remote
shq pull --remote team

# Pull specific client's data
shq pull --remote team --client laptop@user

Remote Types¶

Type	URI Format	Description
`file`	`/path/to/bird.duckdb`	Local or network file
`s3`	`s3://bucket/path/bird.duckdb`	S3-compatible storage
`motherduck`	`md:database_name`	MotherDuck cloud
`postgres`	`postgres:dbname=...`	PostgreSQL database

Environment Variables¶

Variable	Default	Description
`BIRD_ROOT`	`~/.local/share/bird`	Base directory for BIRD data

Privacy & Tracking Control¶

Multiple ways to exclude commands from capture:

Caller Opt-Out (Shell-Level)¶

# Leading space - not captured
 echo "secret password"

# Leading backslash - not captured
\curl -H "Authorization: $TOKEN" api.example.com

Environment Variables¶

# Disable all capture
export SHQ_DISABLED=1

# Exclude by pattern (colon-separated)
export SHQ_EXCLUDE="*password*:*secret*:*token*"

Command Opt-Out (OSC Escape)¶

Programs can signal they don't want to be tracked:

# In your script:
printf '\e]shq;nosave\a'  # Disable capture for this command

Auto-Excluded Commands¶

Query commands (shq i, shq o, shq sql, etc.) are automatically excluded to prevent recursion.

Troubleshooting¶

Commands Not Being Captured¶

Check that hooks are installed:
```
type __shq_precmd
```
Verify BIRD is initialized:
```
ls ~/.local/share/bird/db/bird.duckdb
```
Check the error log:
```
cat ~/.local/share/bird/errors.log
```

Slow Shell Startup¶

The hooks are designed to be lightweight. If you experience slowness:

Check if BIRD_ROOT is on a slow filesystem
Ensure the database isn't corrupted: shq sql "SELECT 1"

Permission Errors¶

Ensure you have write access to BIRD_ROOT:

ls -la ~/.local/share/bird/

Next Steps¶

Shell Integration - Deep dive into how hooks work
SQL Queries - Advanced query examples
BIRD Specification - Technical architecture details