Local RAG System

A git-driven, local retrieval-augmented generation (RAG) system for Go codebases. Indexes committed code into semantic chunks for intelligent search using vector embeddings and dependency graphs.

Overview

This tool automatically discovers git repositories, tracks commits, and maintains a searchable index of your codebase. It uses Go's AST parser to extract semantic chunks (functions, methods, types) and stores them with vector embeddings in Valkey for fast similarity search.

Key Principles:

• Git-native: Indexes committed code, uses diffs for efficient updates
• Local-first: Runs entirely on your machine
• Go-optimized: Deep AST understanding for precise code extraction
• Dependency-aware: Tracks and queries import relationships

Prerequisites

• Go 1.24+
• Git (repositories must be git-initialized)
• Valkey or Redis (for vector storage)

Install Valkey:

# macOS
brew install valkey
brew services start valkey

# Or use Docker
docker run -d -p 6379:6379 valkey/valkey:latest

Quick Start

# Build
go build -o rag ./cmd/rag

# Configure (edit scan paths and Valkey settings)
cp rag.example.yaml rag.yaml

# Discover repositories
./rag discover

# Index all discovered repos
./rag index

# Search your code (interactive mode)
./rag query
> find authentication functions

# Search with a single query
./rag query "authentication functions"

# Watch for commits and auto-update
./rag watch

Commands

All commands support global flags:

• --config or -c: Path to config file (default: rag.yaml)
• --log-level or -l: Set logging level (debug|info|warn|error, default: info)
• --version: Print version information

discover

Find all git repositories in configured scan paths:

./rag discover

# With custom config
./rag --config=custom.yaml discover

# With debug logging
./rag --log-level=debug discover

Shows repositories, commits, and tracked file counts without requiring Valkey.

index

One-time indexing of all discovered repositories:

./rag index

# With custom config and debug logging
./rag -c custom.yaml -l debug index

Parses tracked Go files, generates embeddings, stores in Valkey.

query

Interactive or single-query search:

# Interactive mode
./rag query

# Single query
./rag query "user authentication"

# More results
./rag query "user authentication" --top-k=10

# Disable graph expansion
./rag query "find imports" --no-expand

# Combine options
./rag query "http handlers" -t 20 --no-expand

Query flags:

• --top-k or -t: Number of results to return (default: 5)
• --expand / --no-expand: Enable/disable import graph expansion (default: enabled)

watch (Recommended)

Continuously poll for commits and index changes:

./rag watch

# With custom poll interval (set in config file)
./rag --config=rag.yaml watch

• Performs initial indexing
• Polls for new commits (configurable interval)
• Uses git diff to detect changed files
• Incrementally updates index (add/modify/delete/rename)

Configuration

Basic rag.yaml:

# Scan paths - discovers git repositories
scan_paths:
  - path: .
    recursive: true
    exclude_submodules: true

# Git configuration
git:
  poll_interval: 10s      # Duration between commit checks
  include_staged: false   # Index uncommitted staged files
  file_patterns:
    - "*.go"

# Embedder (currently mock only)
embedder:
  provider: mock
  dimension: 384

# Valkey/Redis
valkey:
  address: localhost:6379
  db: 0
  key_prefix: rag

Features

Current (v0.2.0)

Git Integration:

• Auto-discovery of repositories in scan paths
• Commit-based change tracking
• Efficient diff-based updates (only changed files)
• Tracks last_indexed_commit per repository

Code Parsing:

• Go AST parsing for functions, methods, types, constants, variables
• Preserves package context and imports
• Maintains file/line number references

Search:

• Vector similarity search (cosine similarity)
• Graph expansion via import relationships
• Configurable result count and score thresholds

Storage:

• Valkey/Redis vector storage
• Optimized batch operations (reduced round trips)
• Metadata and relationship tracking

Advanced Usage

Multi-workspace indexing:

scan_paths:
  - path: ~/projects
    recursive: true
  - path: ~/work
    recursive: true

Index staged changes:

git:
  include_staged: true  # Index uncommitted files

Git hooks for instant indexing:

# .git/hooks/post-commit
#!/bin/sh
/path/to/rag index &

Branch-specific indexing:

git checkout feature-branch
./rag index

Examples

Basic Workflow

# 1. Discover what repositories will be indexed
./rag discover

# 2. Index everything once
./rag index

# 3. Search interactively
./rag query
> how does authentication work
> find database connection code
> exit

# 4. Single query from command line
./rag query "error handling patterns"

# 5. Keep index synchronized with git commits
./rag watch

Advanced Queries

# Search with more results
./rag query "http handlers" --top-k=20

# Disable import graph expansion (faster, less context)
./rag query "main function" --no-expand

# Use custom config
./rag --config=prod.yaml query "authentication"

# Debug logging for troubleshooting
./rag --log-level=debug query "complex query"

# Combine all options
./rag -c prod.yaml -l debug query "api handlers" -t 15 --no-expand

Using Justfile

# Quick setup
just setup

# Common tasks
just discover
just index
just watch

# Quick search
just search "parser"
just search-simple "parser"  # without graph expansion

# Development
just dev        # clean, format, build, test
just pre-commit # format, lint, test

Roadmap

See ROADMAP.md for planned features and future enhancements.

Architecture

┌──────────────┐
│ Git Discover │──┐
└──────────────┘  │
                  ▼
┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│  Git Poller  │─▶│   Parser     │─▶│  Embedder    │
│  (Commits)   │  │  (Go AST)    │  │   (Mock)     │
└──────────────┘  └──────────────┘  └──────────────┘
                                            │
                                            ▼
┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│  Retriever   │◀─│    Valkey    │◀─│   Storage    │
│  (Search)    │  │ Vector Store │  │   Layer      │
└──────────────┘  └──────────────┘  └──────────────┘

Components:

• internal/git: Repository discovery, polling, diff analysis
• internal/parser: Go AST parsing and chunking
• internal/embedder: Embedding generation (mock, OpenAI planned)
• internal/store: Valkey storage with optimized batch operations
• internal/retriever: Vector search and graph expansion
• cmd/rag: CLI application

Development

# Run tests
go test ./...

# Run with custom config and debug logging
./rag --config=rag.yaml --log-level=debug watch

# Show help for any command
./rag --help
./rag query --help
./rag discover --help

# Check version
./rag --version

# Check Valkey data
valkey-cli KEYS "rag:*"
valkey-cli HGETALL "rag:chunk:someID:meta"

---

For detailed design and technical decisions, see PLAN.md.