CodeSmriti

1. Problem Statement

Engineering teams accumulate solutions across dozens of repositories over years of development. Without institutional memory, organizations face compounding inefficiencies:

Traditional code search (grep, GitHub search) operates on keywords, not semantics. Questions like "How did we solve rate limiting?" or "What's our authentication pattern?" require understanding intent, not matching strings.

2. Architecture

Components

• Ingestion Pipeline: Bottom-up processor: tree-sitter parsing → LLM chunking → symbol summarization → file aggregation → module aggregation → repo summary
• Vector Database: Couchbase with FTS vector search index (768 dimensions)
• Tool Layer: Shared implementations for all RAG operations (search, explore, file retrieval)
• API Server: FastAPI providing RAG endpoints with hierarchical search (symbol/file/module/repo levels)
• MCP Server: Model Context Protocol for Claude Code—direct tool access without intermediate LLM
• PydanticAI Agent: LLM-driven mode for local models (LMStudio, Ollama)—LLM orchestrates tool calls

Key Technologies

3. Chunking Strategy

Research confirms that naive chunking methods struggle with code: "Naive chunking methods struggle with accurately delineating meaningful segments of code, leading to issues with boundary definition and the inclusion of irrelevant or incomplete information" [5].

CodeSmriti employs a hybrid approach: AST-aware chunking via tree-sitter as the primary method, with LLM-assisted semantic chunking as a fallback for files that resist static analysis. This has been shown to improve Recall@5 by 4.3 points on code retrieval benchmarks [6].

Core Principle: Bottom-Up Aggregation

Summaries are built from the ground up: individual symbols (functions, classes) are summarized first, then aggregated into file-level summaries, then into folder-based module summaries, and finally into a repository overview. This ensures every level has meaningful, accurate context rather than relying on top-down inference.

Hierarchical Document Types

Hybrid Chunking: AST + LLM

Tree-sitter provides AST-level symbol extraction for most languages. However, some files resist static analysis—Svelte components with complex template logic, SQL embedded in Python strings, or domain-specific patterns.

For these "underchunked" files (high line count but few detected symbols), CodeSmriti invokes an LLM semantic chunker to identify logical boundaries:

Primary (tree-sitter):     function_definition, class_definition, method_definition
Fallback (LLM chunker):    SQL queries, business logic blocks, configuration sections,
                           template event handlers, data transformations

The LLM chunker identifies semantic units like user_subscription_data_fetch or zone_calculation_algorithm that tree-sitter cannot detect, ensuring comprehensive coverage even for unconventional code patterns.

Junk Filtering

Skip files that add noise without value:

node_modules/    package-lock.json    *.min.js
dist/            yarn.lock            *.min.css
__pycache__/     Cargo.lock           *.map
.git/            poetry.lock          *generated*

4. Document Schema

Hierarchical Document Structure

Quality Tracking

Each document carries quality metadata indicating its enrichment level:

A circuit breaker pattern prevents cascading failures: if LLM calls fail repeatedly, the system gracefully degrades to basic enrichment without blocking ingestion.

Content-Based Deduplication

Document IDs are SHA256 hashes of content keys (repo, path, symbol name, commit). Re-indexing the same commit produces identical IDs, enabling efficient upserts without duplicate detection logic.

Embedding Strategy

Embeddings capture both semantic meaning (from LLM summary) and code patterns (from actual code at index time). For symbols, the embedding combines the summary with a code snippet. This answers both "what does this do?" and "how is it implemented?"

5. Performance

V4 Production Stats (November 2025)

Full ingestion of 101 repositories on Mac M3 Ultra with LM Studio (qwen3-30b-a3b-2507):

Throughput

Resumable Ingestion

The --skip-existing flag enables resumable batch ingestion. Repos with existing V4 documents are skipped, allowing recovery from failures without re-processing completed work.

6. Case Studies

V4 evaluation suite: 47 questions across 9 categories, tested against 32 proprietary repositories. The evaluation covers symbol lookups, pattern discovery, documentation retrieval, and cross-repo queries.

Evaluation Results (v0.4.0)

Overall: 47/47 passing (100%), average quality 78%

Example Output: Symbol Query

Query: "job_counter decorator implementation"

# `job_counter` Decorator Implementation

The `job_counter` decorator is implemented in `common/consumer_decorators.py`
and tracks task execution metrics using Redis.

## Key Features
- Tracks success, failure, and pending task counts
- Uses Redis for state management
- Supports both synchronous and asynchronous functions

## Implementation
```python
def job_counter(func: Callable) -> Callable:
    @wraps(func)
    async def async_wrapper(*args: Any, **kwargs: Any) -> Any:
        key_success, key_failed_list, key_pending_list, job_string = _make_keys(func, *args)
        try:
            redis_connection.lpush(key_pending_list, job_string)
            result = await func(*args, **kwargs)
            redis_connection.incr(key_success)
            return result
        except Exception as e:
            redis_connection.lpush(key_failed_list, f"Error: {str(e)}")
            redis_connection.ltrim(key_failed_list, 0, 50)
            raise
        finally:
            redis_connection.lpop(key_pending_list)
```

Example Output: Documentation Query

Query: "L1 Python best practices"

# L1 Python Best Practices

## Naming Conventions
- Use `snake_case` for function and variable names
- Use `PascalCase` for class names
- Constants should be `UPPER_CASE`

## Error Handling
- Use specific exception types rather than bare `except:`
- Always include error messages for debugging
- Use context managers for resource management

## Type Hints
- Always use type hints for function parameters and return values
- Use `typing` module for complex types

```python
def process_user_data(
    user_list: List[Dict[str, str]],
    filter_criteria: Optional[Dict[str, str]] = None
) -> Tuple[Dict[str, List], int]:
    ...
```

Source: `docs/source/developer_guide/audit_standards/L1_python_best_practices.rst`

Search Level Selection

The LLM automatically selects the appropriate search level based on query intent:

• symbol — "Find the job_counter decorator" → specific function/class
• file — "How do we handle authentication?" → implementation files
• module — "What's in the permissions folder?" → directory overview
• doc — "What are our coding standards?" → RST/MD documentation
• repo — "What repos handle geospatial data?" → cross-repo discovery

Strengths

• Excellent at finding code patterns across repositories
• Strong at comparing implementations (decorators, mixins)
• Good at synthesizing answers from scattered sources
• Documentation queries now use dedicated doc level search

7. Quick Start

# Clone repository
git clone https://github.com/kbhalerao/code-smriti
cd code-smriti

# Configure environment
cp .env.example .env
# Edit .env with your GitHub token and Couchbase credentials

# Start services
docker-compose up -d

# Verify
curl http://localhost/health

MCP Integration (Claude Desktop / Claude Code)

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "code-smriti": {
      "command": "uv",
      "args": ["run", "--with", "mcp", "--with", "httpx",
               "path/to/code-smriti/services/mcp-server/rag_mcp_server.py"],
      "env": {
        "CODESMRITI_API_URL": "http://localhost",
        "CODESMRITI_USERNAME": "your-username",
        "CODESMRITI_PASSWORD": "your-password"
      }
    }
  }
}

Available Tools

Two Usage Modes