TraceForge ⚡

TraceForge is an agentic recording compiler, headless orchestration engine, and code synthesizer implemented in Rust. It captures browser interactions (DOM mutations, input traces, viewport screencasts, and network exchanges) across Chromium and Firefox, runs them through differential analysis & semantic refinement, and compiles them into:

1. Typed Zero-Browser API SDKs (Pure async Rust clients with automated cookie/token threading and JA4 TLS mimicry)
2. OpenAPI 3.1 Specifications (Private API reverse-engineering with schema inference)
3. Headless Browser Scripts (Resilient Playwright TypeScript/JavaScript and Chromiumoxide Rust automation with selector fallback chains)
4. Embedded Model Context Protocol (MCP) Server (Exposing recording, code synthesis, challenge solving, and live screencast/HAR streaming directly to AI agents)

---

Workspace Architecture

The repository is structured as a modular Cargo workspace across 10 crates:

traceforge/
├── crates/
│   ├── traceforge-core/         # Core telemetry models (UserAction, NetworkEvent, ViewportFrame), DriverEngine trait, AF-IR
│   ├── traceforge-transport/    # Mode 1 runtime: NoBrowserEngine, JA4 TLS mimicry, ProxyRouter with sticky sessions, HttpTaskRunner
│   ├── traceforge-browser/      # Mode 2 & 3: Chromium CDP driver (chromiumoxide) & Firefox WebDriver BiDi client (tokio-tungstenite)
│   ├── traceforge-stealth/      # Anti-detection scripts: navigator.webdriver evasion, WebGL spoofing, 2D canvas noise
│   ├── traceforge-solver/       # Pluggable challenge solver trait, TurnstileDetector, TurnstileSolver
│   ├── traceforge-recorder/     # Append-only .tflog storage, temporal-spatial correlator (ActionFlowIR DAG builder)
│   ├── traceforge-codegen/      # Differential token induction, OpenAPI 3.1 generator, Rust SDK generator, Playwright generator, LLM layer
│   ├── traceforge-mcp/          # Model Context Protocol JSON-RPC 2.0 server (tools & streaming resources over stdio/SSE)
│   ├── traceforge-cli/          # Unified CLI suite and browser WebExtension native messaging IPC bridge
│   ├── traceforge-extension/    # Browser WebExtension Rust WASM core (DOM selector induction, telemetry observers, popup dashboard)
│   └── traceforge-cloud/        # Axum REST & WebSocket SaaS Gateway (Scraper API, workflow replay, & extension live-sync)
├── extension/                   # Manifest V3 browser extension (loaders, popup UI, dark-mode CSS, icons, LiveSyncManager)
├── scripts/                     # Automated build & packaging pipelines (build-extension.sh)
└── docs/                        # Technical documentation, SDD plans, and guides
    ├── CLOUD_API.md             # TraceForge Cloud REST & WebSocket API specification
    ├── EXTENSION.md             # In-depth WebExtension UI, metrics, cookie editor & telemetry guide
    └── WORKFLOW_REPLICATION.md  # Step-by-step Twitter/X action parameterization & replay tutorial

---

Prerequisites

• Rust: 1.80+ (2021 edition)
• Chromium / Chrome: (optional, for Chromium CDP driver & headed recording)
• Firefox: (optional, for Firefox WebDriver BiDi driver)

---

Quick Setup & Build

Clone the repository and build the workspace:

# Build all crates and binaries
cargo build --release

# Run all 127 automated integration & unit tests
cargo test --workspace

# Check clippy linter
cargo clippy --workspace --all-targets

The compiled binary will be located at target/release/traceforge-cli.

---

☁️ TraceForge Cloud Gateway (traceforge-cloud)

traceforge-cloud is the commercial SaaS execution gateway that turns TraceForge into a Stealth Web Scraping API & Hosted Workflow Replay Platform.

Starting the Cloud Server

# Run the Axum REST & WebSocket gateway server on http://localhost:8080
cargo run -p traceforge-cloud

Core API Endpoints

1. Instant Stealth Web Scraping API (POST /v1/scrape)

Send a POST request with X-TraceForge-Key: tf_live_...:

curl -X POST http://localhost:8080/v1/scrape \
  -H "X-TraceForge-Key: tf_live_12345" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/data",
    "render_js": true,
    "bypass_cloudflare": true
  }'

2. Hosted Workflow Replay API (POST /v1/run/:workflow_id)

Execute a reverse-engineered workflow trace on demand:

curl -X POST http://localhost:8080/v1/run/wf_live_987654 \
  -H "X-TraceForge-Key: tf_live_12345" \
  -H "Content-Type: application/json" \
  -d '{
    "variables": { "username": "elonmusk" }
  }'

3. Real-Time Extension Live-Sync (WS /v1/workflows/live-sync)

Stream browser actions from the Manifest V3 extension directly into TraceForge Cloud in real-time.

For detailed request/response schemas, see docs/CLOUD_API.md.

---

Usage Guide

1. Recording a Workflow (record)

Record user actions, network flows, and screencast frames into durable .tflog traces.

# Record in Mode 1 (Zero-browser fast HTTP recorder)
traceforge-cli record --url https://example.com/login --engine no-browser --output session.tflog

# Record in Mode 2 (Headless Chromium with CDP and screencasting)
traceforge-cli record --url https://example.com/login --engine chromium --output session.tflog --duration-secs 10

# Record in Mode 3 (Firefox WebDriver BiDi)
traceforge-cli record --url https://example.com/login --engine firefox --output session.tflog

---

2. Synthesizing Code & Specs (compile)

Compile a recorded .tflog trace into client code or API specifications:

A. Generate an OpenAPI 3.1 Specification

traceforge-cli compile --trace session.tflog --target openapi --output api-spec/
# Emits api-spec/openapi.json with inferred request/response JSON schemas, query params, and headers

B. Generate a Typed Async Rust Client SDK

traceforge-cli compile --trace session.tflog --target rust-sdk --output src/
# Emits src/client.rs with:
# - Strongly typed request/response structs
# - Automatic cookie jar persistence
# - Token origin discovery (CSRF tokens automatically extracted from cookies or HTML and threaded into mutating requests)

C. Generate Resilient Playwright Browser Scripts

# Playwright TypeScript
traceforge-cli compile --trace session.tflog --target playwright-ts --output scripts/
# Emits scripts/script.ts with locator fallback chains (.or()) and load state waits

# Playwright JavaScript
traceforge-cli compile --trace session.tflog --target playwright-js --output scripts/

# Chromiumoxide Rust automation script
traceforge-cli compile --trace session.tflog --target chromiumoxide-rust --output scripts/

---

3. Executing Automated Workflows (run)

Execute automated workflows directly from workflow configurations.

Mode 1: Zero-Browser HTTP Task Runner

Create a workflow config (workflow.json):

{
  "steps": [
    {
      "url": "https://example.com/login",
      "method": "GET"
    },
    {
      "url": "https://example.com/api/v1/auth/login",
      "method": "POST",
      "headers": {
        "Content-Type": "application/json",
        "X-CSRF-Token": "{{csrf_token}}"
      },
      "post_data": "{\"username\": \"alice\", \"password\": \"secret\"}"
    }
  ]
}

Run via CLI (with optional runtime parameter overrides):

# Basic run
traceforge-cli run --target no-browser --config workflow.json

# Parameterized run with inline token overrides
traceforge-cli run --target no-browser --config workflow.json --param csrf_token="custom_token_123"

# Parameterized run with parameters file
traceforge-cli run --target no-browser --config workflow.json --params-file secrets.json

Features:

• Automated cookie jar synchronization across steps.
• Automatic extraction and injection of dynamic token variables ({{csrf_token}}).
• Command-line parameter overrides (--param key=value and --params-file <path>). See WORKFLOW_REPLICATION.md for Twitter/X like and tweet replication examples.
• Built-in Cloudflare Turnstile challenge detection and reporting.
• Sticky proxy pool routing via ProxyRouter.

---

4. Running as a Model Context Protocol (MCP) Server (serve-mcp)

TraceForge includes a native JSON-RPC 2.0 MCP server that allows AI agents (such as Claude Desktop, Cursor, or custom LLM orchestrators) to control browsers, extract APIs, and stream live screencasts.

Start the server over standard I/O:

traceforge-cli serve-mcp --transport stdio

Configuring in Claude Desktop / Cursor

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "traceforge": {
      "command": "/path/to/traceforge/target/release/traceforge-cli",
      "args": ["serve-mcp", "--transport", "stdio"]
    }
  }
}

Tools Exposed to AI Agents:

• record_action_flow: Starts recording a browser session given a target URL, engine (chromium, firefox, no-browser), and viewport.
• compile_recorded_task: Compiles a recorded session ID into OpenAPI 3.1, typed Rust client SDK, or Playwright script.
• execute_no_browser: Runs parameterized HTTP steps in zero-browser mode.
• solve_active_challenge: Intercepts and solves Cloudflare Turnstile / CAPTCHA challenges.

Streaming Resources Exposed to AI Agents:

• browser://screencast: Real-time JPEG screencast frames from the active browser viewport.
• browser://live-har: Real-time correlated HTTP transactions formatted as structured JSON.
• codegen://openapi.json: Live synthesized OpenAPI 3.1.0 document.

---

5. WebExtension Native Messaging Host (native-host)

TraceForge can bridge directly with Chrome or Firefox browser extensions via 32-bit length-prefixed native messaging:

traceforge-cli native-host
# or run with standard browser extension flags:
traceforge-cli --native-messaging-host chrome-extension://<EXTENSION_ID>/

---

6. TraceForge WebExtension (extension/, traceforge-extension)

The TraceForge Recorder is a Manifest V3 browser extension powered by a compiled Rust WebAssembly core (traceforge-extension). It performs real-time user interaction capture, resilient selector induction fallback chains, network telemetry interception via chrome.webRequest, and viewport screencasting. Full documentation is available in EXTENSION.md.

Key Features:

• DevTools "TraceForge Blueprint" Panel: Dedicated Chrome DevTools panel providing a live 4-column reactive architecture schematic (L1: DOM Triggers ➔ L2: Protocol Endpoints ➔ L3: Auth Credentials ➔ L4: Transport/Egress) with SVG Bezier connection wires, 4-Layer Exploded Protocol Assembly Scrubber, and real-time Simulate Pulse dry-run verification.
• In-Page X-Ray HUD Overlay (Ctrl+Shift+X): Shadow-DOM isolated on-page telemetry overlay with real-time hover reticles, animated SVG tether lines connecting DOM elements to their underlying API endpoints, session timers, and threat level readouts.
• Disassembly Mastery Gamification Engine: Evaluates targets across a 5-tier Threat Matrix (Class D to Class S Apex Anomalies), tracking an automated 0–100% Deconstruction Meter (Endpoints, Auth Mechanics, Schemas, Transport) and rewarding Technopath XP and achievements.
• Resilient Fallback Selectors: Automatically generates multi-tiered selector chains (id, data-*, aria-*, name/type, CSS paths, and XPath) for robust automation playback with visual highlight overlays.
• In-Popup Cookie Drawer: Inspect, search, edit, add, delete, and export active tab cookies as JSON for instant credential extraction (auth_token, ct0).
• Dual ISP/ASN Network Telemetry: Live badges displaying public IP, ISP name, ASN, and country code for both your egress connection and destination origin servers.
• Domain-Enriched .tflog Export: Export files named by domain and timestamp (traceforge_<domain>_<id>_<timestamp>.tflog) with metadata-enriched start markers.
• Privacy & Password Masking: Automatically detects sensitive inputs (type="password", card numbers) and redacts values in recorded payloads.
• Modern Dark-Mode Dashboard: Popup UI providing live recording status, elapsed time counter, event counters (actions, requests, frames), and one-click .tflog NDJSON export.
• Native Messaging Bridge: Direct streaming to traceforge-cli native-host or standalone in-browser buffering.

Building the WebExtension:

Run the automated build and packaging pipeline:

# Build optimized release WebAssembly package and validate Manifest V3
./scripts/build-extension.sh

# Or build development mode without wasm-opt
./scripts/build-extension.sh --dev --no-opt

The build pipeline:

1. Compiles crates/traceforge-extension targeting wasm32-unknown-unknown via wasm-pack.
2. Optimizes WASM binary footprint with wasm-opt.
3. Emits ES module wrappers to extension/pkg/.
4. Executes integrity validation on extension/manifest.json, icon assets, permissions, and bootstrap loaders.

Loading into Chromium-based Browsers (Chrome, Brave, Edge):

1. Navigate to chrome://extensions (or brave://extensions, edge://extensions).
2. Toggle Developer mode in the upper-right corner.
3. Click the Load unpacked button in the top-left toolbar.
4. Select the extension/ directory in this repository.
5. Pin TraceForge Recorder to your browser toolbar and click the icon to open the recording popup.

Loading into Firefox:

1. Navigate to about:debugging in the address bar.
2. Select This Firefox from the left navigation panel.
3. Under Temporary Extensions, click Load Temporary Add-on....
4. Select extension/manifest.firefox.json (or extension/manifest.json if on Firefox 121+).
5. The extension will activate with temporary developer permissions.

---

Rust Library API Usage

You can also use TraceForge programmatically in your own Rust projects:

In Cargo.toml:

[dependencies]
traceforge-core = { path = "path/to/crates/traceforge-core" }
traceforge-transport = { path = "path/to/crates/traceforge-transport" }
traceforge-codegen = { path = "path/to/crates/traceforge-codegen" }
traceforge-browser = { path = "path/to/crates/traceforge-browser" }
traceforge-stealth = { path = "path/to/crates/traceforge-stealth" }
traceforge-solver = { path = "path/to/crates/traceforge-solver" }
traceforge-mcp = { path = "path/to/crates/traceforge-mcp" }

Example: Running a Zero-Browser HTTP Task in Rust

use traceforge_transport::runner::{HttpTaskRunner, TaskStep};
use traceforge_transport::proxy::{ProxyRouter, ProxyEndpoint, ProxyProtocol};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // 1. Setup proxy router (optional)
    let router = ProxyRouter::new();
    router.add_proxy(ProxyEndpoint::new(ProxyProtocol::Http, "proxy.example.com", 8080))?;

    // 2. Build task runner
    let runner = HttpTaskRunner::builder()
        .with_proxy_router(router)
        .build();

    // 3. Define steps with dynamic token injection
    let steps = vec![
        TaskStep::get("https://example.com/login"),
        TaskStep::post(
            "https://example.com/api/v1/auth/login",
            "{\"user\":\"admin\"}",
        ).with_header("X-CSRF-Token", "{{csrf_token}}"),
    ];

    // 4. Execute workflow
    let result = runner.execute(&steps).await?;
    println!("Execution status: {:?}, steps completed: {}", result.status, result.steps.len());
    Ok(())
}

Example: Synthesizing OpenAPI from Recorded ActionFlowIR

use traceforge_codegen::OpenApiGenerator;
use traceforge_core::model::ActionFlowIR;

fn generate_spec(ir: &ActionFlowIR) {
    let generator = OpenApiGenerator::new("Reverse Engineered API", "1.0.0");
    let openapi_json = generator.generate_json(ir);
    println!("{}", openapi_json);
}

---

Testing & Quality Assurance

TraceForge enforces strict test coverage across every module:

# Run all workspace integration tests (127 tests)
cargo test --workspace

# Run tests for a specific crate
cargo test -p traceforge-transport
cargo test -p traceforge-browser
cargo test -p traceforge-codegen
cargo test -p traceforge-mcp
cargo test -p traceforge-cli

# Run formatting & clippy checks
cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings

---

License

Licensed under the Apache License, Version 2.0.