ft — Token-Optimized CLI Filter Kernel

ft streamlines verbose CLI output for LLM-driven development. When your tools spew pages of compilation noise, test summaries, and git chatter, ft filters it down to what matters — saving 60–90% of your token budget without losing signal.

Inspired by RTK (Rust Token Killer).

Problem

Running cargo test, git log, npm test, or kubectl get pods produces pages of output. In an LLM context, every line costs tokens, and most lines are noise:

   Compiling libc v0.2.153
   Compiling cfg-if v1.0.0
   Compiling serde v1.0.200
   ...
   Finished dev [unoptimized + debuginfo] target(s) in 15.23s

→ becomes:

cargo build (3 crates compiled)
    Finished dev [unoptimized + debuginfo] target(s) in 15.23s

Quick Start

[dependencies]
ft = "0.1"

use ft::core::runner::{run_filtered, run_passthrough};
use ft::cmds::git::git::compact_diff;

// Filter a cargo build
let result = run_filtered(
    &mut std::process::Command::new("cargo").args(["check"]),
    Box::new(ft::cmds::rust::cargo_cmd::filter_cargo_build),
)?;
println!("{}", result.filtered);

// Compact a git diff
let compact = compact_diff(&raw_diff, 200);

Architecture

ft (Rust crate)
├── core/          # Always-compiled filter engine
│   ├── filter.rs     Language-aware comment removal + smart truncation
│   ├── stream.rs     Streaming filter engine (StreamFilter, BlockHandler)
│   ├── runner.rs     Command execution with filtered/passthrough/capture modes
│   ├── toml_filter.rs  8-stage declarative TOML filter pipeline
│   ├── config.rs     ConfigBuilder + TOML deserialization
│   ├── utils.rs      truncate, strip_ansi, format_tokens, exit_code handling
│   ├── tee.rs        Raw output recovery on failure
│   └── constants.rs  Path and filename constants
├── cmds/          # Ecosystem-specific command filters
│   ├── git/          git log/status/diff, gh, glab, gt
│   ├── rust/         cargo build/test/clippy/install/nextest
│   ├── js/           npm, pnpm, vitest, tsc, prettier, prisma, playwright, next
│   ├── python/       pytest, mypy, ruff, pip
│   ├── go/           go build/test, golangci-lint
│   ├── ruby/         rspec, rubocop, rake
│   ├── dotnet/       dotnet build/test, binlog, trx
│   ├── jvm/          gradlew
│   ├── cloud/        docker, kubectl, curl, wget, psql, aws
│   └── system/       find, grep, deps, ls, tree, env, json, log, pipe, read, wc
├── parser/        # (feat = "parser") Structured output parsing
│   └── types.rs/formatter.rs  TestResult, DependencyState, TokenFormatter
└── ffi/           # (feat = "ffi") C ABI for cross-language bindings
    └── types.rs/mod.rs   extern "C" init/exec/destroy/free

Feature Flags

Feature	Default	Description
stream	yes	Core filter engine + TOML pipeline + config + utils
parser	yes	Structured output parsing (TestResult, DependencyState)
ffi	no	C ABI bindings for Go/Node/Python interop

Ecosystem Filters

Category	Commands
Git	git log/status/diff/show/add/commit/push/pull/branch/fetch/stash/worktree, gh pr/issue/run/repo/api, glab mr/release/ci, gt
Rust	cargo build/test/clippy/check/install/nextest
JavaScript	npm, pnpm, vitest, tsc, prettier, prisma, playwright, next, lint
Python	pytest, mypy, ruff, pip
Go	go build/test, golangci-lint
Ruby	rspec, rubocop, rake
.NET	dotnet build/test, binlog, trx, format report
JVM	gradlew
Cloud	docker, kubectl, curl, wget, psql, aws
System	find, grep, ls, tree, env, json, wc, pipe, read, log, deps, summary, format

TOML Filter Pipeline

Define custom filters in TOML — an 8-stage processing pipeline that transforms command output declaratively:

[[filters]]
command = "cargo"
args = "check"
pipeline = [
  { strip_lines = "^\\s*(Compiling|Checking|Downloading|Finished)" },
  { on_empty = "cargo check: clean" },
]

[[filters]]
command = "make"
pipeline = [
  { strip_lines = "\\[\\d+%\\]" },
  { strip_ansi = true },
  { head_lines = 20 },
  { on_empty = "make: done" },
]

Stages: strip_ansi → replace → match_output → strip/keep_lines → truncate_lines_at → head/tail_lines → max_lines → on_empty

Config

Programmatic config via ConfigBuilder — no filesystem dependency:

use ft::core::config::ConfigBuilder;

let config = ConfigBuilder::new()
    .max_lines(500)
    .max_line_length(120)
    .build();

Or load from a TOML file:

let config = ft::core::config::Config::from_file("ft.toml")?;

FFI / Cross-Language

Enable the ffi feature for C ABI bindings:

FtHandle ft = ft_init(NULL);
FtResult* r = ft_exec(ft, "cargo check");
printf("%s\n", r->filtered_stdout);
ft_free_result(r);
ft_destroy(ft);

This enables bindings for Go (cgo), Node.js (napi-rs), and Python (PyO3).

License

MIT