mendmark.html

<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <meta
      name="description"
      content="Mendmark mutation-tests agent eval suites by planting controlled tool-use failures and checking whether the team's existing evaluators catch them."
    />
    <meta name="theme-color" content="#0b0d0c" />
    <meta property="og:title" content="Mendmark | Mutation Testing for Agent Evals" />
    <meta
      property="og:description"
      content="An open-source Python package that plants controlled trace faults and shows which agent failures an eval suite misses."
    />
    <meta property="og:type" content="article" />
    <meta property="og:url" content="https://danielgaskins.com/mendmark.html" />
    <meta property="og:site_name" content="Daniel Gaskins" />
    <meta property="article:author" content="Daniel Gaskins" />
    <meta property="article:published_time" content="2026-07-23" />
    <meta property="article:modified_time" content="2026-08-04" />
    <meta property="og:image" content="https://danielgaskins.com/assets/img/mendmark-og-v1.jpg" />
    <meta property="og:image:secure_url" content="https://danielgaskins.com/assets/img/mendmark-og-v1.jpg" />
    <meta property="og:image:width" content="1200" />
    <meta property="og:image:height" content="630" />
    <meta property="og:image:alt" content="A controlled fault being exposed inside an AI agent tool-call chain" />
    <meta name="twitter:card" content="summary_large_image" />
    <meta name="twitter:title" content="Mendmark | Mutation Testing for Agent Evals" />
    <meta
      name="twitter:description"
      content="An open-source Python package that plants controlled trace faults and shows which agent failures an eval suite misses."
    />
    <meta name="twitter:image" content="https://danielgaskins.com/assets/img/mendmark-og-v1.jpg" />
    <meta name="twitter:image:alt" content="A controlled fault being exposed inside an AI agent tool-call chain" />
    <link rel="canonical" href="https://danielgaskins.com/mendmark.html" />
    <title>Mendmark | Mutation Testing for Agent Evals | Daniel Gaskins</title>
    <link rel="icon" href="./favicon.svg" type="image/svg+xml" />
    <link rel="preconnect" href="https://fonts.googleapis.com" />
    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
    <link
      href="https://fonts.googleapis.com/css2?family=DM+Mono:wght@300;400;500&amp;family=Manrope:wght@400;500;600;700&amp;display=swap"
      rel="stylesheet"
    />
    <link rel="stylesheet" href="./styles.css" />
    <script src="./script.js" defer></script>
    <script type="application/ld+json">
      {
        "@context": "https://schema.org",
        "@type": "SoftwareSourceCode",
        "@id": "https://danielgaskins.com/mendmark.html#software",
        "name": "Mendmark",
        "description": "Open-source mutation testing for AI agent evaluation suites.",
        "url": "https://danielgaskins.com/mendmark.html",
        "codeRepository": "https://github.com/danielgaskins/mendmark",
        "downloadUrl": "https://pypi.org/project/mendmark-evals/",
        "programmingLanguage": "Python",
        "runtimePlatform": "Python 3",
        "license": "https://opensource.org/license/mit",
        "version": "0.4.2",
        "author": {"@id": "https://danielgaskins.com/#daniel-gaskins", "@type": "Person", "name": "Daniel Gaskins"}
      }
    </script>
  </head>
  <body>
    <a class="skip-link" href="#main">Skip to content</a>

    <header class="site-header" data-header>
      <a class="wordmark" href="./" aria-label="Daniel Gaskins, home">
        <span>DG</span>
        <span class="wordmark__name">Daniel Gaskins</span>
      </a>

      <button class="nav-toggle" type="button" aria-expanded="false" aria-controls="site-nav">
        <span></span><span></span>
        <span class="sr-only">Toggle navigation</span>
      </button>

      <nav id="site-nav" class="site-nav" aria-label="Primary navigation">
        <a href="./#work">Work</a>
        <a href="./#capabilities">How I work</a>
        <a href="./blog.html">Blog</a>
        <a href="./#about">About</a>
        <a href="./resume.html">Résumé</a>
        <a class="nav-cta" href="mailto:hello@danielgaskins.com">Talk with me</a>
      </nav>
    </header>

    <main id="main" class="case-study">
      <section class="case-hero section-shell">
        <p class="eyebrow"><span class="status-dot"></span> Open source / PyPI 0.4.2 / 2026</p>
        <h1>Your agent passed the test. Would the test notice if it broke?</h1>
        <p class="case-hero__lede">
          Mendmark plants controlled faults in passing agent traces, then runs the same evals
          again. A missed tool call, wrong argument, repeated side effect, or hidden error becomes
          a concrete test gap instead of a surprise in production.
        </p>
        <div class="case-hero__actions">
          <a class="button button--primary" href="https://pypi.org/project/mendmark-evals/" target="_blank" rel="noreferrer">Install from PyPI</a>
          <a class="button button--quiet" href="https://github.com/danielgaskins/mendmark" target="_blank" rel="noreferrer">View source on GitHub</a>
        </div>
      </section>

      <section class="case-signal" aria-label="Current project scope">
        <div class="section-shell case-signal__grid">
          <div><strong>24</strong><span>reviewable golden cases</span></div>
          <div><strong>263</strong><span>pinned mutations</span></div>
          <div><strong>10</strong><span>operating domains</span></div>
          <div><strong>100%</strong><span>complete-profile kill rate</span></div>
        </div>
      </section>

      <section id="method" class="case-section section-shell">
        <div class="case-section__label">
          <p class="section-index">01 / THE TEST OF THE TEST</p>
        </div>
        <div class="case-section__body">
          <h2>Plant a fault. See if the eval catches it.</h2>
          <p class="case-section__lead">
            A passing case is the starting point. Mendmark changes one part of its output or tool
            trace, then reruns the team’s existing evaluators. The mutation is killed when an eval
            changes from pass to fail. If every eval still passes, Mendmark names the blind spot.
          </p>
          <div class="case-flow" aria-label="Mendmark task lifecycle">
            <div><span>01</span><strong>Load</strong><small>passing cases + tool contracts</small></div>
            <i>→</i>
            <div><span>02</span><strong>Mutate</strong><small>one controlled fault at a time</small></div>
            <i>→</i>
            <div><span>03</span><strong>Rerun</strong><small>the same evaluator suite</small></div>
            <i>→</i>
            <div><span>04</span><strong>Gate</strong><small>block survivors and regressions</small></div>
          </div>
          <p>
            The resulting kill rate measures the eval suite, not the agent. A surviving critical
            fault is a specific evaluator problem the team can reproduce and fix.
          </p>
        </div>
      </section>

      <section class="case-section case-section--paper-dark">
        <div class="section-shell case-section__inner">
          <div class="case-section__label">
            <p class="section-index">02 / PUBLIC GOLDEN SET</p>
          </div>
          <div class="case-section__body">
            <h2>Three evaluators. The same 263 broken runs.</h2>
            <p class="case-section__lead">
              The versioned golden set contains 24 passing cases, 13 tool contracts, and 39
              tool calls across search, commerce, payments, CRM, communications, databases,
              scheduling, weather, support, and multi-step workflows. Every case and expected
              trace is public.
            </p>
            <div class="case-task-list">
              <article>
                <span>RESPONSE ONLY</span>
                <h3>87 caught. 176 missed.</h3>
                <p>The final answer exposed some failures, but 162 critical tool-behavior faults still passed.</p>
              </article>
              <article>
                <span>TRACE ONLY</span>
                <h3>215 caught. 48 missed.</h3>
                <p>The ordered calls were correct, but removed or replaced final responses escaped.</p>
              </article>
              <article>
                <span>TRACE + OUTCOME</span>
                <h3>263 caught. 0 missed.</h3>
                <p>Exact checks of the ordered tool trace and final outcome detected every planted fault in the set.</p>
              </article>
            </div>
            <div class="case-note">
              <span>SCOPE</span>
              <p>
                This result measures evaluator sensitivity to these controlled faults. It does
                not establish overall agent accuracy or safety. The benchmark is deterministic,
                runs offline without model calls, and is pinned so an unreviewed change to its
                cases, mutations, or expected results fails CI.
              </p>
            </div>
          </div>
        </div>
      </section>

      <section class="case-section case-section--dark">
        <div class="section-shell case-section__inner">
          <div class="case-section__label">
            <p class="section-index">03 / CONTROLLED FAILURES</p>
          </div>
          <div class="case-section__body">
            <h2>What Mendmark breaks on purpose.</h2>
            <div class="case-task-list">
              <article>
                <span>TOOL SELECTION</span>
                <h3>A required call disappears</h3>
                <p>Mendmark removes each tool call and adds an undeclared tool to see whether the trace still passes.</p>
              </article>
              <article>
                <span>ARGUMENTS</span>
                <h3>The right tool gets the wrong input</h3>
                <p>One argument changes at a time, exposing evals that check the tool name but ignore its payload.</p>
              </article>
              <article>
                <span>SIDE EFFECTS</span>
                <h3>The payment happens twice</h3>
                <p>Calls marked as side effecting are duplicated to test whether the suite notices unsafe retries.</p>
              </article>
              <article>
                <span>RECOVERY</span>
                <h3>The tool fails. The agent says it worked.</h3>
                <p>A successful result becomes an error while the final response still claims completion.</p>
              </article>
              <article>
                <span>FINAL RESPONSE</span>
                <h3>The answer vanishes or changes</h3>
                <p>The response is removed or replaced to confirm that outcome checks are doing real work.</p>
              </article>
            </div>
          </div>
        </div>
      </section>

      <section class="case-section section-shell">
        <div class="case-section__label">
          <p class="section-index">04 / TOOL ROLLOUTS</p>
        </div>
        <div class="case-section__body">
          <h2>A new tool should change the release conversation.</h2>
          <p class="case-section__lead">
            Mendmark hashes every declared tool contract. When a tool appears or its schema
            changes, the report shows whether any case exercises it and whether those evals catch
            faults in its calls. Basic schema checks also find missing required arguments and
            wrong value types without recording the values themselves.
          </p>
          <div class="case-comparison">
            <div>
              <span>BEFORE</span>
              <p>A pull request adds a refund tool. The aggregate agent score stays green.</p>
            </div>
            <div>
              <span>AFTER</span>
              <p>The report marks the new side effect, shows its mutation coverage, and blocks any critical survivor.</p>
            </div>
          </div>
          <p>
            A committed baseline remembers which faults the suite caught before. If a changed
            metric lets one of those faults pass, CI reports the exact regression.
          </p>
        </div>
      </section>

      <section class="case-section case-section--paper-dark">
        <div class="section-shell case-section__inner">
          <div class="case-section__label">
            <p class="section-index">05 / WHAT SHIPS TODAY</p>
          </div>
          <div class="case-section__body">
            <h2>A released local engine with a small data footprint.</h2>
            <div class="case-columns">
              <div>
                <h3>Open-source engine</h3>
                <ul>
                  <li>DeepEval suite adapter</li>
                  <li>Framework-neutral JSON adapter</li>
                  <li>Nine built-in mutation operators</li>
                  <li>Per-tool mutation coverage</li>
                  <li>Tool contract and schema tracking</li>
                  <li>Regression-aware CI gates</li>
                </ul>
              </div>
              <div>
                <h3>Report boundary</h3>
                <ul>
                  <li>JSON, JUnit, and SARIF output</li>
                  <li>Custom mutation plugins</li>
                  <li>Source, policy, and CI provenance</li>
                  <li>Signed report verification with Cosign</li>
                  <li>PyPI Trusted Publishing provenance</li>
                </ul>
              </div>
            </div>
            <div class="case-note">
              <span>LIMIT</span>
              <p>
                A passing mutation audit does not prove an agent is safe. It shows that the
                configured eval suite caught the faults Mendmark tested. Teams still need
                product-specific cases and mutations for their highest-risk behavior.
              </p>
            </div>
          </div>
        </div>
      </section>

      <section class="case-next section-shell">
        <p class="section-index">06 / TRY IT</p>
        <h2>Run a complete audit without an API key.</h2>
        <p>
          The repository includes both a small refund-agent demonstration and the public golden
          set with 263 pinned mutations. Clone the project and reproduce the results without an
          API key. Teams with an existing suite can use the DeepEval adapter or connect any local
          evaluator through the versioned JSON protocol.
        </p>
        <div class="case-next__actions">
          <a class="button button--primary" href="https://github.com/danielgaskins/mendmark#quick-start" target="_blank" rel="noreferrer">Run the quick start</a>
          <a class="button button--quiet" href="https://github.com/danielgaskins/mendmark/issues/1" target="_blank" rel="noreferrer">Request a design-partner pilot</a>
        </div>
      </section>
    </main>

    <footer class="site-footer section-shell">
      <p>© <span data-year></span> Daniel Gaskins</p>
      <p>Physics · Models · Products</p>
      <a href="#main">Back to top ↑</a>
    </footer>
  </body>
</html>