systima

We built a CLI tool that scans your codebase for EU AI Act compliance risks.

`npx @systima/comply scan` analyses your repository to detect AI framework usage, traces how AI outputs flow through the program, and flags patterns that may trigger regulatory obligations.

It runs in CI and posts findings on pull requests (no API keys required).

Under the hood it performs AST-based import detection using the TypeScript Compiler API and web-tree-sitter WASM across 37+ AI frameworks. It then traces AI return values through assignments and destructuring to identify four patterns:

1. conditional branching on AI output

2. persistence of AI output to a database

3. rendering AI output in a UI without disclosure

4. sending AI output to downstream APIs

Findings are severity-adjusted by system domain. You declare what your system does (customer support, credit scoring, legal research, etc) and the scanner adjusts accordingly.

Example:

- a chatbot routing tool using AI output in an `if` statement produces an informational note

- a credit scoring system doing the same produces a critical finding

We tested it against Vercel’s 20k-star AI chatbot repository; the scan took about 8 seconds. Example PR comment with full results: https://github.com/systima-ai/chatbot-comply-test/pull/1

Comply ships as an npm package, a GitHub Action (systima-ai/comply@v1), and a TypeScript API. It can also generate PDF reports and template compliance documentation.

Repo and explanation: https://systima.ai/blog/systima-comply-eu-ai-act-compliance-...

Feedback welcome on the call-chain tracing approach and whether the domain-based severity model makes sense.

EU legislation (which affects UK and US companies in many cases) requires being able to truly reconstruct agentic events.

I've worked in a number of regulated industries off & on for years, and recently hit this gap.

We already had strong observability, but if someone asked me to prove exactly what happened for a specific AI decision X months ago (and demonstrate that the log trail had not been altered), I could not.

The EU AI Act has already entered force, and its Article 12 kicks-in in August this year, requiring automatic event recording and six-month retention for high-risk systems, which many legal commentators have suggested reads more like an append-only ledger requirement than standard application logging.

With this in mind, we built a small free, open-source TypeScript library for Node apps using the Vercel AI SDK that captures inference as an append-only log.

It wraps the model in middleware, automatically logs every inference call to structured JSONL in your own S3 bucket, chains entries with SHA-256 hashes for tamper detection, enforces a 180-day retention floor, and provides a CLI to reconstruct a decision and verify integrity. There is also a coverage command that flags likely gaps (in practice omissions are a bigger risk than edits).

The library is deliberately simple: TS, targeting Vercel AI SDK middleware, S3 or local fs, linear hash chaining. It also works with Mastra (agentic framework), and I am happy to expand its integrations via PRs.

Blog post with link to repo: https://systima.ai/blog/open-source-article-12-audit-logging

I'd value feedback, thoughts, and any critique.