Plumbline

Reliability & architecture analysis for agentic systems · by ActaClad

49 files scanned25 rules run3 findingsPlumbline v0.0.1

93/100

Readiness Score

Reliability802 issues

Architecture & Agentic Maturity100no issues

Harness Engineering991 issue

Security100no issues

✗ Quality Gate failed — 2 blocking findings (Blockers and High-confidence Criticals), listed below

3 findings2 Critical1 Major

Severity	Rule	Location	What & why
Major Medium	PLB-OBS-001 · NIST-AI-RMF:MEASURE	llm/default_plugins/openai_models.py:961:26	This project ships LLM/agent code with no in-code tracing or instrumentation; production failures will be undiagnosable. (If you enable tracing via env vars / auto-instrumentation, ignore this.) Note: harness rules reason about the whole repo — scan the project root (including tests/ and CI), not just src/. An LLM/agent app with no tracing is unobservable in production — failures can't be traced. How to fix Instrument the app so production runs are traceable. - An observability SDK: OpenTelemetry, LangSmith, Langfuse, Phoenix/Arize, Logfire, Helicone, Traceloop, … - or the framework's tracing callbacks (LangChain callbacks, CrewAI telemetry). If you already enable tracing via environment variables (LANGCHAIN_TRACING_V2=true, OTEL_* auto-instrumentation), this finding does not apply — Plumbline cannot see env-var activation; ignore or suppress it.
Critical High	PLB-OUT-001	llm/default_plugins/openai_models.py:1027:35	LLM output is parsed with json.loads and no error handling; a malformed generation will raise and crash the request. `json.loads` on raw LLM output with no guard crashes on the first malformed generation. How to fix Validate model output against a schema and handle parse failure (retry or fallback). Bad: data = json.loads(resp.choices[0].message.content) Good: try: data = json.loads(resp.choices[0].message.content) except json.JSONDecodeError: data = repair_or_retry(...) # better: use the provider's structured-output / response_format and validate
Critical High	PLB-OUT-001	llm/default_plugins/openai_models.py:1143:35	LLM output is parsed with json.loads and no error handling; a malformed generation will raise and crash the request. `json.loads` on raw LLM output with no guard crashes on the first malformed generation. How to fix Validate model output against a schema and handle parse failure (retry or fallback). Bad: data = json.loads(resp.choices[0].message.content) Good: try: data = json.loads(resp.choices[0].message.content) except json.JSONDecodeError: data = repair_or_retry(...) # better: use the provider's structured-output / response_format and validate

Severity

Rule

Location

What & why

Major

Medium

PLB-OBS-001 · NIST-AI-RMF:MEASURE

llm/default_plugins/openai_models.py:961:26

This project ships LLM/agent code with no in-code tracing or instrumentation; production failures will be undiagnosable. (If you enable tracing via env vars / auto-instrumentation, ignore this.) Note: harness rules reason about the whole repo — scan the project root (including tests/ and CI), not just src/.

An LLM/agent app with no tracing is unobservable in production — failures can't be traced.

How to fix

Instrument the app so production runs are traceable.

- An observability SDK: OpenTelemetry, LangSmith, Langfuse, Phoenix/Arize,
  Logfire, Helicone, Traceloop, …
- or the framework's tracing callbacks (LangChain callbacks, CrewAI telemetry).

If you already enable tracing via environment variables
(LANGCHAIN_TRACING_V2=true, OTEL_* auto-instrumentation), this finding does not
apply — Plumbline cannot see env-var activation; ignore or suppress it.

Critical

High

PLB-OUT-001

llm/default_plugins/openai_models.py:1027:35

LLM output is parsed with json.loads and no error handling; a malformed generation will raise and crash the request.

`json.loads` on raw LLM output with no guard crashes on the first malformed generation.

How to fix

Validate model output against a schema and handle parse failure (retry or fallback).

Bad:
    data = json.loads(resp.choices[0].message.content)

Good:
    try:
        data = json.loads(resp.choices[0].message.content)
    except json.JSONDecodeError:
        data = repair_or_retry(...)
    # better: use the provider's structured-output / response_format and validate

Critical

High

PLB-OUT-001

llm/default_plugins/openai_models.py:1143:35

LLM output is parsed with json.loads and no error handling; a malformed generation will raise and crash the request.

`json.loads` on raw LLM output with no guard crashes on the first malformed generation.

How to fix

Validate model output against a schema and handle parse failure (retry or fallback).

Bad:
    data = json.loads(resp.choices[0].message.content)

Good:
    try:
        data = json.loads(resp.choices[0].message.content)
    except json.JSONDecodeError:
        data = repair_or_retry(...)
    # better: use the provider's structured-output / response_format and validate