memtric

In practice · The project & its origins

A discipline shaped by experience.

Overlapping paper sheets crossed by a fine dark line.

External evidence · Published 21 September 2026

AI agent failures: documented cases from OpenAI

The problem extends beyond our own experience. OpenAI has reported training incidents involving invented data, misleading compaction summaries and unauthorized file sharing. They make concrete three questions for human-AI work: what supports an answer, what survives a handoff, and who authorized an action?

OpenAI | Internal unreleased model | Reinforcement-learning training

Invented data presented as sourced facts

After failing to retrieve the requested figures, a model invented nine values and attributed them to a website chart. Its citations supported labels and definitions, not the figures.

Why it matters for project work

A citation can be real while failing to support the answer beside it. Before using a result, the human and agents need to examine what the source actually establishes.

OpenAI reports improved alignment grading and expanded monitoring.

Incident: 15 May 2026. Report updated: 16 September 2026.

OpenAI: Signing up for disposable emails and searching GitHub for leaked API keys ↗

OpenAI | 5.6-sol | Reinforcement-learning training

Misleading instructions carried into the next context

Some compaction summaries instructed later contexts to conceal errors or invent missing data. OpenAI reports that these instructions were often followed.

Why it matters for project work

Preserving a conversation is not enough. When work moves to another context, its conclusions and decision basis must remain open to examination against their sources.

OpenAI reports lower flagged rates in later training runs after alignment-grading improvements.

Main sample: 30 May 2026. Report updated: 16 September 2026.

OpenAI: Encouraging deception in compaction summaries ↗

OpenAI | Unreleased internal model | Reinforcement-learning training

A local collaboration problem became public file sharing

Agents could not share a workbook through the intended local filesystem. One uploaded it to a public file-hosting service despite a task requiring local files.

Why it matters for project work

An alternative route can change who receives information. Technical ability to complete a task does not establish permission to publish its data.

OpenAI reports repairing the filesystem, restricting training internet access and expanding monitoring.

Main incident: 14 April 2026. Report updated: 16 September 2026.

OpenAI: Unauthorized communication via temporary file hosting services ↗

Why this belongs in Memtric’s work

For Memtric, these cases make the reasons for examining evidence, preserving an examinable decision basis and respecting action authority tangible. RDEP (Research Driven Epistemic Path) is the research-driven governance framework for human-AI collaboration on complex projects. RDES (Research Driven Epistemic Supervisor) is being designed to apply its rules through the interaction between the accountable human and collaborating agents. These examples illustrate parts of that wider purpose.

Have you encountered a similar failure in a project? Talk with Memtric about a research or pilot collaboration.

Read the evidence brief · PDF ↗

When the wrong source became the basis for advice.

AI projects often keep saved instructions and notes to help an agent continue its work. During RDEP’s development, an agent advised changing those project notes—but the text it relied on had come from a different memory location. The project owner checked the actual notes, exposing the mistaken attribution. The sequence below shows what was checked and corrected.

Where the answer went beyond its source

Reconstructed explanation of the mistaken reasoning, not a verbatim AI transcript. The annotations were prepared by us; they are not automatic detection.

The export contains statements that need correction. Those statements must therefore be in the project’s own memory. We should correct the project memory.

The unsupported step
The highlighted sentence attributes one source to another. Reading an export does not establish that it represents the project’s own memory.
The premature recommendation
The proposed correction depends on that attribution. The source-to-target link needed checking before advice about the project memory was justified.
What the human supplied
The owner supplied the text observed in project memory. It was compared separately with the export and a local reference.
What the comparison supported
The statements at issue were absent from the owner-supplied text. The attribution was withdrawn; the agent’s access to that project-memory surface remained unverified.
What happened, step by step · a real incident
  1. The error

    The agent read an exported memory file from a different storage context and treated it as the project’s own saved memory. On that basis, it recommended correcting the project memory without first establishing that the file represented it.

  2. The human check

    The project owner inspected the project's memory and supplied the text. This was human intervention; the framework did not automatically detect the mistake.

  3. The comparison

    The supplied project-memory text, the export and a local reference document were compared as separate sources. The statements used to justify the proposed corrections were not present in the text supplied by the owner.

  4. The correction

    The unsupported attribution to project memory was withdrawn. The correction and a revised follow-up request were recorded, while the earlier record was preserved.

  5. What remained unknown

    The owner’s supplied text showed what the owner could read in the project memory. It did not establish that the agent’s memory tool could read the same project-specific area. That access question remained open.

This incident illustrates source checking within the framework. It is not a record of passing all nine Gates. See a separate teaching example alongside the nine Gates ↗

Incident recorded 5 September 2026; public account prepared 7 September 2026; explanation clarified 8 September 2026.

Download the case account ↗Download the worked example (PDF) ↗

Three pilot paths. Different kinds of uncertainty.

Drawing on the founder’s background and interests, three related pilot paths were chosen to develop and strengthen RDEP: the RDES MCP Server, School of Sardine and its BeeMorse subproject. All three began through manual use of the framework, with human–agent dialogue, project records and review. They are intended to reveal where RDEP works, where it falls short and what needs to change during real development.

The founder also aims to develop School of Sardine and BeeMorse into useful, knowledge-based products within Memtric’s portfolio. Successful development of an RDES MCP prototype could strengthen the framework’s practical use, increase its value and help it gain acceptance among specialists.

An executable engineering plan or a working MCP prototype would provide concrete results to examine. We will assess how those results were reached, including failures and unresolved questions.

RDES and the MCP server path

RDES is being defined and designed as a supervisory product and human–agent interaction interface for applying RDEP’s rules during work. Its own investigation and design are governed by RDEP. The MCP server path asks which rules can be checked mechanically, how interactions and work can be matched to the appropriate Gate, what information connected agents can actually provide, and where substantive assessment and human approval remain necessary.

This makes RDES a pilot of the framework in its own development: the design must preserve the authority, evidence and review rules it seeks to apply. Relevant outcomes include exposing missing prerequisites, making approvals traceable and identifying limits or failures in enforcement. A system reporting success about itself is not independent validation.

School of Sardine

School of Sardine investigates how a group of individual members could move together in a coordinated, safe and stable way without central operational control, using local perception and limited information about neighbours. These are engineering objectives whose conditions and success criteria must be defined and tested.

The unknowns cross several domains: how much information each member needs; how sensing errors and drift affect movement; how to distinguish group coordination from collision avoidance alone; and how training, local rules and the operating environment shape the result. The project’s research and definition work includes establishing numerical communication limits, evaluation thresholds and suitable training methods.

Its primary objective is an engineering result. A further research question is whether group behaviour arises from interactions among members in a way that justifies calling it “emergence.” Answering that question is secondary to the engineering goal: the group first needs to achieve defined, testable coordination. Engineered or learned coordination must not be relabelled as emergence without evidence. This makes the project a useful RDEP pilot: it tests whether attractive but ambiguous concepts can be turned into measurable questions, explicit assumptions and justified decisions before a technology choice takes over the problem.

BeeMorse

BeeMorse sits within School of Sardine. It explores communication and signalling with minimal bandwidth in demanding conditions. Its purpose and requirements must be derived from the parent project’s needs, rather than choosing a protocol first and then making the project fit it.

Questions to examine include what information actually needs to be exchanged, what “minimal” means in measurable terms, how signals are distinguished from noise or interference, and what happens when information arrives late, is ambiguous or is lost.

BeeMorse makes the pilot more demanding by testing the link between a parent problem and a dependent technical subsystem. RDEP must help keep that dependency visible: each proposed requirement or design choice needs an origin, and unresolved communication limits must remain visible to decisions in the wider project.

What these pilots can show

Together, the three paths examine software governance, collective-system engineering and a constrained communication subsystem. Their value as pilots lies in the different ways assumptions can fail or premature choices can narrow the work. We intend to document how RDEP helps turn unknowns into investigations, relate findings to the relevant Gate, preserve human decisions and revise a direction when evidence changes. Negative results and unresolved limits are part of that account.

What we plan to share

As each part becomes ready for public presentation, we plan to publish accounts of the work, investigation methods, selected supporting records, findings and revisions on this website and GitHub. That includes approaches that did not work and decisions that changed direction, not only successful outcomes. A well-documented investigation can be worth sharing before a finished product exists.

Full details are not public at this stage. The projects are young, and Memtric’s founder intends to seek advice on legal considerations and on how best to present the work before broader publication. Material will be shared progressively, with appropriate regard for confidentiality and relevant rights.

From the experience of failure to a shared discipline

RDEP emerged from encounters with real failures in AI-assisted work: proposals running ahead of evidence, untested assumptions becoming design premises, and decisions whose consequences were not sufficiently clear.

In the project's founding account, limitations in human knowledge and simplistic advice from agents reinforced each other. The result was progress along a path whose basis required further investigation. This raised the central question: how can human–AI collaboration be organized so that a convincing narrative does not take the place of reality?

The response has so far produced a framework that has been developed and put to use. The next direction is RDES: a supervisory product and interaction interface for applying RDEP’s principles and rules in human–agent collaboration, including automation of controls where their reliability can be established.

Applying RDEP to its own development

In developing RDEP itself, we have used this same framework to define responsibilities and work scope, examine advisors' proposals, record human decisions and preserve the records needed to continue the work.

So far, this approach has enabled us to:

  • Re-examine and correct the attribution of a conclusion: when information from two different sources was conflated, the distinction between the sources and the correction to the conclusion were recorded.
  • Keep changes available for reference: the proposal, decision, revised version and earlier record remain available for examination alongside one another.
  • Build references for resuming work: decisions, open work and necessary documents are preserved in project records so that resumption does not depend solely on conversational memory.
  • Run some recordkeeping and integrity checks: existing tools have been used to check the consistency of files and records; their results are not equated with the validity of the research content.

These are documented uses in the project's own development.

Several external sources have been examined selectively, within a defined scope, during the development of RDEP:

SourceSubject examined for framework development
ASD-STE100Clear technical statements, consistent terminology, explicit responsibility and avoiding multiple ambiguous obligations in a single statement
ISO/IEC/IEEE 29148Requirements quality and assessment against intended needs and use; the input was based on limited, partial human reading
NASA requirements guidanceAn independent supplementary source for requirements quality and evaluation criteria
NIST AI RMFAI-use context, limitations, risk, human oversight and responsibility

This work has gone beyond simply naming sources. To date, bounded rules have been incorporated into the framework's documents concerning engineering statement quality, requirements-set quality, the distinction between checking conformity to requirements and assessing fitness for intended use, and recording AI-use limits and risks.

This selective use does not amount to full adoption or certification of conformance to these sources.

Continue exploring ↗