The Law Firm’s Method Should Outlive Its AI Model

IA
Ignacio Adrián Lerer

9 min de leitura

A lawyer asks an assistant to review a contract. The answer is lucid, the clauses are numbered correctly, and the proposed changes look familiar. Six months later, the firm changes its AI provider. The new assistant produces equally polished work, but asks different questions, overlooks an exception that the previous workflow usually caught, and cannot recover the basis of an earlier recommendation.

What did the firm actually retain from its investment in AI? A subscription, a collection of conversations, or a professional method it can inspect and carry forward?

My proposal is that a law firm should treat its method as an institutional asset separate from the model that happens to execute parts of it. Skills can help express that asset. They become useful when they preserve the questions, evidence requirements, boundaries and tests of a task. A folder of instructions does not, by itself, supply the infrastructure needed to put those instructions into practice.

This is an architectural argument, not a report of measured productivity gains. The contract example below is hypothetical. It follows one matter through the points where a reusable method can preserve professional judgment, and where it can merely conceal its absence.

What survives a successful conversation?

Suppose an experienced lawyer reviews a change-of-control clause. Before proposing language, she checks the definition of control, the transaction structure, the relevant annexes and the consequence of breach. She asks whether consent is required before signing or before closing. She notices that an exception discussed in an email never reached the contract.

A saved answer preserves the proposed wording. It may preserve none of the reasons those questions were necessary. A reusable skill should specify the inputs to request, the comparisons to make, the exceptions to retain and the point at which missing evidence prevents a recommendation. It should also distinguish a draft revision from permission to send it.

Some judgment will remain outside that specification. The lawyer may recognize that the proposed transaction changes the commercial meaning of a familiar clause. Encoding a checklist cannot establish that an assistant will recognize the same change. Making the method explicit is valuable partly because it exposes this limit: colleagues can examine what the procedure covers and what still requires judgment.

Sam Zhai’s discussion of individual and institutional AI literacy raises a related challenge: lawyers need support for deciding what to delegate, verify and rely on. The question I pursue here is what a practice should preserve so those decisions remain inspectable when its technology changes. Instructions, working permissions, source records and evaluations each preserve a different part of that answer.

A portable method needs a working environment

It is useful to separate the model, the application or harness, the tools and the skill. The model processes context and generates responses. The harness organizes the interaction and connects available capabilities. Tools retrieve documents or perform operations. The skill describes a procedure for a class of tasks. The lawyer defines the engagement and takes the professional decisions that remain hers.

These distinctions reveal dependencies. An instruction to verify the annex is ineffective if the assistant cannot retrieve it. An instruction to obtain approval before sending is a weak boundary if the connected tool permits unrestricted sending. A retained source reference is of little help if the reviewer cannot open the source version that supported the conclusion.

The firm therefore needs a small operational contract around each reusable method: which inputs are available, what the system can access, which actions it can take, what evidence it must return, and who can stop or correct the work. The contract need not be elaborate. A draft-only task with selected public documents may require very little infrastructure. A workflow that reads a client data room and communicates with third parties requires substantially more.

Technological independence comes in degrees. A Markdown procedure is easier to move than a workflow whose logic exists only in a provider’s interface. But portability of text does not establish portability of behavior. A replacement model may interpret an instruction differently; a replacement application may expose different context or permissions. The method can survive the substitution while its execution still needs testing.

Follow the matter through its changes of state

Return to the change-of-control review. The assistant finds the clause and concludes that the transaction can proceed without consent. The conclusion might fail at several different points.

The annex may be absent. The assistant may have treated management’s description of it as documentary evidence. The clause may be accurately quoted but interpreted without the governing law or transaction facts. A short client summary may have removed the condition on which the conclusion depends. The Word file may preserve the wording while losing comments or pending revisions. Finally, a properly qualified draft may be sent before anyone with the relevant mandate approves it.

Calling all these failures “hallucination” obscures the remedies. A missing annex calls for retrieval or a narrowed conclusion. An unsupported inference calls for revised analysis. A lost qualification calls for checking the final communication against its basis. A document mutation calls for file-integrity checks. Unauthorized transmission calls for permission boundaries and an examination of what actually happened.

This suggests organizing controls around transitions: material received becomes evidence; evidence supports analysis; analysis becomes a recommendation; a recommendation becomes a communication or action. At each transition, the question is what must remain true for the next step to be justified.

The firm should preserve both the conclusion and the unresolved dependency. “No material restriction identified in the agreements supplied” has a different scope from “the transaction has no material contractual restriction.” A system that silently converts the first statement into the second has changed the work even if every citation remains real.

Jurisdiction belongs in the method

The same issue arises when a procedure crosses legal systems. A method developed around one jurisdiction’s treatment of precedent cannot acquire universal scope by being translated. A citation network may help identify influential decisions, but network position alone does not establish the legal effect of a judgment in another legal order. Absence from a corpus establishes a limit of that corpus or search, not absence of law.

The reusable method should therefore preserve a place for jurisdiction, source hierarchy, time and the intended use of the analysis. These are inputs to legal work rather than decorative labels. They affect which documents to obtain, which propositions need verification and who can assess them.

The language or country associated with a model’s developer does not establish competence in a legal system. Neither does a fluent answer in the local language. For the proposed workflow, the relevant evidence comes from its treatment of the applicable sources, distinctions and exceptions under defined conditions. No comparison of English-, Chinese- or French-developed models is claimed here.

This makes a practical case for keeping legal knowledge and source access distinguishable from the model. A firm may change the generator while retaining its authorized review standard and source records. It must then check whether the new arrangement still applies them adequately.

Improvement must preserve the evidence of failure

Once a method is explicit, changing it becomes tempting. An assistant misses an exception, so a new instruction is added. Another case produces a lengthy answer, so the instruction is shortened. After enough such repairs, the procedure may become difficult to follow, or may fix one problem by creating another.

A more defensible cycle starts with a frozen version and a concrete failure. The team records the task, initial state, available tools, expected result and observed defect. It proposes a bounded change and checks it against the original failure, other known cases, cases withheld from the revision process, and requests on which the skill should not activate. Recovery and rollback belong in that record.

Different checks answer different questions. A deterministic comparison can establish that a required clause survived a file transformation. A lawyer may be needed to assess whether the retained qualification expresses the relevant legal uncertainty. An AI judge can assist with a rubric, but the generator should not be the only judge of its own improvement. Some failures, such as unauthorized disclosure, should block acceptance regardless of an otherwise attractive score.

Self-improvement, in this sense, is a governed process for proposing and accepting changes. It is not permission for a system to expand its own authority or promote its own results. The original automatic result and subsequent human review should remain separate. Rewriting the record after a reviewer repairs the answer would make the metric look better while destroying evidence of what the system actually did.

The cost of this discipline must also be examined. A workflow that blocks every task has not become useful merely because it avoids visible mistakes. A review chain can consume more professional time than it saves. Evaluation should include whether valid work finishes, what correction it needs and how much effort remains. This article does not establish net savings for any implementation.

Start with one method the firm can defend

A small practice need not begin with a platform or an extensive catalogue. It can choose one repeated task, preserve its current method, identify its evidence and action boundaries, and test a proposed skill on permitted synthetic or public material. Another person or a later session should be able to use the package without reconstructing the original conversation.

What should survive is modest but valuable: editable instructions, an identifiable version, authorized sources or retrieval requirements, representative tests, known limitations and a responsible owner. Confidential matter files need not become part of a public skill. A public package can explain a general procedure while client-specific standards and records remain within their authorized environment.

Skills then occupy a useful place between personal experience and common infrastructure. They make portions of professional practice available for inspection and revision. The surrounding system gives those instructions access to appropriate evidence, constrains their effects and preserves a record that another lawyer can examine.

A firm that changes models should not have to rediscover why it asks for an annex, retains a reservation or refuses to send an unapproved draft. It should be able to point to the method, the conditions under which it was tested and the decision that remains with the professional. That is a more durable ambition than accumulating successful conversations: building a practice whose knowledge can outlive its current tools.

Reference: Sam Zhai, Legal skills in the genAI age, Part 3: Can you (still) smell when AI is wrong?

The present essay develops an architectural proposal; it does not independently verify the empirical studies discussed in that article.

Recursos mencionados

IA

Autor

Ignacio Adrián Lerer

Argentine attorney with 35 years of practice in contracts, M&A, corporate law and compliance. I research adjudicative AI, legal-agent governance and computational law, focusing on authority, impartiality, traceability and effective human review. My peer-reviewed articles in the Journal of Computational Law and Legal Technology are Computational Detection of Constitutional Drift: Network Analysis and Semantic Measurement of Argentine Supreme Court Jurisprudence (1922–2025) and An Agent-Based Framework for Simulating Institutional Lock-in Under Repressive Regimes. I also publish an open research programme on Zenodo, most recently Machine-Legible Justice. Founder of JustitIA and IntegridAI, I build public legal-agent skills that turn professional methods and institutional safeguards into inspectable, reusable workflows.