Structured Extraction at Scale: How Iterative Self-Refinement Turns Regulatory Text into Machine-Readable Obligations

The deployment of large language models in high-stakes domains such as finance, healthcare, and legal services has exposed a fundamental tension in modern AI alignment. While much research focuses on aligning models with human preferences through reinforcement learning from human feedback, regulatory compliance demands a stricter form of alignment with codified legal obligations. These obligations are not suggestions but binding constraints encoded in dense, hierarchically structured documents that resist simple parsing. The manual extraction of these rules by legal experts remains a significant bottleneck, creating what amounts to a compliance tax on AI deployment. In their paper "De Jure: Iterative LLM Self-Refinement for Structured Extraction of Regulatory Rules," researchers from The Vanguard Group present a compelling solution that automates this extraction process without requiring human annotation or domain-specific prompting.

The Architecture of Hierarchical Decoupling

De Jure operates through a four-stage pipeline that treats regulatory documents as structured dependency graphs rather than flat text. The first stage normalizes source PDFs or HTML documents into structured Markdown, applying SHA-256 fingerprinting for version control. This preprocessing step is crucial because regulatory documents often contain complex formatting, nested sections, and amendment histories that confuse naive text extraction.

The second stage employs a domain-agnostic LLM prompt to decompose sections into typed rule units conforming to a detailed JSON schema. This schema represents a significant departure from simpler text classification approaches. Each extracted rule contains granular semantic fields including action, action_object, method, conditions, constraints, exceptions, penalties, purpose, and verbatim source spans. This level of structural decomposition allows downstream systems to reason not just about what a rule says, but about the mechanistic logic of compliance; specifically, under what conditions certain actions become obligatory or prohibited.

The third stage introduces a multi-criteria LLM-as-a-judge evaluation framework that scores extractions across 19 distinct dimensions organized hierarchically. Six criteria evaluate section metadata (citations, titles, effective dates), five assess definitional accuracy (term specificity, scope, cross-references), and eight examine rule unit semantics (action clarity, condition completeness, exception handling). The hierarchical organization is not merely taxonomic. It enforces a dependency-aware repair strategy where upstream components are verified and regenerated before downstream rule units are evaluated. This design principle recognizes that legal rules are semantically dependent on defined terms, which in turn depend on accurate section metadata. Repairing a rule that references an incorrectly extracted definition would be futile, so De Jure ensures definitional context reaches 90% average accuracy before the most demanding decomposition stage begins.

If any stage scores below the 90% threshold, the pipeline enters a bounded regeneration loop with a default retry budget of three attempts. The highest-scoring generation is retained, and ablation studies confirm this budget is both necessary and sufficient for convergence. This iterative refinement substitutes the traditional single-pass extraction paradigm with a self-correcting process that mimics expert legal review, where initial drafts are refined through targeted questioning.

Generalization and Downstream Utility

The authors evaluate De Jure across four different language models on three distinct regulatory corpora: the SEC Advisers Act for finance, HIPAA for healthcare, and the EU AI Act for AI governance. This selection tests the pipeline against structurally distinct regulatory styles, from the principle-based approach of European AI regulation to the prescriptive technical requirements of American healthcare privacy law.

Results indicate consistent, monotonic increases in extraction quality across iterations, with peak performance achieved within three judge-guided cycles. Notably, the pipeline maintains high performance across both open-source and closed-source models, suggesting that the structured evaluation criteria provide sufficient signal to guide improvement regardless of the underlying model architecture.

More compelling than the extraction metrics themselves is the downstream evaluation using compliance question answering via retrieval-augmented generation (RAG). When grounding responses in De Jure extracted rules, a judge LLM preferred these answers over prior work in 73.8% of cases at single-rule retrieval depth. This preference rises to 84.0% under broader retrieval contexts. These figures matter because they demonstrate that extraction fidelity translates directly into utility for end users. A structured rule set that captures conditions and exceptions accurately enables more precise retrieval than keyword-based or flat-text approaches, reducing the risk of compliance hallucinations where models generate plausible but legally incorrect interpretations.

Regulatory Alignment Beyond Preferences

The De Jure framework represents a conceptual shift from preference-based alignment toward obligation-based alignment. Constitutional AI and related approaches generate alignment principles from broad policy text, optimizing for helpfulness and harmlessness as judged by human or model preferences. Regulatory rules, however, are not preferences but constraints with logical structure. They contain defeasible conditions, scope limitations, and cross-references that require precise parsing rather than semantic approximation.

My analysis suggests that the De Jure approach exposes a deeper connection between software engineering and legal reasoning. The hierarchical decoupling strategy mirrors dependency management in complex codebases, where modules are compiled and tested in topological order. By treating legal definitions as header files upon which rule implementations depend, the pipeline achieves a form of semantic compilation that traditional information extraction techniques, which often treat documents as bag-of-words or flat sequences, cannot replicate.

However, the reliance on LLM-as-a-judge in high-stakes legal contexts introduces significant epistemological risks. Without human-annotated gold standards, the evaluation criteria themselves are enforced by the same class of models performing the extraction. While the multi-dimensional scoring rubric mitigates circularity by breaking evaluation into 19 specific checks, the system lacks external legal validation. A model might consistently score high on extraction completeness while systematically misinterpreting legal negations or scope modifiers. The 90% threshold provides a stopping criterion, but not a guarantee of legal correctness.

Furthermore, the pipeline assumes regulatory stability. Financial and healthcare regulations undergo frequent amendment, and the current architecture, while fingerprinting documents via SHA-256, does not explicitly model temporal validity or regulatory deltas. A rule extracted today may be superseded tomorrow, and the current schema does not appear to capture versioning relationships or sunset clauses with the same granularity applied to semantic decomposition.

Conclusion

De Jure demonstrates that explicit, interpretable evaluation criteria can substitute for costly human annotation in complex regulatory domains, offering a scalable path toward compliance-aware AI. The combination of structured schema extraction, hierarchical dependency management, and iterative self-refinement produces machine-readable rule sets that significantly improve downstream retrieval and reasoning tasks.

Looking forward, several questions remain open. How does the system handle jurisdictional conflicts where regulations from different authorities impose contradictory obligations? Can the extraction schema be extended to capture the argumentative structure of regulatory preambles, which often contain crucial interpretive guidance? And perhaps most critically, how can we introduce adversarial legal review into the evaluation loop, where expert attorneys deliberately attempt to find edge cases that break the extraction logic?

As AI systems increasingly operate in regulated environments, the ability to automatically compile legal text into executable constraints will become as fundamental as code compilation is to software engineering. De Jure offers a robust architectural foundation for this capability, though the path to fully automated regulatory compliance will require continued innovation in verification methods that extend beyond model self-assessment.