Interpretable Safety in Reinforcement Learning Through Geometric Policy Constraints

Introduction

Reinforcement learning has achieved remarkable success in game playing, robotics, and autonomous systems. Yet these successes mask a fundamental tension. The policies driving these agents are typically deep neural networks; functionally opaque matrices of parameters that resist human inspection. In safety critical domains, this opacity is unacceptable. We cannot deploy agents that might, during exploration, execute catastrophically unsafe actions simply because they have not yet learned to associate those actions with penalties.

The standard approach to safe reinforcement learning relies on Lagrangian relaxation or constrained policy optimization. These methods treat safety constraints as soft penalties in the reward function. While theoretically sound, they offer only asymptotic guarantees. The agent violates constraints during training, learning gradually which regions of the action space are forbidden. For physical systems interacting with humans, this trial and error approach to safety is untenable.

In "Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policies," Rietz et al. propose a fundamentally different architecture. Their constrained normalizing flow policies (CNFPs) embed safety constraints directly into the policy's representational structure, ensuring that unsafe actions are not merely discouraged but geometrically impossible to sample.

From Penalty Functions to Geometric Transformations

Traditional constrained reinforcement learning frames the problem as an optimization with instantaneous constraints c_k(s_t, a_t) ≤ ε_k. Standard algorithms like CPO or Lagrangian based methods solve this by modifying the objective function, adding penalty terms that increase when constraints are violated. The policy learns to avoid these penalties, but the action space remains fundamentally unconstrained during the learning process.

Rietz et al. exploit a crucial observation about instantaneous constraints. Each constraint partitions the action space into two regions: the feasible set A_{φ,k} and the violating set A_{ψ,k}. The intersection of all feasible sets, A_φ, defines a safe action space that varies by state. Rather than learning to avoid A_ψ, the authors propose to redefine the underlying Markov Decision Process entirely, replacing the original action space A with the safe subset A_φ.

They achieve this through normalizing flows, a class of generative models that learn complex distributions through invertible transformations. In a normalizing flow policy, an initial simple distribution (typically Gaussian) is passed through a sequence of bijective mappings to produce a complex policy distribution. The key innovation of CNFPs lies in constructing specific flow steps that correspond analytically to constraint satisfaction. Each transformation maps the distribution closer to the feasible region, with intermediate representations that align with specific physical or logical constraints.

This geometric approach differs sharply from projection methods, which map proposed actions into the safe set by solving optimization problems at each step. Such projections are computationally expensive and can introduce bias into policy gradients. By contrast, the normalizing flow approach maintains tractable likelihoods and allows standard policy gradient algorithms like Soft Actor