Training Lab · Reinforcement learning

Your robot learns by practice.
Here is what happens in every step.

Nobody programs the gait. The robot tries, the task scores every attempt, and an algorithm called PPO slowly shifts the robot’s behaviour toward what scored well. This page explains that loop, the choices kualia makes for you, and how to read what comes out of a run.

Teach a new trick Read the docs

PPO

algorithm in every Studio run

2,048

steps collected per update

episodes behind each chart point

deterministic evaluation episodes

1The basics

Observe, act, get a reward, repeat

Reinforcement learning has two parties. The agent is a neural network, the policy, that decides what the motors do. The environment is everything else: the MuJoCo robot, the world around it and the task’s rules. They take turns, many times per simulated second.

Every control step the policy reads an observation, outputs one command per actuator, MuJoCo advances the physics, and the task computes a reward.

Word	In the Studio
Observation	What the policy sees: joint angles and velocities, and the position of world objects relative to the robot.
Action	One continuous number per actuator. Walker2d has 6, Humanoid 17.
Reward	A number the task code computes every step: progress, staying upright, small costs for effort.
Episode	One attempt, from the start pose until the robot fails (terminated) or time runs out (truncated).
Return	The sum of an episode’s rewards. Training tries to make it as large as possible.

A policy that starts random mostly flails. Some attempts score a little better than others, and PPO makes the actions behind those attempts a bit more likely. After thousands of episodes the flailing turns into a gait, a kick or a balance.

Why the future counts less

PPO optimises a discounted return: a reward k steps ahead is weighted by γ^k, with γ = 0.99. The weight halves roughly every 70 steps, so the policy plans about a hundred control steps ahead. How long that is in seconds depends on the robot’s control rate: about 0.8 s for Walker2d at 125 Hz, 1.5 s for Humanoid at 67 Hz, 5 s for Ant at 20 Hz.

That is why dense, per-step rewards help so much: a reward that only arrives at the end of a long episode is almost invisible to the first few hundred steps of it.

Weight of a reward k control steps in the future, γ = 0.99.

2Exploration vs exploitation

To find better moves, the robot has to try worse ones

If the policy always did what it currently thinks is best, it would never discover anything better. So during training every action is sampled from a bell curve around the policy’s best guess.

For each actuator the policy outputs a mean μ. Training samples around it with a spread σ that starts at 1.0 and is learned; replays, evaluation and the exported file use μ alone.

Exploration is the noise: actions near the best guess, sometimes far from it. It is how a random policy stumbles on a first step forward.
Exploitation is the mean: doing what worked. As training goes on, PPO usually shrinks σ as it grows confident, and the behaviour becomes more consistent.
The training chart shows exploring episodes. Checkpoint replays, the evaluation episodes, the live policy and the ONNX export are deterministic: the same observation always gives the same action.
This is why a checkpoint often looks better than the chart suggests, and why a policy can look shaky during training but steady in evaluation.

No bonus for randomness

The Studio keeps PPO’s entropy bonus at zero: nothing rewards acting randomly for its own sake. Exploration comes only from the learned spread σ, which PPO narrows as the policy improves.

3The algorithm

Why the Studio trains with PPO

Proximal Policy Optimization is the workhorse of robot learning in simulation. The Studio runs it through Stable-Baselines3 with the library’s default settings, on a single environment per run. Three ideas make it work.

1 · Actor and critic

Two small networks read the same observation. The actor is the policy. The critic predicts how much return to expect from here, so each action can be judged against expectations instead of in absolute terms.

2 · Advantages with GAE

The advantage says how much better an action turned out than the critic expected. Generalised Advantage Estimation blends short and long look-aheads (λ = 0.95) into one steady signal.

3 · A clipped update

PPO compares how likely each action is under the new policy versus the old one (the ratio r). Beyond 1 ± 0.2 there is nothing more to gain, so one lucky batch cannot drag the policy off a cliff.

clipped objective PPO maximises unclipped r · A trust region, ε = 0.2

The clipped objective. Left: a better-than-expected action is made more likely, but only up to 20 %. Right: a worse one is made less likely, again at most 20 % per update. Inside the shaded band the update behaves like a plain policy gradient.

obs

normalised

Actor · the policy

64 tanh64 tanhμ per actuator+ learned σ

Critic · value function

64 tanh64 tanhV(s): expected return

Actor and critic are separate multilayer perceptrons (two hidden layers of 64, tanh). Only the actor is exported.

1Collect
2,048 steps with the current policy (n_steps)
2Score
advantages with GAE · γ 0.99 · λ 0.95
3Update
10 epochs × 32 minibatches of 64 = 320 gradient steps
4Discard
on-policy: the batch is thrown away, repeat

PPO is on-policy: every update uses fresh experience from the current policy. A 10K-step first try is about 5 of these iterations; a 2M-step Humanoid run is about 980.

Setting	Value	What it does
n_steps	2,048	steps collected with the current policy before each update
batch_size	64	minibatch size for each gradient step
n_epochs	10	passes over each collected batch
learning_rate	3 × 10⁻⁴	Adam step size, constant
gamma (γ)	0.99	discount: how much future reward counts
gae_lambda (λ)	0.95	bias/variance trade-off of the advantage estimate
clip_range (ε)	0.2	how far one update may move the policy
ent_coef	0.0	no extra entropy bonus; exploration comes from the learned σ
vf_coef	0.5	weight of the value-function loss
max_grad_norm	0.5	gradient clipping for stable updates
policy	MlpPolicy	separate 64-64 tanh networks for actor and critic

One algorithm, on purpose

Every Studio run uses PPO with these values, and continuing a run keeps them. When version 4 beats version 3, the difference comes from the task, not from a tweaked optimiser.

Other algorithms

The training service underneath is built on Stable-Baselines3 and can also run A2C, SAC, TD3 and DQN, and the ONNX exporter understands their policies. The Studio does not offer them: every Studio run is PPO.

4Scaling

Normalised observations and rewards

A torso height of 1.25 m and a joint spinning at 8 rad/s are different kinds of number. Neural networks learn much faster when every input has a similar scale, so training wraps the task in Stable-Baselines3’s VecNormalize.

raw values · very different scales

torso height

1.25 m

joint velocity

−8.3 rad/s

ball offset x

3.0 m

pole angle

0.02 rad

normalised · comparable, clipped ±10

torso height

joint velocity

ball offset x

pole angle

Illustrative. Each observation value becomes (x − running mean) / running std, clipped to ±10.

Observations: running mean and standard deviation per value, updated during training, clipped to ±10.
Rewards: scaled by a running estimate of the discounted return’s spread (γ = 0.99), clipped to ±10. This keeps the critic’s targets in a sane range whatever the units of your reward.
The chart stays honest: a Monitor wrapper sits inside the normalisation and records the raw episode return and length, in the task’s own units.
Evaluation freezes the statistics and reports raw rewards. The same frozen observation statistics are baked into the exported ONNX file, so on the robot you feed raw sensor values.

training, simplified

# What a Studio run boils down to (simplified)
from stable_baselines3 import PPO
from stable_baselines3.common.monitor import Monitor
from stable_baselines3.common.vec_env import DummyVecEnv, VecNormalize

env = DummyVecEnv([lambda: Monitor(TaskEnv())])        # Monitor records RAW episode returns
env = VecNormalize(env, norm_obs=True, norm_reward=True,
                   clip_obs=10.0, clip_reward=10.0, gamma=0.99)

model = PPO("MlpPolicy", env)                          # Stable-Baselines3 defaults
model.learn(total_timesteps=budget)                    # e.g. 10_000 for a first try

env.training = False; env.norm_reward = False          # freeze the statistics
# up to 30 evaluation episodes (seeds 0-29, 90 s budget after the first 10), deterministic=True, raw rewards, 95% Wilson interval on success

5Designing the experience

How alia turns a sentence into a task

The robot can only learn what the task pays for. alia writes that task from your brief: a world planner first decides which objects the trick needs, then alia works out the goal, a metric you could measure and the constraints, restates the trick in one sentence with a 3–5 step picture of a successful episode (shown on the Plan page as “What success looks like”), and implements it all as reward terms, success and termination rules, observations and curriculum knobs.

Walker2d walks up to the ball and kicks it into the goal

Goal

ball moves into the goal

Metric

ball within 0.3 m of the goal mouth

Constraint

no falling, no flailing

Reward terms

progress toward the goal, staying upright, small effort cost

Success

a measurable end state, read when the episode ends

Termination

failure: fell over, flipped, left the area

Truncation

time limit: max_steps control steps

Observations

joint angles and speeds, object positions

Curriculum

knobs for easier or harder starts

From a brief to the six cards of the training plan on the Plan page.

Reward shaping

A reward is normally a weighted sum of terms. Dense shaping, the default, pays for progress every step: distance closed to the ball, speed toward the goal, height gained. The robot gets a useful signal from its very first attempts. Goal-based rewards pay a clear bonus for finishing plus light shaping: cleaner behaviour, slower learning, because the bonus is rare early on.

forward velocity× 1.0
upright× 0.5
control cost− 0.001 Σa²
fall penalty− 10, once

Illustrative locomotion reward. Your task’s terms and weights are generated for your robot and brief, and shown on the Plan page.

Per-step rewards should stay roughly within ±10; alia’s rules clip them there and the Reward is sane check warns when random actions produce values outside that range.

Pitfalls alia and the review look for

What you see	Why it happens	The fix
The robot stands still and the reward looks fine	A survival bonus outweighs the progress terms: standing still collects it risk-free.	Lower the bonus or make it depend on progress; raise the progress weight.
The robot falls on purpose	Every step is net negative, so ending the episode early is the best strategy.	Make a normal upright step slightly positive; keep penalties smaller.
The ball flies sideways	Reward hacking: the task pays for ball speed in any direction.	Reward progress toward the goal instead of raw speed.
No progress in a short run	A goal-based reward on a hard trick: the bonus is never seen.	Switch to dense shaping or start with an easier stage.
Reward rises, success stays at 0 %	The reward and the success criterion disagree.	Align them, or loosen the criterion in the next version.

Success

Reported in info["is_success"] and read at the end of an episode. In kualia’s MuJoCo runtime an episode only counts as a success if it reached the time limit without failing and is_success() holds, so tasks don’t end on success; the robot holds the result until time runs out.

Termination vs truncation

Terminated means failure, for example the robot fell below half its start height or flipped over. Truncated means time ran out after max_steps. Every threshold must already hold at the start pose, or every episode ends at step one.

Curriculum

The starting difficulty (Easy by default) sets episode length and how forgiving the thresholds are. Curriculum knobs, such as a target distance, are plain constructor arguments of the task, and the plan lists curriculum stages. Making it harder is a new version: “move the ball to 2 m”, “tighten success to 0.2 m”.

6Guard rails

Checks that make sure there is something to learn

Before a single paid training step, every task version runs 13 checks in an isolated sandbox. Three of them are about learning itself (reward sanity and the two below); training cannot start until all 13 pass.

action_effect

Learnability estimate

Over five seeds, a 50-step rollout with random actions must earn a different return than one with zero actions. If nothing the robot does changes the reward (a constant bonus, a term reading the wrong variable, a reward that is a function of time), no algorithm can learn from it.

episode_length

Episodes survive reset

Five random-action episodes (up to 300 steps) must last more than two steps on average. The classic broken task terminates at the start pose, for example a height test on a value that is zero at reset, and would burn the whole budget on one-step episodes.

Automated checks — The checks on the Plan page. alia repairs failures itself, up to four attempts when the plan is first built and up to three after every chat change, always continuing from the best-scoring version.

Reward is saneFinite at every step
Deterministic resetSame seed, same start
No unstable physicsNo NaN, bounded state, episodes end
Objects start clearWorld objects start away from the robot and each other
…plus syntax, security scan, import, reset, step, observation and action space checks.

7Watching it learn

Reading the curve, then judging fairly

While training runs, the Studio adds a point to the learning chart about every 5 % of the budget. Each point summarises the last 50 finished training episodes.

Illustrative chart. The line is the mean episode reward over the last 50 episodes; the band spans their minimum and maximum.

Metric	Read it as
Mean episode reward	Is the robot collecting more of what the task pays for?
Min / max band	How consistent the attempts are. A wide band early is normal.
Success rate	Share of the last 50 episodes that met the success criterion.
Episode length	Longer usually means fewer falls; very short means early termination.
Throughput	Steps per second, used to estimate the remaining time.

The Training screen: live simulation, the checkpoint film strip, learning progress and the training log.

Checkpoints

The untrained policy is recorded first, then at 25, 50 and 75 % of the budget, then the final policy. Each checkpoint plays one deterministic episode that is saved as a video and as a MuJoCo trajectory for the 3D replay. From the 25 % checkpoint on, the newest checkpoint policy can run live in your browser while training continues.

Deterministic evaluation

After training, the final policy plays 5 episodes with seeds 0–4, deterministic actions and frozen normalisation, and the raw rewards are recorded. An episode that lasts at least 95 % of max_steps counts as a full episode. These are the numbers on the run’s results page.

8Iterate

The AI review, versions, and how long it takes

Reinforcement learning is a loop of experiments: run, look, change one thing, run again. kualia keeps every run of every task version so that loop stays honest.

The first time you open a run’s results, an AI review reads the training curve, the evaluation numbers, the recorded episodes and the task code. It writes a short summary of what the robot does, 2–4 observations with a severity, and 3–6 proposed changes tagged reward, termination, observation, curriculum or hyperparameters, each with the current value, the proposed value and why. One click turns its instruction into the next version, which runs the 13 checks again.

Runs & versions — Every run stays pinned to the version it trained on. Compare two runs, train a version again, keep training the best run, or make an older version current.

AI review — The review proposes concrete changes for the next version.

How many steps a robot typically needs

The Studio’s “Learn it” budget per robot, with what it means for PPO: the number of 2,048-step iterations, and how much simulated time the robot practises at its control rate. These are rough guides for a real skill, not guarantees. The realistic Menagerie robots need far more practice than the classics.

Robot	Learn it	PPO iterations	Control rate	Simulated practice
InvertedPendulum	30K	≈ 15	25 Hz	≈ 20 min
Reacher	100K	≈ 49	50 Hz	≈ 33 min
Hopper	300K	≈ 147	125 Hz	≈ 40 min
Walker2d	500K	≈ 245	125 Hz	≈ 1.1 h
HalfCheetah	300K	≈ 147	20 Hz	≈ 4.2 h
Ant	1M	≈ 489	20 Hz	≈ 13.9 h
Humanoid	2M	≈ 977	66.7 Hz	≈ 8.3 h
Dynamixel2R	50K	≈ 25	Hz	≈ NaN h
LowCostArm	200K	≈ 98	Hz	≈ NaN h
SO101	300K	≈ 147	Hz	≈ NaN h
Crazyflie	300K	≈ 147	Hz	≈ NaN h
Duck	2M	≈ 977	Hz	≈ NaN h
Toddlerbot	5M	≈ 2,442	Hz	≈ NaN h
OP3	5M	≈ 2,442	50 Hz	≈ 27.8 h
Panda	500K	≈ 245	50 Hz	≈ 2.8 h
Go2	3M	≈ 1,465	50 Hz	≈ 16.7 h
Spot	3M	≈ 1,465	50 Hz	≈ 16.7 h
G1	10M	≈ 4,883	50 Hz	≈ 55.6 h

Uploaded robots default to 300K steps. A 10K-step first try is still worth it on a big robot: it shows whether episodes survive, whether the reward rises and whether the behaviour points the right way, before you pay for a long run.

Questions

Why PPO and not SAC or another algorithm?

PPO is forgiving: it works with the same settings on a cart-pole and on a humanoid, its clipped update rarely collapses, and it handles the continuous motor commands of every robot in the Studio. Using one algorithm with one set of defaults also keeps runs comparable, so when a new version does better you know the task changed, not the optimiser.

Can I change the hyperparameters?

Not in the Studio. Every run uses the PPO defaults listed above. In practice the levers that decide whether a robot learns a trick are the reward, the episode rules, the world and the number of steps, and those you change with alia.

Why is the checkpoint reward higher than the chart?

The chart averages episodes of the exploring policy, which samples noisy actions on purpose. Checkpoints and the evaluation run the deterministic policy (the mean action), which usually scores higher.

Why is the success rate different on the chart and on the results page?

The chart is the share of the last 50 training episodes that succeeded. The results page uses up to 30 deterministic evaluation episodes and shows a 95% interval for the success rate.

All 13 checks pass. Will the robot learn the trick?

Not necessarily. The checks prove the task runs, is stable and can be learned in principle: actions change the return and episodes survive the reset. Whether the reward actually describes the behaviour you want is something only training, the replays and the AI review can show.

How many steps does my robot need?

Mostly a matter of the robot: roughly 30K for a cart-pole, 500K for a planar walker, 2M for the classic humanoid, and for the realistic robots 500K (Panda), 3M (Go2, Spot) up to 10M (G1). These are orders of magnitude, and more steps cost more ($0.10 per 1,000). Start with a 10K-step first try to check the task, then spend the robot’s “Learn it” budget.

Put the loop to work.

Describe a trick, check the plan, and run a 10K-step first try.

Open Studio How it works

Next · Training LabSimulationMuJoCo physics, robots, worlds and the browser simulator.Also in the labExport the experienceThe ONNX policy, on-robot inference and sim-to-real.

Your robot learns by practice.Here is what happens in every step.

Observe, act, get a reward, repeat

Why the future counts less

To find better moves, the robot has to try worse ones

Why the Studio trains with PPO

1 · Actor and critic

2 · Advantages with GAE

3 · A clipped update

Normalised observations and rewards

How alia turns a sentence into a task

Reward shaping

Pitfalls alia and the review look for

Success

Termination vs truncation

Curriculum

Checks that make sure there is something to learn

Learnability estimate

Episodes survive reset

Reading the curve, then judging fairly

Checkpoints

Deterministic evaluation

The AI review, versions, and how long it takes

How many steps a robot typically needs

Questions

Put the loop to work.

Your robot learns by practice.
Here is what happens in every step.