← Projects

What Testing the Robot Training Loop Taught Me

Part 1 is the build: the five-stage loop implemented as a real pipeline, with two tasks carried around it on one GPU. This part is the technical reading of those runs. The loop kept its shape but changed its insides: how a policy learns rewires three of the five stages, and that fork is the first real decision when you build a training center. The rest is the mechanism behind that, grounded in the two tasks, and the questions the build left open.

The two methods I ran

I ran two common learning methods, one for each task.

Reinforcement learning (RL) learns by trial and error against a reward. The policy acts, the environment returns a scalar reward, and an algorithm (PPO, in my Cartpole run, via rsl_rl) updates the policy toward higher-reward behavior. My PPO setup used no demonstrations, so the policy had to interact with the simulator during training to find out what worked. Other RL setups can use demonstrations or offline data; this one did not.

Imitation learning (IL), specifically behavior cloning (BC), learns by copying. The policy is given demonstrations — recorded (observation, action) pairs — and trained with ordinary supervised learning to reproduce the action given the observation (robomimic’s bc_rnn_low_dim, in my Franka run). The correct actions are already in the dataset, so training is offline pattern-fitting over a fixed file, with no environment interaction while it learns.

Two everyday versions: RL is learning to ride a bike — nobody can hand you balance, so you wobble, fall, and adjust by feel. Imitation is learning a dish by watching someone cook it a few times and reproducing it, no invention required.

The distinction that makes the rest follow: RL is online (it needs the physics running to generate its own training signal), BC is offline (its training signal is already on disk). Once you hold that, the same five stages take on two shapes:

StageReinforcement learning (Cartpole)Imitation / BC (Franka)Is the sim running?
CaptureSkipped — a reward function, no demonstrations10 seed human demos (HDF5)RL: n/a · IL: no (downloaded)
Simulate & SynthesizeEnv config only; instantiated at TrainMimic expands 10 → 1,000 demos (~46 min)IL: heavily
TrainPPO inside the sim (4,096 environments)Supervised BC, offline on the datasetRL: yes · IL: no
ValidateRoll the policy out in simRoll the policy out in simboth: yes
DeployONNX export + latency benchmarkInference onlyboth: no
The full browser dashboard after a completed real Cartpole run. The configuration controls remain visible above a three-minute run summary. Capture reads Skipped RL, Simulate takes effectively no time, Train shows a reward of 4.94 and the rising reward curve, Validate reports 32 episodes, and Deploy reports 0.009 milliseconds p50 latency.
The reinforcement-learning shape in the actual interface. Cartpole skips Capture, treats Simulate as configuration, and spends nearly all of its three minutes in Train. The completed view also keeps the evidence attached to the stages: reward curve, evaluation count, policy video, and deployment latency.

Where the sim’s role flips

Three things change between the two methods.

First, where the simulator runs. RL needs the sim during Train, because generating the training signal means acting in physics. BC uses the sim to manufacture data (Mimic) and again to validate, but the BC training step itself touches no physics — it’s number-crunching over an HDF5 file.

Second, whether Capture exists. RL needs no demonstrations, so Capture is honestly skipped (skipped-rl). BC is nothing but demonstrations, so Capture is the raw material and everything downstream is bounded by it.

Third, which stage is expensive. For RL the cost is in Train (the sim runs for every step of learning). For BC the cost moves upstream to Simulate and Synthesize — in my run, Mimic spent 46 minutes turning 10 demos into 1,000, and the supervised training that followed was comparatively quick.

A detailed dashboard view of the completed Franka imitation-learning run. Above the time bar, all five stages are done and the return belt points back to Capture. Expanded cards below show 10 seed demonstrations in Capture, 1,000 generated demonstrations from Isaac Lab Mimic in Simulate and Synthesize, behavior cloning in Train, and 72 percent success over 50 episodes in Validate.
The imitation-learning shape, with the handoffs exposed. Capture starts with ten demonstrations; Mimic expands them to 1,000; Train fits behavior cloning offline; Validate returns to simulation and reaches 72% over 50 episodes. The expanded cards are useful here because they show what crossed each stage boundary, not just that the stage completed.
A grid of the five stages (Capture, Simulate, Train, Validate, Deploy) with two rows: reinforcement learning (Cartpole) on top, imitation / BC (Franka) below. Teal cells mark where the simulator is running: for RL that's Train and Validate, for imitation it's Simulate and Validate. An amber outline marks each method's slow stage: Train (~2:27) for RL, Simulate (~46 min) for imitation. RL's Capture is a dashed 'skipped' box; imitation's Capture is a real '10 seed demos' box. RL's Train runs in the sim; imitation's Train is offline with no sim. Validate and Deploy are identical for both.
The same five stages, two shapes. Teal marks where the simulator is actually running; the amber outline marks each method's slow stage. Both move: reinforcement learning runs the sim during Train (its slow stage), while imitation runs it during Simulate to generate data (its slow stage) and trains offline. Capture is skipped for RL and real for imitation; Validate and Deploy are identical.

Why I used imitation for this manipulation task

In my setup, RL avoided demonstrations but needed a reward, and the reward was the awkward part. A reward for “keep the pole up” is a one-liner. A reward for “stack these cubes” can become brittle: I would have to specify what counts as a good grasp, lift, and placement, and each term creates another way to score without completing the task. For this contact-rich, multi-step example, demonstrating success was easier than scoring every part of it, so I used imitation and shifted the simulator from training arena to data factory. That is a task choice, not a claim that manipulation always belongs to imitation learning.

The Franka training being offline follows from one question: do you already have the correct actions? For BC you do — they’re in the demonstrations — so training is fitting a function to (observation → action) pairs that already exist, with nothing to simulate. For RL you don’t, so the policy has to discover them by acting and checking the reward, which requires physics.

And Mimic turning 10 demonstrations into 1,000 is a legitimate technique, not a shortcut. Robot learning does not have an internet-scale supply of physical demonstrations, so one practical move is to expand a small seed set across randomized conditions. The ten seed trajectories were human demonstrations supplied with the task; Mimic generated simulated variations of them. On my box, the run with 50 generated demos reached about 2%, while the run with 1,000 reached 72%, from the same ten seeds. Because I changed the dataset size across two runs without controlled repeats, I treat that as a strong clue about data variety, not a measured law.

Which method needs which parts of the loop

Different jobs put the weight on different stages:

The jobMethodWhere the work lands
Balancing, locomotion, walking, flightReinforcement learningTrain — a reward is cheap to write, exploration is safe and fast in sim
Stacking, insertion, pouring, foldingImitationCapture + synthesis — the task is easy to show, hard to score
Assembly, long-horizon, sparse-rewardHybrid (BC warm-start + RL fine-tune)Both — demos bootstrap exploration, reward polishes the last mile

A useful starting question is whether the task is easier to score or to show. A clean reward made RL a good fit for Cartpole; demonstrations made imitation a simpler first fit for the cube stack. Hybrid, offline-RL, and demonstration-assisted methods blur that line, so it is a starting point rather than a taxonomy.

This is also where the article’s “own the middle, rent the edges” claim became concrete, with one correction. The same pipeline and report contract carried both tasks even though Simulate and Train did different work. That reuse is real in this build. Whether the middle is the durable thing worth owning — technically or commercially — is still open, because the seed data coming through Capture may matter more than the plumbing around it.

What I haven’t tested yet

Building the loop established the mechanics. It also left a handful of things where I have an opinion but haven’t done the run to back it up — so I’ll mark each one as untested by me, not settled.

Whether a good score in sim means it’s safe on a real robot. The Franka policy stacks cubes 72% of the time in simulation. More or better demonstrations might improve that number, and a stronger method might too. But whatever the score, I have no way to turn it into “safe to run on a real line,” so every run reports safety_certified: false. Closing it would take safety and compliance testing on hardware, which I did not build. The sim score shows that this policy can sometimes do the task in this environment; it does not establish real-world safety.

Whether owning the middle is the right bet. In Part 1 I argued for owning the reusable middle (Simulate, Train, Validate) and renting the edges. But the biggest lever on my Franka result was the quality of the ten seed demos — which argues the opposite: that the data coming in through Capture is the real moat. I didn’t run the experiment that would settle which matters more. My working answer is “rent the capture rig, guard the captured data” — but that’s a hunch until I’ve actually run the comparison.

Whether the flywheel actually compounds. The return belt works mechanically — Deploy writes next_run, the next run reads it. What I haven’t done is run it a few times in a row and watch the success rate climb as fresh data flows back. To show it, I’d deploy, collect the rollouts, add them to the dataset, retrain, and repeat. I think it compounds; I just haven’t watched it happen.

Whether any of this holds up on a real robot. Everything here is a simulation number. on_robot is false throughout — there’s no physical arm — so 72% in sim is not 72% in the real world. Testing it would take a real Franka and the sim-to-real work that comes with it, and I’ve done neither.

Whether a stronger method beats plain behavior cloning. This behavior-cloning run plateaued near 70%. I did not isolate whether that came from the method, the data coverage, the training setup, or run variance. A useful next comparison would warm-start from the demonstrations and fine-tune with reinforcement learning, or post-train a larger model on the same data. I expect one of those to improve the result, but I have not run it.

The parts of the map the runs did convince me of: the middle is real and reusable, the return belt works in code, and the learning method is a fork that reshapes the loop. The few just above are the ones I haven’t tested yet. As in the original piece, I see all of this from the NVIDIA seat, so weight the defaults accordingly and treat the open alternatives as equally valid starting points. If you’ve built one of these for real, I’d like to compare notes — especially on the middle-versus-moat bet, the one I’m least sure I’ve got right.

Disclaimer: The views and opinions expressed in this account are those of my own and do not represent those of my employer, NVIDIA.

← All projects