Part 1 is the build: the five-stage loop implemented as a real pipeline, with two tasks carried around it on one GPU. This part is the technical reading of those runs. The loop kept its shape but changed its insides: how a policy learns rewires three of the five stages, and that fork is the first real decision when you build a training center. The rest is the mechanism behind that, grounded in the two tasks, and the questions the build left open.
The two methods I ran
I ran two common learning methods, one for each task.
Reinforcement learning (RL) learns by trial and error against a reward. The policy acts, the environment returns a scalar reward, and an algorithm (PPO, in my Cartpole run, via rsl_rl) updates the policy toward higher-reward behavior. My PPO setup used no demonstrations, so the policy had to interact with the simulator during training to find out what worked. Other RL setups can use demonstrations or offline data; this one did not.
Imitation learning (IL), specifically behavior cloning (BC), learns by copying. The policy is given demonstrations — recorded (observation, action) pairs — and trained with ordinary supervised learning to reproduce the action given the observation (robomimic’s bc_rnn_low_dim, in my Franka run). The correct actions are already in the dataset, so training is offline pattern-fitting over a fixed file, with no environment interaction while it learns.
Two everyday versions: RL is learning to ride a bike — nobody can hand you balance, so you wobble, fall, and adjust by feel. Imitation is learning a dish by watching someone cook it a few times and reproducing it, no invention required.
The distinction that makes the rest follow: RL is online (it needs the physics running to generate its own training signal), BC is offline (its training signal is already on disk). Once you hold that, the same five stages take on two shapes:
| Stage | Reinforcement learning (Cartpole) | Imitation / BC (Franka) | Is the sim running? |
|---|---|---|---|
| Capture | Skipped — a reward function, no demonstrations | 10 seed human demos (HDF5) | RL: n/a · IL: no (downloaded) |
| Simulate & Synthesize | Env config only; instantiated at Train | Mimic expands 10 → 1,000 demos (~46 min) | IL: heavily |
| Train | PPO inside the sim (4,096 environments) | Supervised BC, offline on the dataset | RL: yes · IL: no |
| Validate | Roll the policy out in sim | Roll the policy out in sim | both: yes |
| Deploy | ONNX export + latency benchmark | Inference only | both: no |
Where the sim’s role flips
Three things change between the two methods.
First, where the simulator runs. RL needs the sim during Train, because generating the training signal means acting in physics. BC uses the sim to manufacture data (Mimic) and again to validate, but the BC training step itself touches no physics — it’s number-crunching over an HDF5 file.
Second, whether Capture exists. RL needs no demonstrations, so Capture is honestly skipped (skipped-rl). BC is nothing but demonstrations, so Capture is the raw material and everything downstream is bounded by it.
Third, which stage is expensive. For RL the cost is in Train (the sim runs for every step of learning). For BC the cost moves upstream to Simulate and Synthesize — in my run, Mimic spent 46 minutes turning 10 demos into 1,000, and the supervised training that followed was comparatively quick.
Why I used imitation for this manipulation task
In my setup, RL avoided demonstrations but needed a reward, and the reward was the awkward part. A reward for “keep the pole up” is a one-liner. A reward for “stack these cubes” can become brittle: I would have to specify what counts as a good grasp, lift, and placement, and each term creates another way to score without completing the task. For this contact-rich, multi-step example, demonstrating success was easier than scoring every part of it, so I used imitation and shifted the simulator from training arena to data factory. That is a task choice, not a claim that manipulation always belongs to imitation learning.
The Franka training being offline follows from one question: do you already have the correct actions? For BC you do — they’re in the demonstrations — so training is fitting a function to (observation → action) pairs that already exist, with nothing to simulate. For RL you don’t, so the policy has to discover them by acting and checking the reward, which requires physics.
And Mimic turning 10 demonstrations into 1,000 is a legitimate technique, not a shortcut. Robot learning does not have an internet-scale supply of physical demonstrations, so one practical move is to expand a small seed set across randomized conditions. The ten seed trajectories were human demonstrations supplied with the task; Mimic generated simulated variations of them. On my box, the run with 50 generated demos reached about 2%, while the run with 1,000 reached 72%, from the same ten seeds. Because I changed the dataset size across two runs without controlled repeats, I treat that as a strong clue about data variety, not a measured law.
Which method needs which parts of the loop
Different jobs put the weight on different stages:
| The job | Method | Where the work lands |
|---|---|---|
| Balancing, locomotion, walking, flight | Reinforcement learning | Train — a reward is cheap to write, exploration is safe and fast in sim |
| Stacking, insertion, pouring, folding | Imitation | Capture + synthesis — the task is easy to show, hard to score |
| Assembly, long-horizon, sparse-reward | Hybrid (BC warm-start + RL fine-tune) | Both — demos bootstrap exploration, reward polishes the last mile |
A useful starting question is whether the task is easier to score or to show. A clean reward made RL a good fit for Cartpole; demonstrations made imitation a simpler first fit for the cube stack. Hybrid, offline-RL, and demonstration-assisted methods blur that line, so it is a starting point rather than a taxonomy.
This is also where the article’s “own the middle, rent the edges” claim became concrete, with one correction. The same pipeline and report contract carried both tasks even though Simulate and Train did different work. That reuse is real in this build. Whether the middle is the durable thing worth owning — technically or commercially — is still open, because the seed data coming through Capture may matter more than the plumbing around it.
What I haven’t tested yet
Building the loop established the mechanics. It also left a handful of things where I have an opinion but haven’t done the run to back it up — so I’ll mark each one as untested by me, not settled.
Whether a good score in sim means it’s safe on a real robot. The Franka policy stacks cubes 72% of the time in simulation. More or better demonstrations might improve that number, and a stronger method might too. But whatever the score, I have no way to turn it into “safe to run on a real line,” so every run reports safety_certified: false. Closing it would take safety and compliance testing on hardware, which I did not build. The sim score shows that this policy can sometimes do the task in this environment; it does not establish real-world safety.
Whether owning the middle is the right bet. In Part 1 I argued for owning the reusable middle (Simulate, Train, Validate) and renting the edges. But the biggest lever on my Franka result was the quality of the ten seed demos — which argues the opposite: that the data coming in through Capture is the real moat. I didn’t run the experiment that would settle which matters more. My working answer is “rent the capture rig, guard the captured data” — but that’s a hunch until I’ve actually run the comparison.
Whether the flywheel actually compounds. The return belt works mechanically — Deploy writes next_run, the next run reads it. What I haven’t done is run it a few times in a row and watch the success rate climb as fresh data flows back. To show it, I’d deploy, collect the rollouts, add them to the dataset, retrain, and repeat. I think it compounds; I just haven’t watched it happen.
Whether any of this holds up on a real robot. Everything here is a simulation number. on_robot is false throughout — there’s no physical arm — so 72% in sim is not 72% in the real world. Testing it would take a real Franka and the sim-to-real work that comes with it, and I’ve done neither.
Whether a stronger method beats plain behavior cloning. This behavior-cloning run plateaued near 70%. I did not isolate whether that came from the method, the data coverage, the training setup, or run variance. A useful next comparison would warm-start from the demonstrations and fine-tune with reinforcement learning, or post-train a larger model on the same data. I expect one of those to improve the result, but I have not run it.
The parts of the map the runs did convince me of: the middle is real and reusable, the return belt works in code, and the learning method is a fork that reshapes the loop. The few just above are the ones I haven’t tested yet. As in the original piece, I see all of this from the NVIDIA seat, so weight the defaults accordingly and treat the open alternatives as equally valid starting points. If you’ve built one of these for real, I’d like to compare notes — especially on the middle-versus-moat bet, the one I’m least sure I’ve got right.
Disclaimer: The views and opinions expressed in this account are those of my own and do not represent those of my employer, NVIDIA.