NOULI

Examples

Robotic picking exceptions

A picking robot continues one Nouli session through a pick that goes wrong: a failed grasp, an uncertain next move, a double pick, and a torn bag. Each sensor update is a new turn, and the robot uses the returned probabilities to decide when to ask a person.

The problem

A robotic picking arm handles most items without help. The cost and the risk sit in the exceptions: suction cups that seal on loose packaging, two items lifted together, damaged goods. Each exception is a decision made with partial information while the arm holds an item, and often a reason to call a remote operator, which costs time.

A decision layer for these moments needs memory of the whole pick so far, typed answers the controller can act on, and probabilities it can use to decide when not to act alone. When something goes wrong, the team also needs to replay exactly what the robot knew at each step.

Nouli makes these task-level decisions, one model call each. Motion planning, grasp execution, and safety-rated stops stay in the robot's real-time control stack. The item, station, order, and rules here are illustrative.

What we will do

We load the item's grasp history and the station rules as separate sessions, then start the pick as a blend of both. The robot adds each sensor update to that one session. We check an alternative outcome without saving it, and finish by blending the episode with the item history to update the item's grasp profile.

Before you start

Configure a provider and model, then run the commands in order in the same shell. Nouli reads credentials from your shell or the .env in your current directory. Each decision calls your configured provider and may cost money.

The outputs below were recorded with typesafe/jev-1.14.0. Your probabilities, session IDs, and answers may differ.

Set a session prefix for this walkthrough. Change it before running the walkthrough again, so the new sessions get unique names:

example=pick
show the steps as

Open the first session with nouli open catalog, then type each step's lines at the prompt. A question sends the turn.

01Load the item's grasp history

Load what the station knows about this item into its own session. Every pick of this SKU can start from it.

$ nouli open catalog
new session "catalog" · /help for commands

catalog › Illustrative item record: SKU 30418 is a 400 ml shampoo bottle in a clear polybag, weighing 0.45 kg. At this station this month the arm picked it 112 times. Suction on the flat label area succeeded first time in 81 of those picks. Most failures happened when the suction cup sealed on loose bag film instead of the bottle.
catalog › ?grasp: Which grasp should the arm prefer for this item? [Suction on the flat label area of the bottle, Suction on any flat bag surface, A two-finger pinch on the bottle neck]
  turn 1 · 1 fact · 1 question
  grasp       ▶ Suction on the flat label area … ███████████████████▊ 0.99
                Suction on any flat bag surface                       0.00
                A two-finger pinch on the bottl… ▎                    0.01
nouli --format text new "$example-catalog" \
  'Illustrative item record: SKU 30418 is a 400 ml shampoo bottle in a clear polybag, weighing 0.45 kg. At this station this month the arm picked it 112 times. Suction on the flat label area succeeded first time in 81 of those picks. Most failures happened when the suction cup sealed on loose bag film instead of the bottle.' \
  -q 'grasp: Which grasp should the arm prefer for this item? [Suction on the flat label area of the bottle,
    Suction on any flat bag surface,
    A two-finger pinch on the bottle neck]'

Output

  turn 1 · 1 fact · 1 question
  grasp       ▶ Suction on the flat label area … ███████████████████▊ 0.99
                Suction on any flat bag surface                       0.00
                A two-finger pinch on the bottl… ▎                    0.01
  typesafe/jev-1.14.0 · session 01a18790-5ea3-7201-8b8c-22cc17d76fd3 · turn 01a18790-5f47-7068-80f3-12779b29a857

Suction on the label is the clear first choice (0.99), matching 81 first-time successes in 112 picks.

02Load the station rules

Load the station rules into a second session. The first rule is about uncertainty: the robot's controller reads the probability Nouli returns and acts on its own only at 0.6 or above.

catalog › /new policy
  new session "policy"
policy › Illustrative station rules for the picking arm: the arm acts on its own only when its chosen action has a probability of at least 0.6; otherwise a remote operator chooses. After two failed grasps on one item, stop and request remote assistance. Never place a load whose measured weight differs from the expected item weight by more than 15%. A torn or open bag goes to the exception station, never into a shipping box.
policy › ?overweight: May the arm place a load that weighs 1.9 kg when one 0.45 kg item is expected?
  turn 1 · 1 fact · 1 question
  overweight    yes                              ▋                    0.03
nouli --format text new "$example-policy" \
  'Illustrative station rules for the picking arm: the arm acts on its own only when its chosen action has a probability of at least 0.6; otherwise a remote operator chooses. After two failed grasps on one item, stop and request remote assistance. Never place a load whose measured weight differs from the expected item weight by more than 15%. A torn or open bag goes to the exception station, never into a shipping box.' \
  -q 'overweight: May the arm place a load that weighs 1.9 kg when one 0.45 kg item is expected?'

Output

  turn 1 · 1 fact · 1 question
  overweight    yes                              ▋                    0.03
  typesafe/jev-1.14.0 · session 01a18790-5f55-76f2-b4c0-5c958b4f9ff3 · turn 01a18790-6026-75fa-a1b8-bcd142b5b19b

The model confirms that a 1.9 kg load can't be placed when one 0.45 kg item is expected (P(yes)=0.03). That rule matters three turns later.

03Start the pick

Start the pick as a blend of the item history and the station rules. From here the robot continues one session, adding each sensor update as a new turn.

policy › /blend catalog policy as pick
  new session "pick" from catalog + policy · no standing questions
pick › Task: pick one unit of SKU 30418 for order 4471 into box B-12. Tote camera: 6 items. The target is detected with confidence 0.82, lying label-down, partly under another polybagged item. The only flat surface visible on it is 3 square centimeters of loose bag film.
pick › ?action: What should the arm do first? [Nudge the covering item aside to expose the target's label, Attempt suction on the visible bag film, Request remote assistance now]
  turn 1 · 1 fact · 1 question
  action      ▶ Nudge the covering item aside t… ████████████████▋    0.83
                Attempt suction on the visible … ▊                    0.04
                Request remote assistance now    ██▋                  0.13
nouli --format text new "$example-pick" --from "$example-catalog" --from "$example-policy" \
  'Task: pick one unit of SKU 30418 for order 4471 into box B-12. Tote camera: 6 items. The target is detected with confidence 0.82, lying label-down, partly under another polybagged item. The only flat surface visible on it is 3 square centimeters of loose bag film.' \
  -q "action: What should the arm do first? [Nudge the covering item aside to expose the target's label,
    Attempt suction on the visible bag film,
    Request remote assistance now]"

Output

  turn 1 · 1 fact · 1 question
  action      ▶ Nudge the covering item aside t… ████████████████▋    0.83
                Attempt suction on the visible … ▊                    0.04
                Request remote assistance now    ██▋                  0.13
  typesafe/jev-1.14.0 · session 01a18790-603b-7042-93bf-5af9d742e8f6 · turn 01a18790-60c2-7542-8407-b4353c3886db

The model nudges the covering item aside to expose the label (0.83) instead of trying suction on loose film. At 0.83, the arm acts on its own.

04React to a failed grasp

The robot sends only what changed: the nudge worked, but the suction attempt failed when the film peeled. The camera scene, item history, and rules come from the session.

pick › Sensor update: the nudge worked and the target now lies label-up with a 6 square centimeter flat label area. Suction attempt 1 on the label lost vacuum at 2 cm of lift, because the bag film peeled away from the bottle. The loose film now hangs over the label area. The bottle neck is exposed and clear. That is one failed grasp.
pick › ?action: What should the arm do next? [Try a two-finger pinch on the bottle neck, Retry suction on the same spot, Stop and request remote assistance]
  turn 2 · 1 fact · 1 question · re-asked 1
  action      ▶ Try a two-finger pinch on the b… ██████████████▍      0.72
                Retry suction on the same spot   █████▎               0.26
                Stop and request remote assista… ▍                    0.02
nouli --format text add "$example-pick" \
  'Sensor update: the nudge worked and the target now lies label-up with a 6 square centimeter flat label area. Suction attempt 1 on the label lost vacuum at 2 cm of lift, because the bag film peeled away from the bottle. The loose film now hangs over the label area. The bottle neck is exposed and clear. That is one failed grasp.' \
  -q 'action: What should the arm do next? [Try a two-finger pinch on the bottle neck,
    Retry suction on the same spot,
    Stop and request remote assistance]'

Output

  turn 2 · 1 fact · 1 question · re-asked 1
  action      ▶ Try a two-finger pinch on the b… ██████████████▍      0.72
                Retry suction on the same spot   █████▎               0.26
                Stop and request remote assista… ▍                    0.02
  typesafe/jev-1.14.0 · session 01a18790-603b-7042-93bf-5af9d742e8f6 · turn 01a18790-61a9-7295-9b8b-7e8020bd3f6a

The model chooses the neck pinch (0.72) over retrying suction (0.26) or asking for help (0.02). The top answer is above 0.6, so the controller acts on its own. Retrying suction would aim at a label now covered by loose film.

05Check a clean lift without saving

Before saving the real outcome, check what the arm would do if the pinch lifted one bottle cleanly. whatif evaluates it without saving. --drop leaves the pick question out of this one turn.

This what-if asks a question of its own, so it runs from the shell:

nouli --format text whatif "pick" \
  'What-if: the arm pinches the bottle neck, and pinch attempt 2 lifts the target. The wrist force sensor reads 0.46 kg against an expected 0.45 kg, and the tote camera shows a single bottle in the gripper.' \
  -q 'load: What should the arm do with the load? [Lower the load back into the tote and request remote assistance,
    Place the load in box B-12,
    "Shake the load to check for anything attached, then place it"]' \
  --drop action
nouli --format text whatif "$example-pick" \
  'What-if: the arm pinches the bottle neck, and pinch attempt 2 lifts the target. The wrist force sensor reads 0.46 kg against an expected 0.45 kg, and the tote camera shows a single bottle in the gripper.' \
  -q 'load: What should the arm do with the load? [Lower the load back into the tote and request remote assistance,
    Place the load in box B-12,
    "Shake the load to check for anything attached, then place it"]' \
  --drop action

Output

  (not saved) · 1 fact · 1 question
  load          Lower the load back into the to… ▎                    0.01
              ▶ Place the load in box B-12       ██████████████████▍  0.92
                Shake the load to check for any… █▍                   0.07
  typesafe/jev-1.14.0

With one bottle at 0.46 kg, the model places it in the box (0.92). Nothing is saved, so this alternative isn't in the graph.

06React to a double pick

The real outcome: the neck pinch lifted two bottles stuck together. -q sets the question for the new turn, and --drop action stops the session from asking the earlier one.

pick › /drop action
  queued: drop action
pick › Sensor update: the arm pinched the bottle neck, and pinch attempt 2 lifted the target. The wrist force sensor reads 1.9 kg against an expected 0.45 kg. The tote camera shows a second bottle stuck to the target by an adhesive strip on the bags.
pick › ?load: What should the arm do with the load? [Lower the load back into the tote and request remote assistance, Place the load in box B-12 and let the packer remove the extra bottle, "Shake the load to separate the bottles, then place the target"]
  turn 3 · 1 fact · 1 question
  load        ▶ Lower the load back into the to… ████████████████████ 1.00
                Place the load in box B-12 and …                      0.00
                Shake the load to separate the …                      0.00
nouli --format text add "$example-pick" \
  'Sensor update: the arm pinched the bottle neck, and pinch attempt 2 lifted the target. The wrist force sensor reads 1.9 kg against an expected 0.45 kg. The tote camera shows a second bottle stuck to the target by an adhesive strip on the bags.' \
  -q 'load: What should the arm do with the load? [Lower the load back into the tote and request remote assistance,
    Place the load in box B-12 and let the packer remove the extra bottle,
    "Shake the load to separate the bottles, then place the target"]' \
  --drop action

Output

  turn 3 · 1 fact · 1 question
  load        ▶ Lower the load back into the to… ████████████████████ 1.00
                Place the load in box B-12 and …                      0.00
                Shake the load to separate the …                      0.00
  typesafe/jev-1.14.0 · session 01a18790-603b-7042-93bf-5af9d742e8f6 · turn 01a18790-62f2-7087-b24c-a423ec074079

The load weighs four times what one bottle should, and the station rules forbid placing it. The model lowers it back into the tote and asks for help (P=1). It doesn't shake the load or leave the problem for the packer.

07Take the remote operator's report

Add the remote operator's report as the next turn, again swapping the session's question for the new one.

pick › /drop load
  queued: drop load
pick › Remote assistance, 40 seconds: the operator separated the bottles by marking pinch points, and the arm now holds only the target (0.46 kg). The operator reports that the target'\''s bag is torn along one edge.
pick › ?destination: Where should the arm put the target? [Send it to the exception station, Place it in box B-12, Return it to the tote]
  turn 4 · 1 fact · 1 question
  destination ▶ Send it to the exception station ████████████████████ 1.00
                Place it in box B-12                                  0.00
                Return it to the tote                                 0.00
nouli --format text add "$example-pick" \
  'Remote assistance, 40 seconds: the operator separated the bottles by marking pinch points, and the arm now holds only the target (0.46 kg). The operator reports that the target'\''\'\'''\''s bag is torn along one edge.' \
  -q 'destination: Where should the arm put the target? [Send it to the exception station,
    Place it in box B-12,
    Return it to the tote]' \
  --drop load

Output

  turn 4 · 1 fact · 1 question
  destination ▶ Send it to the exception station ████████████████████ 1.00
                Place it in box B-12                                  0.00
                Return it to the tote                                 0.00
  typesafe/jev-1.14.0 · session 01a18790-603b-7042-93bf-5af9d742e8f6 · turn 01a18790-63dc-7318-b86b-93a1eeb6fa42

The operator separated the bottles, but the target's bag is torn, and torn bags go to the exception station (1.00). Returning it to the tote (0.00) would put a damaged item back into stock.

08Update the item's grasp profile

Blend the item history with the whole pick episode to decide how the arm should grasp this SKU next time. One robot's episode becomes station knowledge without changing the original item session.

pick › /blend catalog pick as catalog-review
  new session "catalog-review" from catalog + pick · no standing questions
catalog-review › Review the completed pick against the item'\''s history and decide how the arm should grasp this SKU next time.
catalog-review › ?profile: How should the grasp profile for SKU 30418 change? ["Prefer a neck pinch, and check the weight before placing", Keep suction on the label area as the first choice, Always request remote assistance for this item]
  turn 1 · 1 fact · 1 question
  profile     ▶ Prefer a neck pinch, and check … ████████████████▎    0.81
                Keep suction on the label area … ███▊                 0.19
                Always request remote assistanc…                      0.00
nouli --format text new "$example-catalog-review" --from "$example-catalog" --from "$example-pick" \
  'Review the completed pick against the item'\''\'\'''\''s history and decide how the arm should grasp this SKU next time.' \
  -q 'profile: How should the grasp profile for SKU 30418 change? ["Prefer a neck pinch, and check the weight before placing",
    Keep suction on the label area as the first choice,
    Always request remote assistance for this item]'

Output

  turn 1 · 1 fact · 1 question
  profile     ▶ Prefer a neck pinch, and check … ████████████████▎    0.81
                Keep suction on the label area … ███▊                 0.19
                Always request remote assistanc…                      0.00
  typesafe/jev-1.14.0 · session 01a18790-63f3-7046-ae13-418b20c0418f · turn 01a18790-6495-747f-b1a4-39c6168402d6

The model now prefers a neck pinch with a weight check (0.81) over keeping suction on the label (0.19). The record shows why: suction failed on loose film, and the pinch that followed lifted a second bottle, and a weight check catches that. Anyone reviewing grasp profiles can see the evidence behind both numbers.

What the graph keeps

The pick session holds the whole episode in order: the scene, the failed grasp, the double pick, and the operator's report. Every answer records the context it was based on, so a failure review can see that the robot asked for help at 1.00 once the weight was wrong, instead of placing the load. These commands make decisions only; they don't move an arm.