Metadata-Version: 2.4
Name: videosdk-teleop
Version: 0.3.0
Summary: Robot teleoperation over WebRTC: a lerobot transport with frame-accurate observation/action correlation and dataset recording
Author: VideoSDK
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/zujonow/videosdk-teleops
Project-URL: Source, https://github.com/zujonow/videosdk-teleops
Project-URL: Issues, https://github.com/zujonow/videosdk-teleops/issues
Keywords: robotics,teleoperation,webrtc,lerobot,so-101,imitation-learning,ros2
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Networking
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS :: MacOS X
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: transport
Requires-Dist: videosdk; extra == "transport"
Provides-Extra: camera
Requires-Dist: vsaiortc; extra == "camera"
Requires-Dist: numpy; extra == "camera"
Requires-Dist: av; extra == "camera"
Requires-Dist: pillow; extra == "camera"
Provides-Extra: so101
Requires-Dist: feetech-servo-sdk; extra == "so101"
Provides-Extra: export
Requires-Dist: lerobot; extra == "export"
Requires-Dist: pillow; extra == "export"
Requires-Dist: numpy; extra == "export"
Provides-Extra: lerobot
Requires-Dist: lerobot; extra == "lerobot"
Requires-Dist: videosdk; extra == "lerobot"
Requires-Dist: vsaiortc; extra == "lerobot"
Requires-Dist: numpy; extra == "lerobot"
Requires-Dist: av; extra == "lerobot"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Dynamic: license-file

# videosdk-teleop

Robot teleoperation over WebRTC. A **transport for lerobot**, with a
first-class link between **what the operator saw** and **what the robot did**
— and a recording that keeps it, all the way to a LeRobot dataset.

```
remote operator ─► video observations ─► observation/action correlation
      ─► WebRTC teleoperation ─► safety-controlled robot
      ─► demonstration recording ─► LeRobot dataset ─► policy
```

Your arm stays an ordinary lerobot `Robot`; your leader stays an ordinary
lerobot `Teleoperator`. The network is substituted at a seam lerobot already
has, so `lerobot-record`, policy servers and the dataset tooling work
unchanged, and any arm lerobot already supports is already supported here.

```python
# on the machine holding the arm
follower = VideosdkFollower(cfg, robot=SafeRobot(arm, SafetyConfig(slew=12.0)))
for _ in pace(30):
    follower.send_feedback(robot.get_observation())
    if action := follower.get_action():
        robot.send_action(action)
```

---

## Install

```bash
pip install videosdk-teleop[lerobot]     # the lerobot plugin + transport
pip install videosdk-teleop[transport,camera,so101]   # without lerobot
pip install -e .                         # from a checkout
```

The base package has **no dependencies at all**: the protocol, the safety
layer and the whole test suite run on a bare interpreter, which is what lets
CI and a laptop exercise the SDK with no robot and no WebRTC stack. Everything
real is a per-role extra.

## Quick start, no hardware

```bash
pip install -e .

python examples/protocol_demo.py --gates          # the wire format, live
python examples/remote_teleop.py --ticks 100      # full chain, loopback
python examples/record_episode.py --out ./sessions --episodes 2
python examples/inspect_session.py sessions/*  --trace 3
python examples/export_lerobot.py sessions/*  --dry-run
python examples/print_stats.py --demo
```

`remote_teleop.py` runs a real follower against a real leader through an
in-process transport: real gates, real clamps, real observation log. Only the
network is swapped out.

## On the rig

```bash
python -m videosdk_teleop.devices                 # stable by-path handles
python examples/camera_test.py --wrist cam_a --front cam_b
python examples/camera_test.py --servo-probe /dev/ttyACM0

export VIDEOSDK_TOKEN=... MEETING_ID=xxxx-xxxx-xxxx
cd examples/lerobot/finalExample
python follower.py      # on the machine with the arm
python leader.py        # on the machine with the leader arm
python record.py        # anywhere: watches only, cannot move the arm
```

Nothing moves until an operator claims control, and `SafeRobot` holds the arm
the moment they stop sending. That is deliberate.

More under `examples/lerobot/`: `observer.py` (watch a session),
`my_arm.py` (a robot lerobot has never heard of), `custom_robot.py` /
`custom_teleoperator.py` (driving it). The v1 adapter examples
(`so101_follower.py`, `so101_leader.py`) still work.

## On ROS 2

Same two arms, same meeting, one node per side — each owning both its servo bus
and the SDK runtime, so **no ROS sits in the control path**. ROS 2 is the
operator and observability surface: topics to `ros2 bag record`, services to
call, params and launch files to configure. Full detail in [ROS2.md](ROS2.md).

Install into the interpreter ROS 2 itself runs on, not whichever one your shell
points at — on Ubuntu 24.04 that needs `--break-system-packages` (it still only
writes to `~/.local`):

```bash
/usr/bin/python3 -m pip install --user --break-system-packages \
    -e '.[transport,camera,so101]'
```

Then build the ament package once:

```bash
ln -s "$(pwd)/ros2_ws/src/videosdk_teleop_ros" ~/ros2_ws/src/
cd ~/ros2_ws && colcon build --symlink-install && source install/setup.bash
ros2 pkg executables videosdk_teleop_ros     # expect leader_node, follower_node
```

Now **three terminals**. `ros2 launch` holds the foreground until Ctrl-C, so they
cannot share one — and each starts from a fresh shell that has none of the
environment:

```bash
# run these in EVERY terminal
source /opt/ros/jazzy/setup.bash
source ~/ros2_ws/install/setup.bash
export VIDEOSDK_MEETING_ID=abcd-efgh-ijkl    # both ends MUST join the same room
export VIDEOSDK_TOKEN=...                    # keep it here, not in the YAML
```

```bash
# 1 — follower, the arm that moves
ros2 launch videosdk_teleop_ros follower.launch.py \
    port:=/dev/serial/by-id/usb-... robot_id:=my_follower_arm \
    cameras:=cam_a,cam_b labels:=wrist,front

# 2 — leader, the arm you backdrive
ros2 launch videosdk_teleop_ros leader.launch.py \
    port:=/dev/serial/by-id/usb-... robot_id:=my_leader_arm

# 3 — look before you arm, then take control
ros2 topic echo /leader_node/joint_states --once
ros2 topic echo /leader_node/remote/follower_states --once   # same units
ros2 service call /leader_node/enable std_srvs/srv/SetBool "{data: true}"
```

Both nodes print a **schema hash** at startup and they must match. That is the
check that catches the two ends disagreeing about units, and reading one log
line is cheaper than watching an arm move wrongly.

Nothing moves until that `enable` call. Release it with `{data: false}`, or stop
the arm outright:

```bash
ros2 service call /follower_node/estop std_srvs/srv/Trigger        # latched
ros2 service call /follower_node/clear_estop std_srvs/srv/Trigger  # only way out
```

Watch what it is doing:

```bash
ros2 topic hz   /follower_node/joint_states      # ~10 Hz
ros2 topic echo /follower_node/applied          # obs_to_applied_ms, command_id
ros2 topic echo /follower_node/safe_state
ros2 bag record /follower_node/joint_states /follower_node/applied
```

---

## The API

Two objects, one on each machine. The class name says which machine; the
lerobot base class it satisfies says what your loop sees.

```python
import videosdk_teleop as teleop
from lerobot_videosdk import VideosdkFollower, VideosdkFollowerConfig

# ---- follower host: the machine holding the arm ----------------------------
arm    = SO101Follower(SO101FollowerConfig(port="/dev/ttyACM0", cameras={...}))
robot  = teleop.SafeRobot(arm, teleop.SafetyConfig(slew=12.0,
                                                   watchdog_timeout_s=0.5))
remote = VideosdkFollower(VideosdkFollowerConfig(
    meeting_id="robot-lab-1", token=TOKEN), robot=robot)

robot.connect(); remote.connect()
for _ in pace(30):
    remote.send_feedback(robot.get_observation())
    if action := remote.get_action():
        robot.send_action(action)
```

```python
from lerobot_videosdk import VideosdkLeader, VideosdkLeaderConfig

# ---- leader host: the machine holding the leader arm -----------------------
leader = SO101Leader(SO101LeaderConfig(port="/dev/ttyACM1"))
remote = VideosdkLeader(VideosdkLeaderConfig(
    meeting_id="robot-lab-1", token=TOKEN), teleop=leader)

leader.connect(); remote.connect()          # connect() claims control
for _ in pace(30):
    observation = remote.get_observation()  # joints + one array per camera
    remote.send_action(leader.get_action())
```

`remote` on the leader host is an ordinary lerobot `Robot`, so `lerobot-record`
and the policy tooling accept it unmodified.

### Safety is a wrapper, not a runtime

```python
robot = teleop.SafeRobot(arm, teleop.SafetyConfig(
    slew=12.0,                  # max change per joint per tick, arm's own units
    watchdog_timeout_s=0.5,     # no actions for this long -> hold position
    on_starvation=teleop.FailsafeAction.HOLD))
```

`SafeRobot` presents the same interface as the arm and is the object your loop
holds, so **there is no path to the motors that bypasses it**. It clamps every
action, holds if the operator disappears, and latches on `estop()` until a
human calls `clear_estop()`. A limit enforced inside the transport would be
advisory — the loop could always talk to the arm directly.

| | |
|---|---|
| `on_safe_state(state, reason)` | every failsafe transition |
| `on_limited(event)` | one per clamp, with magnitude |
| `on_applied(applied, target, limits, mono)` | what actually reached the motors |
| `estop()` / `clear_estop()` | latched stop |
| `.safe_state` / `.safety` | current state, limits in force |

### Recording

Same four methods on either side:

```python
session = remote.start_recording("./sessions")
remote.start_episode("pick up the cube")
...                                          # your loop runs
remote.end_episode(success=True)
remote.stop_recording()
```

On the **follower** that records the arm's own observations — pixels as
captured, before an encoder or a network touched them. On the **leader** it
records the synced stream: each frame paired with joints interpolated to that
frame's own capture instant. Export either with
`examples/export_lerobot.py`.

### Correlation, for free

`get_observation()` remembers which observation it returned, and
`send_action()` stamps the next command with it — so every action in a
recording points at the exact observation the operator was looking at, by
integer id, with no clock shared between the machines.
`remote.last_observation_id` exposes the edge.

### Adding a robot

Nothing here is SO-101 specific. Both classes read `observation_features` and
`action_features` off whatever object you pass and ask no other questions, so
implementing lerobot's `Robot` interface for your hardware is the whole
integration — see `examples/lerobot/my_arm.py`.

### The v1 runtime

`teleop.follower()` / `teleop.leader()` and the `RobotAdapter` protocol still
ship and still work. They own the control loop rather than handing it to you,
which is why the plugin above is now the recommended path. See
[API.md](API.md).

---

## The idea

The SDK can answer both directions of one question:

> *For command X, which camera observation did the operator act upon?*
> *For observation Y, which action came out and what actually happened?*

```
  FOLLOWER                                        LEADER
  mint Observation(oid=N) ──── state frame ────►  remember last_oid = N
    wrist frame 500                                       │
    front frame 900                              operator moves the arm
    robot state_seq 4410                                  │
        │                                                 ▼
  ObservationLog.get(N)  ◄──── cmd frame ──── Command(cid=M, oid=N)
    O(1), no clocks involved
        │
  AppliedAction(cid=M, oid=N, target=…, action=…, limits=[…])
        │
  RobotState(sseq=K, cid=M)
```

Each id is minted by the machine that owns the thing it names and only ever
echoed by the other. **No shared counter, no clock sync, no agreement problem.**

Read [SYNC.md](SYNC.md) next — it is the load-bearing document.

---

## Documents

| | |
|---|---|
| [EXAMPLES.md](EXAMPLES.md) | bring your own hardware: complete code for both ends, no SO-101 |
| [FOLLOWER.md](FOLLOWER.md) | the robot side: the tick, every method, every event, the state machine |
| [LEADER.md](LEADER.md) | the operator side: lease, deadman, arming, every event |
| [API.md](API.md) | every public symbol: signatures, defaults, threading, exceptions |
| [ARCHITECTURE.md](ARCHITECTURE.md) | threads, modules, safety layers, adding a robot or camera |
| [PROTOCOL.md](PROTOCOL.md) | wire format, receive gates, v1↔v2 compatibility matrix |
| [SYNC.md](SYNC.md) | every clock, what is measurable and what is not |
| [RECORDING.md](RECORDING.md) | staging schema, drop accounting, LeRobot export |
| [ROS2.md](ROS2.md) | the two ROS 2 nodes, LeRobot units, what ROS is and is not here |

---

## Safety

| Layer | Where |
|---|---|
| lease | transport, receive |
| sequence (duplicate / reorder) | transport, receive |
| staleness | control loop |
| watchdog | control loop, every tick |
| deadman | control loop |
| position + slew clamps | `Clamp` |
| e-stop | **latched**, only a human clears it |

Defaults are timid on purpose. `watchdog_timeout_s` and `max_staleness_ms`
should be re-derived on **your** link under load — the numbers that matter are
the ones your network actually produces.

`HOLD` is the default failsafe, not `TORQUE_OFF`: cutting torque on a
gravity-loaded arm drops it.

Every clamp is recorded with its **magnitude**, not just a count. A joint
clamped by 0.2° is rounding; the same joint clamped by 40° means calibration is
wrong, and a bare counter cannot tell them apart.

---

## Testing

```bash
pip install -e ".[dev]"
pytest -q
```

No test touches a network, a servo, a camera, or `sleep()`. Everything
timing-sensitive takes an explicit monotonic value. `LoopbackTransport`
*subclasses* the real transport and replaces only the three methods that touch
the network, so the receive gates under test are production code.

The same harness is importable for your own adapters:

```python
from videosdk_teleop.testing import make_follower, make_leader, connect
```

---

## Known limitations

1. **`oid` is a bound, not a proof.** The leader stamps the newest observation
   it received *on the data channel*; what the operator's eye saw is a video
   frame from a separate RTP path. Closing this needs frame metadata inside the
   encoded video (SEI) and vsaiortc owns the codec. Mitigated by recording both
   the claimed and the locally-current observation, so the doubt is measured.

2. **`observation → command` latency is not reported.** It spans two
   unsynchronised clocks and cannot be measured. `observation → applied` is
   exact and is the number that matters. See [SYNC.md](SYNC.md).

3. **`capture_mono` is arrival, not shutter.** True shutter timestamps are not
   exposed through the vsaiortc MediaPlayer path. Inter-camera *skew* stays
   meaningful; the absolute value carries a fixed unknown offset.

4. **Images are re-encoded.** The capture path decodes MJPG before we see it, so
   the recorder re-encodes JPEG on the writer thread. Off the control loop, but
   not free.

5. **`claim` is first-come and unauthenticated.** Any participant can take the
   lease. E-stop is unauthenticated *by design* — anyone should be able to stop
   the arm — but `claim` should be gated by meeting-level auth before this is
   exposed to untrusted participants.

6. **SO-101 control-table addresses are unverified against your firmware.**
   Feetech ships variants. Run `camera_test.py --servo-probe` once per rig.

7. **`videosdk` does not import on Python 3.14** (pydantic v1 inside
   pymediasoup). The transport degrades to unavailable and everything else
   still runs; use 3.10–3.12 for live meetings.

8. **Single-process, single-robot.** No fleet, no multi-robot session, no
   authenticated operator handover queue.

---

## Next

- Gate `claim` behind meeting-level identity, and add an operator handover queue
- SEI frame metadata if/when the codec is reachable — closes limitation 1
- Direct PyAV demux to keep original MJPG packets — closes limitation 4
- Bench-derived `watchdog_timeout_s` / `max_staleness_ms` from a probe script
- Bus-budget measurement to justify raising `observation_hz` toward camera fps
- Policy playback: replay a LeRobot dataset through the same safety envelope
