Sensors

Sensors#

As described in Entity, sensors sit between EntityData and raw simulation arrays in mjlab’s data access hierarchy. At their simplest, they wrap MuJoCo sensor primitives with a clean interface that maps to real robot hardware. Beyond wrapping, they are a general abstraction for transforming simulation data into structured outputs: ContactSensor aggregates contact pairs with reduction and air time tracking, RayCastSensor performs GPU-accelerated terrain scanning, CameraSensor renders RGB and depth images on the GPU, and the base Sensor class can be subclassed for custom measurement logic.

Sensors are configured at the scene level, not on individual entities. A sensor can reference an entity element (a contact sensor on the robot’s feet, an accelerometer attached to a body site), but it can also be independent of any entity entirely. This is why sensors live in SceneCfg rather than EntityCfg.

from mjlab.sensor import (
    BuiltinSensorCfg, ContactSensorCfg, ContactMatch, ObjRef,
)

# A robot with an IMU accelerometer and foot contact detection.
scene_cfg = SceneCfg(
    entities={"robot": robot_cfg},
    sensors=(
        BuiltinSensorCfg(
            name="imu_acc",
            sensor_type="accelerometer",
            obj=ObjRef(type="site", name="imu_site", entity="robot"),
        ),
        ContactSensorCfg(
            name="feet_contact",
            primary=ContactMatch(
                mode="geom", pattern=r".*_foot$", entity="robot",
            ),
            secondary=ContactMatch(mode="body", pattern="terrain"),
            fields=("found", "force"),
        ),
    ),
)

# Access at runtime.
imu = env.scene["robot/imu_acc"].data        # [B, 3] acceleration
feet = env.scene["feet_contact"].data         # ContactData
feet.found                                    # [B, P] contact count per foot
feet.force                                    # [B, P, 3] contact force per foot

mjlab provides four sensor types: BuiltinSensor for native MuJoCo measurements, ContactSensor for structured contact detection, RayCastSensor for GPU-accelerated raycasting, and CameraSensor for RGB-D rendering. The base Sensor class can be subclassed for custom measurement logic; see Extending: custom sensors below.

BuiltinSensor#

BuiltinSensor wraps MuJoCo’s native sensor types. Each sensor is attached to a MuJoCo element (site, joint, body, etc.) via ObjRef and returns a torch.Tensor with shape [num_envs, dim] where dim depends on the sensor type (3 for vectors, 4 for quaternions, 1 for scalars).

Category

Available Sensors

Site

accelerometer, velocimeter, gyro, force, torque, magnetometer, rangefinder

Joint

jointpos, jointvel, jointlimitpos, jointlimitvel, jointlimitfrc, jointactuatorfrc

Frame

framepos, framequat, framexaxis, frameyaxis, framezaxis, framelinvel, frameangvel, framelinacc, frameangacc

Other

actuatorpos, actuatorvel, actuatorfrc, subtreecom, subtreelinvel, subtreeangmom, clock, e_potential, e_kinetic

ObjRef identifies which MuJoCo element the sensor attaches to. The entity field scopes the lookup to a specific entity’s namespace, and the sensor name is auto-prefixed accordingly (e.g., "imu_acc" on entity "robot" becomes "robot/imu_acc").

from mjlab.sensor import BuiltinSensorCfg, ObjRef

# Accelerometer attached to a site.
BuiltinSensorCfg(
    name="imu_acc",
    sensor_type="accelerometer",
    obj=ObjRef(type="site", name="imu_site", entity="robot"),
)

# Joint limit sensor with output clamping.
BuiltinSensorCfg(
    name="knee_limit",
    sensor_type="jointlimitpos",
    obj=ObjRef(type="joint", name="knee_joint", entity="robot"),
    cutoff=0.1,
)

# Relative frame position (end-effector w.r.t. base).
BuiltinSensorCfg(
    name="ee_pos",
    sensor_type="framepos",
    obj=ObjRef(type="body", name="end_effector", entity="robot"),
    ref=ObjRef(type="body", name="base", entity="robot"),
)

Auto-discovery#

Sensors already defined in an entity’s XML are automatically discovered during scene composition and prefixed with the entity name. There is no need to create a BuiltinSensorCfg for these.

<!-- In robot.xml -->
<sensor>
    <accelerometer name="trunk_imu" site="imu_site"/>
    <jointpos name="hip_sensor" joint="hip_joint"/>
</sensor>

# Access by prefixed name.
imu = env.scene["robot/trunk_imu"]
hip = env.scene["robot/hip_sensor"]

ContactSensor#

Each physics step, MuJoCo produces a flat, unstructured list of contact pairs across the entire scene. A single foot geom might generate several simultaneous contacts with the ground, interleaved with contacts from other entities. ContactSensor filters this raw list to the pairs you care about, reduces multiple contacts per element down to a fixed count, and packages the result into clean, batched tensors your policy can consume directly. It builds on MuJoCo’s native contact sensor.

Primary and secondary#

Contacts are pairwise: you typically want to know “did the robot’s feet touch the terrain?”, not just “did something touch something.” primary defines the elements you are measuring (the feet). secondary optionally restricts what they are contacting (the terrain). When secondary is None, any contact with a primary element counts.

Each side is specified with a ContactMatch. The mode selects the MuJoCo element type ("geom", "body", or "subtree") and the pattern accepts a regex or tuple of regexes matched against element names within the entity.

from mjlab.sensor import ContactSensorCfg, ContactMatch

# Foot geoms contacting the terrain body.
ContactSensorCfg(
    name="feet_ground",
    primary=ContactMatch(
        mode="geom", pattern=r".*_foot$", entity="robot",
    ),
    secondary=ContactMatch(mode="body", pattern="terrain"),
    fields=("found", "force"),
)

# Self-collision: pelvis subtree against itself.
ContactSensorCfg(
    name="self_collision",
    primary=ContactMatch(
        mode="subtree", pattern="pelvis", entity="robot",
    ),
    secondary=ContactMatch(
        mode="subtree", pattern="pelvis", entity="robot",
    ),
    fields=("found",),
)

Output shape#

A pattern like r".*_foot$" resolves to P primary elements (e.g. four feet on a quadruped). Each primary becomes one column on the per-contact axis of the output tensors:

Field group

Shape

Notes

Per-contact (found, force, torque, dist, pos, normal, tangent)

[B, P * num_slots, ...]

Primary-major: indices [i * num_slots : (i + 1) * num_slots] belong to primary i.

Per-primary (current_air_time, last_air_time, current_contact_time, last_contact_time)

[B, P]

Air-time fields are accumulated per primary, reducing across slots (any slot in contact counts as the primary in contact).

With the default num_slots=1 the two shape families coincide (N == P), which is why most code can treat both as [B, P, ...].

Use mjlab.sensor.contact_sensor.ContactSensor.primary_names to recover the index-to-name mapping after pattern expansion:

sensor = env.scene["feet_contact"]
sensor.primary_names                # ["FR_foot", "FL_foot", "RR_foot", "RL_foot"]
sensor.data.current_air_time[:, 0]  # air time for FR_foot

Reduction#

A single primary element can have many simultaneous contacts with the secondary (e.g. a flat foot resting on rough terrain has multiple contact points). The reduce mode collapses those raw contacts down to num_slots representative contacts:

Mode

Behavior

"none"

Fast, non-deterministic selection of up to num_slots contacts.

"mindist"

Keep the deepest num_slots contacts.

"maxforce"

Keep the strongest num_slots contacts by force magnitude.

"netforce"

Sum all contacts into a single net wrench at the force-weighted centroid. Always emits one slot per primary regardless of num_slots.

When to set num_slots > 1#

Almost all configurations leave num_slots at its default of 1, because pattern expansion already produces one column per element of interest (one per foot, one per finger, one per body link). Increase num_slots only when a single primary may have several physically distinct contact points that you want to inspect separately, for example:

  • Computing a center of pressure on a flat foot from its corner contacts.

  • Reasoning about grasp quality from multiple fingertip-to-object contact points.

  • Detecting tipping by watching whether one corner of a contact patch loses contact.

In those cases pair num_slots with reduce in {"mindist", "maxforce", "none"}. With reduce="netforce" it has no effect.

Note

num_slots is a ceiling on what the sensor will store, not a guarantee that MuJoCo will produce that many contacts. The collision detector caps the number of contact points generated for each geom-pair according to the geom types involved. For example a sphere-vs-plane pair produces at most one contact, and a box-vs-box pair produces at most four. Setting num_slots=8 on a sphere primary against a plane secondary therefore leaves seven slots permanently zero. Check MuJoCo’s collision documentation for the per-pair limits.

Fields#

The fields tuple selects which contact quantities to extract. Only requested fields are allocated; the rest are None on the output dataclass. Available fields are "found", "force", "torque", "dist", "pos", "normal", and "tangent".

Note

torque and the friction-tangent component of force are zero unless the contact pair has friction enabled, which requires condim >= 3 on at least one geom in the pair. With condim=1 (frictionless), the contact only produces a normal force. This is a physics property of the contact, not a sensor limitation.

Air time tracking#

Locomotion tasks often need to know when feet land and take off for gait rewards. Setting track_air_time=True enables per-primary timing. The sensor maintains four additional tensors on ContactData, each shaped [B, P]: current_air_time, last_air_time, current_contact_time, and last_contact_time. Two helper methods provide edge detection for transition events:

sensor = env.scene["feet_air"]
first_contact = sensor.compute_first_contact(dt)  # [B, P], True for primaries that just landed
first_air = sensor.compute_first_air(dt)           # [B, P], True for primaries that just took off

Air time is per primary even when num_slots > 1: the sensor reduces found across slots so that any slot in contact counts as the primary being in contact.

History (decimation safe contacts)#

When using decimation (multiple physics substeps per policy step), a brief collision can occur and resolve entirely within the substep loop. By the time the policy reads the sensor, the contact is gone and found reports zero. Setting history_length on the sensor config tells the sensor to keep a rolling buffer of the last N substeps for force, torque, and distance fields. The policy can then inspect the full history and decide whether a real contact occurred.

Set history_length equal to your decimation value so the buffer covers exactly one policy step:

ContactSensorCfg(
    name="self_collision",
    primary=ContactMatch(mode="subtree", pattern="pelvis", entity="robot"),
    secondary=ContactMatch(mode="subtree", pattern="pelvis", entity="robot"),
    fields=("found", "force"),
    history_length=4,  # matches decimation=4
)

The history tensors live on ContactData alongside the regular fields:

data = sensor.data
data.force_history   # [B, N, H, 3]  (H = history_length)
data.torque_history  # [B, N, H, 3]
data.dist_history    # [B, N, H]

Index 0 is the most recent substep. To check whether any substep had a contact force above a threshold:

force_mag = torch.norm(data.force_history, dim=-1)  # [B, N, H]
had_contact = (force_mag > 10.0).any(dim=1).any(dim=-1)  # [B]

Note

track_air_time=True already accumulates contact state across substeps for gait rewards, so feet ground sensors typically do not need history_length. Use history for sensors where you need to detect brief collisions that would otherwise be missed (self collisions, illegal contact terminations).

Output#

ContactData is a dataclass whose fields correspond to the fields tuple on the config. Unrequested fields are None.

@dataclass
class ContactData:
    found: Tensor | None     # [B, N] contact count
    force: Tensor | None     # [B, N, 3]
    torque: Tensor | None    # [B, N, 3]
    dist: Tensor | None      # [B, N] penetration depth
    pos: Tensor | None       # [B, N, 3] contact position
    normal: Tensor | None    # [B, N, 3] surface normal
    tangent: Tensor | None   # [B, N, 3]

    # With track_air_time=True.
    current_air_time: Tensor | None
    last_air_time: Tensor | None
    current_contact_time: Tensor | None
    last_contact_time: Tensor | None

RayCastSensor#

RayCastSensor provides GPU-accelerated raycasting for terrain scanning and depth sensing. It supports grid and pinhole camera ray patterns with configurable alignment modes. See RayCast Sensor for full documentation.

RGB-D Camera#

CameraSensor renders RGB and depth images from MuJoCo cameras. See RGB-D Camera for full documentation.

Extending: custom sensors#

All sensors inherit from Sensor[T], a generic base class where T is the data type returned by the data property (e.g., torch.Tensor for BuiltinSensor, ContactData for ContactSensor).

The base class provides automatic per-step caching. The data property calls _compute_data() on first access each step and caches the result. The cache is invalidated automatically when update() or reset() is called, so multiple reads within the same step (from different observation or reward terms) pay the computation cost only once.

Lifecycle methods:

  • edit_spec: Add sensor elements to the MjSpec during scene construction.

  • initialize: Post-compilation setup. Cache sensor indices, allocate buffers, resolve references.

  • update: Called each physics step. Invalidates the data cache. Override to maintain per-step state (e.g., air time counters).

  • reset: Called on environment reset. Invalidates the data cache. Override to clear per-environment state.

  • _compute_data: Compute and return the sensor output. Called lazily by the data property when the cache is stale.

ContactSensor and RayCastSensor are the most complete reference implementations for custom sensor development.

Contents