.. _rgbd_camera: RGB-D Camera ============ ``CameraSensor`` renders RGB and depth images on the GPU using MuJoCo Warp's ray-traced rendering pipeline. It can either wrap an existing MuJoCo camera defined in your XML or create a new one programmatically. Quick start ----------- .. code-block:: python from mjlab.sensor import CameraSensorCfg # Wrap an existing MuJoCo camera from the robot's XML. cam = CameraSensorCfg( name="wrist_cam", camera_name="robot/wrist_camera", data_types=("rgb", "depth"), width=160, height=120, ) scene_cfg = SceneCfg( entities={"robot": robot_cfg}, sensors=(cam,), ) # Access at runtime. data = env.scene["wrist_cam"].data data.rgb # [B, 120, 160, 3] uint8 data.depth # [B, 120, 160, 1] float32 Creating vs wrapping cameras ----------------------------- There are two ways to set up a camera sensor. **Wrap an existing camera.** If your MJCF model already defines a camera, pass its name via ``camera_name``. The sensor uses the camera's position, orientation, and field of view from the model. You can optionally override ``fovy`` or switch to orthographic projection. .. code-block:: python # Wrap the camera named "front_cam" in the robot's XML. CameraSensorCfg( name="front", camera_name="robot/front_cam", data_types=("rgb",), ) **Create a new camera.** When ``camera_name`` is ``None`` (the default), the sensor adds a new camera to the MjSpec during scene construction. Specify ``pos``, ``quat``, and optionally ``fovy`` to place it. .. code-block:: python # Fixed overhead camera on the worldbody. CameraSensorCfg( name="overhead", pos=(0.0, 0.0, 2.0), quat=(0.0, 0.707, 0.707, 0.0), fovy=60.0, width=320, height=240, data_types=("rgb", "depth"), ) Camera parameterization ----------------------- MuJoCo supports two ways to define a camera's projection, and both work with ``CameraSensor``. See the `MuJoCo camera documentation `_ for full details. **FOV-based.** The simpler approach. A single ``fovy`` (vertical field of view in degrees) combined with the image resolution defines the projection. This is the default when creating cameras programmatically via ``CameraSensorCfg``. **Intrinsic-based.** For matching real camera hardware, MuJoCo cameras can be parameterized with ``sensorsize``, ``focal`` (or ``focalpixel``), and ``principal`` (or ``principalpixel``). These fields are set in the MJCF XML and provide direct control over the intrinsic matrix. When intrinsic parameters are present, ``fovy`` is ignored by MuJoCo. When wrapping an existing camera, the sensor inherits whichever parameterization the XML defines. When creating a new camera, the sensor uses ``fovy``. To use intrinsic parameters for a new camera, define it in your XML and wrap it with ``camera_name``. .. note:: If you plan to randomize the field of view with domain randomization, use ``dr.cam_fovy`` for FOV-based cameras or ``dr.cam_intrinsic`` for intrinsic-based cameras. Randomizing ``cam_fovy`` has no effect on cameras that use intrinsic parameters. Body-mounted cameras -------------------- Set ``parent_body`` to attach a new camera to a specific body rather than the worldbody. The ``pos`` and ``quat`` are then relative to the parent body frame, so the camera moves with the body. .. code-block:: python # Camera mounted on the robot's end-effector. CameraSensorCfg( name="ee_cam", parent_body="robot/link_6", pos=(0.0, 0.0, 0.05), quat=(1.0, 0.0, 0.0, 0.0), fovy=45.0, width=160, height=120, data_types=("rgb", "depth"), ) Data types ---------- The ``data_types`` tuple selects which image modalities to render. Only requested types are allocated; the other field on ``CameraSensorData`` is ``None``. .. list-table:: :header-rows: 1 :widths: 15 20 65 * - Type - Shape - Description * - ``"rgb"`` - ``[B, H, W, 3]`` uint8 - Color image. Rendered as packed ABGR uint32 by MuJoCo Warp, then unpacked to RGB channels on the GPU. * - ``"depth"`` - ``[B, H, W, 1]`` float32 - Depth image. Values are distances from the camera plane. * - ``"segmentation"`` - ``[B, H, W, 2]`` int32 - Typed segmentation. Channel 0 stores object IDs and channel 1 stores MuJoCo object types. Background pixels are ``(-1, -1)``. Render settings --------------- All camera sensors in a scene must share identical values for ``use_textures``, ``use_shadows``, and ``enabled_geom_groups``. This is a constraint of the underlying MuJoCo Warp rendering system, which uses a single ``RenderContext`` for all cameras. Mismatched settings raise a ``ValueError`` at scene construction. .. code-block:: python # These two cameras must agree on render settings. cam_a = CameraSensorCfg( name="cam_a", camera_name="robot/front_cam", use_textures=True, use_shadows=False, enabled_geom_groups=(0, 1, 2), data_types=("rgb",), ) cam_b = CameraSensorCfg( name="cam_b", camera_name="robot/wrist_cam", use_textures=True, # Must match cam_a use_shadows=False, # Must match cam_a enabled_geom_groups=(0, 1, 2), # Must match cam_a data_types=("depth",), ) Output ------ ``CameraSensorData`` is a dataclass with one field per data type. .. code-block:: python @dataclass class CameraSensorData: rgb: Tensor | None # [B, H, W, 3] uint8 depth: Tensor | None # [B, H, W, 1] float32 segmentation: Tensor | None # [B, H, W, 2] int32 By default, the returned tensors are zero-copy views into the render buffer. Set ``clone_data=True`` on the config if you modify them in place, to avoid corrupting the shared buffer. Visualization in Viser ---------------------- The Viser viewer automatically discovers all ``CameraSensor`` instances in the scene and displays their RGB and depth outputs as live image panels in the GUI sidebar. A camera frustum is rendered in the 3D viewport showing the camera's position, orientation, and field of view. Depth images include an interactive scale slider for adjusting the visualization range. .. image:: ../_static/viser_camera_pane.png :align: center :alt: Viser viewer showing camera image panels and frustum visualization