Abstract

Assessing physical properties of the environment with vision helps humans to respond appropriately before entering risky areas. However, equipping robots with such perceptual ability is challenging due to the lack of labeled data. To overcome this challenge, we present the Active Exploration and Online Perception (AEOP) framework, enabling legged robots to actively estimate and predict physical information of unknown terrains in real-time. The framework contains a learning-based physical sensing policy that controls the robot to actively estimate terrain physics such as friction coefficient and an incremental terrain classifier which performs temporal and spatial consistent terrain segmentation based on color images. A mapping module conjugates physical properties and visual categories of different terrains, building a point cloud map labeled with physical properties at run-time. Extensive real-world experiments were conducted where the proposed framework correctly distinguished a wide range of terrain from slippery u=0.17 to rough u=1.03 and accurately predicted the friction coefficient of encountered terrains based on vision, providing efficient perception of terrain physics for legged robots.

Abstract

Learning-based contorl System Overview

In this work, we introduce a framework for legged robots to actively estimate and predict physical information of unknown terrains in real-time. The framework includes a physical sensing policy, an incremental terrain classifier, and a mapping module. The physical sensing policy controls the robot to actively interact with the terrain and estimate physical parameters. The incremental terrain classifier performs temporal and spatial consistent terrain segmentation based on color images. The mapping module conjugates texture category and physical parameters of terrains to form a point cloud map labeled with physical properties.

Fig. 1 illustrates the overall framework of our method. The mapping module conjugates physical parameters from physical sensing policy and terrain segmentation image from incremental terrain classifier to build a point cloud map labeled with physical properties.

Physical Sensing Policy

The physical sensing policy commands the robot to perform probing motions using a DRL based end-effector tracking controller, providing effective proprioception for physical parameter estimation.

Fig. 2 demonstrates the process of probing motions.

Incremental Terrain Classifier

The classifier consists of a pre-trained segmentation model, a texture feature extraction network, and an incremental cluster. The classifier inputs an RGB image I, outputs a segmented image H where the value of each pixel represents the terrain type to which the pixel belongs

Fig. 3 illustrates the overall framework of the incremental terrain classifier. Blue arrows indicate the deployment process of the classifier: (a). FastSAM performs fine-grained segmentation for different terrains. (b): The feature extraction network trained by contrastive learning extracts feature vectors for each terrain. The training framework is indicated by gray arrows. (c): feature vectors from terrain textures is used to generate pseudo-label through incremental clustering.