Motion event detection based on LiDAR point stream

Infineon / Mitsubishi / Fuji / Semikron / Eupec / IXYS

Motion event detection based on LiDAR point stream

Posted Date: 2024-01-22

1. Write in front

Today I would like to recommend to you the latest open source work from the MARS Laboratory of the University of Hong Kong, which uses lidar to achieve microsecond-level motion event detection and was published in Nature Communications, a sub-journal of Nature.

Let’s worship this masterpiece together~

2. Summary

In dynamic environments, robots require instantaneous detection of motion events with microsecond latency. This task, called mobile event detection, is typically accomplished using event cameras. Although LiDAR (Light Detection and Ranging, LiDAR) sensors are crucial for robotics due to their dense and precise depth measurements, their application in event detection has not been deeply explored. Current methods involve accumulating LiDAR points into frames and detecting object-level motion, resulting in latencies of tens to hundreds of milliseconds. We propose a different approach, called M-detector, which determines whether a point moves immediately upon arrival, resulting in a point-by-point detection with a latency of only a few microseconds. M-detector is designed based on the occlusion principle and can use multiple types of LiDAR sensors in different environments. Our experiments demonstrate the effectiveness of M-detector on various datasets and applications, showing its superior accuracy, computational efficiency, detection latency, and generalization capabilities.

3. Effect display

M-detector performs instantaneous detection of motion in the scene, that is, detection of pedestrians who suddenly pass through. The pedestrian emerges from behind the wall at 0.01 s and quickly enters the road without checking the road conditions (a). b: Detect that a stationary pedestrian starts to move. The woman waits at the intersection until 0.4 s, when she lifts her legs and starts to cross the street. The remaining pedestrians remained stationary and did not start to move. In both cases, the first row shows the image sequence and the second row shows the detection results of the last 100 ms of the M-detector on a Livox AVIA lidar with unconventional non-repeating irregularities Scan mode. The M-detector determines whether the point is on a moving object immediately after the point arrives, resulting in a detection delay smaller than the LiDAR point sampling interval (several microseconds). M-detector is designed based on the first principle, that is, the occlusion principle, and detects points sampled by any moving part in the scene, regardless of its shape.

4. What is the specific principle?

Analogy between M-Detector and M-Cell. M-detector detects motion events from LiDAR point streams in a point-by-point detection manner, resulting in a detection delay of only 2 ~ 4 μs. The low latency of the M-detector is similar to the giant cells (M-cells) in the lateral geniculate nucleus (LGN) of the human visual system, which also have faster response times but lower resolution. In contrast, accumulating points into frames results in higher resolution but also longer latencies (e.g., 100 ms), a phenomenon similar to that of parvocellular cells in the LGN (P- cells).

The principle of occlusion. a: When an object passes through the laser ray of the lidar, the current point (blue point) will block the point previously collected at time Tk-1 (orange point). b: When the object moves along the laser ray and moves away from the sensor, the current point (blue point) will be obscured by all previous points (i.e. the orange point at Tk-1 and the green point at Tk-i), and then own (i.e. the orange point at Tk-1 is obscured by the green point at Tk-i) is further occluded. As the object moves along the laser ray towards the sensor, the current point (blue point) will occlude all previous points (i.e. orange point at Tk-1 and green point at Tk-i) and further occlude its own ( That is, the orange point at Tk-1 surrounds the green point at Tk-i).

M-detector’s system workflow (a) and event detection steps (b).

5. Comparison with other SOTA methods

Performance of M-detector on different data sets. a: IoU results of different methods on different data sets. b: Comparison of M-detector, LMNet-8* and SMOS on KITTI. The scene image is frame 68 in sequence 15. c: AVIA-Indoor dataset details. The left column shows the images taken in the scene, and the right column shows the corresponding detection results of M-detector in frame-out mode. In all the above detection results, red points represent event points marked by this method, and white points represent non-event points.

Time consumption and detection latency on different datasets. a: Time consumption per frame of different methods on different data sets. Time consumption is expressed in log scale. b: Time consumption breakdown and detection latency of M-detector on different data sets. The outer ring represents the time consumption breakdown, including event detection, clustering and region growing, and depth image construction. The total length of the ring represents the frame period, occupied by the above 3 steps in different colors and corresponding numbers. The unshaded portion of each ring represents idle time per frame. The pink middle ring sector represents the detection delay of M-detector in frame-out mode, which is caused by event detection and clustering together with region growth. The brown inner dots represent the detection delay of M-detector in point-out mode, which is only caused by event detection at a single point. The words and numbers in the center of the ring represent the name of the data set and the corresponding frame period respectively.

Review Editor: Huang Fei

#Motion #event #detection #based #LiDAR #point #stream