Real-Time category-based and general obstacle detection for autonomous driving

Detecting obstacles, both dynamic and static, with near-to-perfect accuracy and low latency, is a crucial enabler of autonomous driving. In recent years obstacle detection methods increasingly rely on cameras instead of Lidars. Camera-based obstacle detection is commonly solved by detecting instances of known categories. However, in many situations the vehicle faces un-categorized obstacles, both static and dynamic. Column-based general obstacle detection covers all 3D obstacles but does not provide object-instance classification, segmentation and motion prediction. In this paper we present a unified deep convolutional network combining these two complementary functions in one computationally efficient framework capable of realtime performance. Training the network uses both manually and automatically generated annotations using Lidar. In addition, we show several improvements to existing column-based obstacle detection, namely an improved network architecture, a new dataset and a major enhancement of the automatic ground truth algorithm.