ODSPC: deep learning-based 3D object detection using semantic point cloud

Three-dimensional object detection plays a key role in autonomous driving, which becomes extremely challenging in occlusion situations. This paper presents a novel multimodal 3D object detection framework which fuses visual semantic information and depth point cloud information to accurately detect targets with distant object features and occlusion situations. The framework consists of the four steps. Firstly, an improved semantic segmentation network is used to extract semantic information of objects containing similar features. Secondly, semantic images and point clouds are combined to generate pixel-level fusion data so that the semantic information and training capability of sparse and far-point clouds can be improved. Thirdly, a deep learning-based point cloud classification network is used for training of the fused data to output accurate detection frames. Fourthly, an extended Kalman filter is incorporated into point cloud prediction for image-based object detection to further enhance the robustness of object detection. Both Cityscapes and KITTI datasets are used in ablation study and experiments to validate the effectiveness of the proposed framework.