国家自然科学基金重点国际(地区)合作研究项目

图像大数据语义分析与城市场景重建

Image Big Data Semantic Analysis and Urban Scene Reconstruction

项目名称：联合的图像大数据语义分析与城市场景重建

项目批准号：61620106003

负责人：张晓鹏

依托单位：中国科学院自动化研究所

执行日期：2017.1——2021.12

项目结题中文摘要

大数据是智慧城市的信息引擎，三维数字城市是智慧城市的信息基础。项目以城市三维形体为载体，研究城市图像大数据语义分析、城市三维重建和三维表示的新方法。研究内容为：(1)城市图像大数据语义信息提取与分析；(2)城市场景三维点云重建；(3)城市对象三维表示及语义表达。项目促进解决三个科学问题：(1)图像语义分析的质量问题。（2)图像大数据驱动的三维点云重建的效率问题。(3)城市对象三维表达的语义和规则结构的支撑问题。项目主要特色为：(1)城市图像的细粒度元素语义分析；(2)城市图像实时稠密三维重建；(3)城市对象结构与规则支撑的三维语义表达。

针对研究任务的要求，项目提出三个方面创新方法：(1)在图像特征与语义分析方面，提出面向语义分析的挤压-刺激深度网络块结构的构造方法、基于非局部信息增强的局部描述子提取方法、基于多尺度空间上下文约束的图像边缘检测方法和融合多尺度三维深度先验的建筑元素提取方法；(2)在图像处理与点云重建方面，提出非对称多值哈希加速的图像检索与匹配方法、基于端到端无监督网络的图像深度估计方法和基于多视几何约束的稠密点云重建方法；(3)在语义建模与优化表达方面，提出非学习的小波能量分解签名的多尺度特征描述方法、基于大小角消除的重新网格化方法和基于形状选择表达式的三维建筑逆向过程建模方法。

项目执行期间，发表或接受论文75篇，其中顶级期刊22篇，顶级会议19篇，其它SCI期刊24篇。申请发明专利14项，其中11项获授权。培养研究生12名，职称晋升人员8名，培养博士后5名。与国外机构和人员联合发表论文25篇，联合培养博士生3名，获2021年度中国图学学会科技进步奖二等奖。

项目成果应用到多个行业，包括自动驾驶环境感知、无人机航拍的建筑建模和损毁检测、无人机航拍图像的地图构建、建筑内户型图空间布局生成和建筑外观三维语义建模。成果形成经济社会效益，可促进有关数字化建设。

关键词：图像大数据；图像分类；语义分析；场景重建；场景语义表达

项目主要研究内容以及关系图

项目主要研究成果以及关系图

Abstract for Project Conclusion

Big data is the information engine of smart city, while 3D digital city is the data basis of smart city. The project takes the 3D shape of the city as the target to research on for new methods of image big data semantic analysis, 3D reconstruction and 3D representation for the actual city. Research contents are in three aspects: (1) Semantic information extraction and analysis of urban image big data; (2) 3D point cloud reconstruction of urban scenes; and (3) 3D representation and semantic representation of urban objects. The project promotes to solve three scientific problems: (1) the quality of urban image semantic analysis; (2) The efficiency of 3D point cloud reconstruction driven by image big data; and (3) The support of semantics and regular structure to three-dimensional representation of urban objects. Main advantages of the project are: (1) fine-grained element semantic analysis of urban images; (2) real-time dense 3D reconstruction from urban images; (3) 3D semantic expression of urban objects with the support of their structures and the rules.

Aiming at the research tasks, the project proposes three aspects of innovative methods: (1) in the aspect of image features and semantic analysis, we propose the squeeze-and excitation block method, the local description method based on non-local information enhancement, image edge detection method based on multi-scale spatial context constraints, and architectural element extraction method based on the fusion with multi-scale 3D depth priors; (2) in the aspect of image processing and point cloud reconstruction, we design the image retrieval and matching method based on the acceleration of asymmetric multi-value hash, image depth estimation method based on end-to-end unsupervised network, and dense point cloud reconstruction method based on multi-view geometric constraints; (3) in the aspect of semantic modeling and optimized expression, we present a multi-scale feature description method of non-learning wavelet energy decomposition signatures, a remeshing method based on the elimination of large and small angles, and an inverse process modeling of 3D buildings based on shape selection expressions, etc.

During the implementation of the project, 75 papers are published or accepted, including 22 top journal papers, 19 in top conference papers, and 24 papers in other SCI journals. 14 invention patents have been applied, 11 of which are authorized. Within the implementation of the project, we have educated 12 graduated students reading for their PhD or MS degrees, 8 researchers have their positions been promoted, and 5 postdoctoral fellows have finished their research work. Besides, we jointly published 25 papers with foreign institutions and personnel, jointly trained 3 doctoral students, and the main work of this project won the second prize of the 2021 China Graphics Society Science and Technology Progress Award.

The project results have been applied to the production of many industries, including autonomous driving environment perception, building modeling and damage detection of UAV aerial photography, map construction of UAV aerial photography images, the generation of spatial layout of building interior floor plans, and 3D semantic modeling of building exteriors. The results form economic and social benefits, which can be used to promote digital construction in relevant areas.

Keywords: Image big data; Image classification; Semantic analysis; Scene reconstruction; Scene semantic expression