Interactive Natural Language Grounding via Referring Expression Comprehension and Scene Graph Parsing

Jinpeng Mi; Jianzhi Lyu; Song Tang; Qingdu Li; Jianwei Zhang

doi:10.3389/fnbot.2020.00043

Interactive Natural Language Grounding via Referring Expression Comprehension and Scene Graph Parsing

Front Neurorobot. 2020 Jun 25:14:43. doi: 10.3389/fnbot.2020.00043. eCollection 2020.

Authors

Jinpeng Mi^{1

2}, Jianzhi Lyu², Song Tang^{1

2}, Qingdu Li¹, Jianwei Zhang²

Affiliations

¹ Institute of Machine Intelligence (IMI), University of Shanghai for Science and Technology, Shanghai, China.
² Technical Aspects of Multimodal Systems, Department of Informatics, University of Hamburg, Hamburg, Germany.

Abstract

Natural language provides an intuitive and effective interaction interface between human beings and robots. Currently, multiple approaches are presented to address natural language visual grounding for human-robot interaction. However, most of the existing approaches handle the ambiguity of natural language queries and achieve target objects grounding via dialogue systems, which make the interactions cumbersome and time-consuming. In contrast, we address interactive natural language grounding without auxiliary information. Specifically, we first propose a referring expression comprehension network to ground natural referring expressions. The referring expression comprehension network excavates the visual semantics via a visual semantic-aware network, and exploits the rich linguistic contexts in expressions by a language attention network. Furthermore, we combine the referring expression comprehension network with scene graph parsing to achieve unrestricted and complicated natural language grounding. Finally, we validate the performance of the referring expression comprehension network on three public datasets, and we also evaluate the effectiveness of the interactive natural language grounding architecture by conducting extensive natural language query groundings in different household scenarios.

Keywords: human-robot interaction; interactive natural language grounding; referring expression comprehension; scene graph; visual and textual semantics.