FY2020 Annual Report

Cognitive Neurorobotics Research Unit
Professor Jun Tani

From left: Takazumi, Tomoe, Nadine, Jungsik, Fabien, Minju, Jeffrey, Jun, Hendry, Ahmadreza, Hengameh, Ho Ching, Prasanna, Siqing, Dongqi, Wataru (10/11/2018)

Abstract

CNRU started in September 2017. Currently CNRU consists of 3 postdoc researchers, 1 technical stuff, 5 Ph.D. students, and one administrator. The group is investigating the essential mechanisms on human embodied cognition by using frameworks of predictive coding, active inference, and model-based reinforcement learning while conducting various robotics experiments.

1. Staff

_{As of March 31, 2021}
Prof. Jun Tani, Professor
Dr. Takazumi Matsumoto, Postdoctoral Scholar
Dr. Fabien Benureau, Postdoctoral Scholar
Dr. Jeffrey Queißer, Postdoctoral Scholar
Dr. Federico Sangati, Technical Staff
Mr. Dongqi Han, PhD Student
Ms. Nadine Wirkuttis, PhD Student
Mr. Wataru Ohata, PhD Student
Mr. Prasanna Vijayaraghavan, PhD Student
Mr. Vsevolod Nikulin, PhD Student
Mr. Tojoarisoa Rakotoaritina, Rotation Student
Dr. Jeffrey White, Visiting researcher
Dr. Shingo Murata, Visiting researcher (Assistant Professor, National Institute of Informatics Graduate University for Advanced Studies)
Ms. Tomoe Furuya, Research Unit Administrator

2. Collaborations

2.1 Computational models on reinforcement learning

Description: Model studies on hierarchical, model-based, variational reinforcement learning.
Type of collaboration: Joint research
Researchers:
- Prof. Kenji Doya, Neural Computation Unit, OIST

2.2 Computational models on the sense of self

Description: Model studies on the sense of primitive and narrative selves.
Type of collaboration: Joint research
Researchers:
- Prof. Jeff White, Twente University

3. Activities and Findings

3.1 A Hybrid Human-Neurorobotics Approach to Primary Intersubjectivity via Active Inference.

Interdisciplinary efforts from developmental psychology, phenomenology, and philosophy of mind, have studied the rudiments of social cognition and conceptualized distinct forms of intersubjective communication and interaction at human early life. Interaction theorists consider primary intersubjectivity a non-mentalist, pre-theoretical, non-conceptual sort of processes that ground a certain level of communication and understanding, and provide support to higher-level cognitive skills. We argue the study of human/neurorobot interaction consists in a unique opportunity to deepen understanding of underlying mechanisms in social cognition through synthetic modeling, while allowing to examine a second person experiential (2PP) access to intersubjectivity in embodied dyadic interaction. Concretely, we propose the study of primary intersubjectivity as a 2PP experience characterized by predictive engagement, where perception, cognition, and action are accounted for an hermeneutic circle in dyadic interaction. From our interpretation of the concept of active inference in free-energy principle theory, we propose an open-source methodology named neural robotics library (NRL) for experimental human/neurorobot interaction, wherein a demonstration program named virtual Cartesian robot (VCBot) provides an opportunity to experience the aforementioned embodied interaction to general audiences. Lastly, through a study case, we discuss some ways human-robot primary intersubjectivity can contribute to cognitive science research, such as to the fields of developmental psychology, educational technology, and cognitive rehabilitation.

Figure 1:Control is shared in the interaction based on the individuals' capacities of enacting agency and compliance. The capacity of agency is studied through the optimization of free energy (surprise) in short-term memory of direct perception. Hence, from the robot's unique past, an intended future is reinterpreted in the hierarchical RNN for deliberation control and projected in the perceptual space. The capacity of motor compliance is modulated as a proportional integral (PI) control scheme for automatic adjustment of the body to the actions induced by the partner.

3.2 Cognitive neurorobotics and self in the shared world, a focused review of ongoing research.

Through brain-inspired modeling studies, cognitive neurorobotics aims to resolve dynamics essential to different emer-gent phenomena at the level of embodied agency in an object environment shared with human beings. This article is areview of ongoing research focusing on model dynamics associated with human self-consciousness. It introduces the freeenergy principle and active inference in terms of Bayesian theory and predictive coding, and then discusses how directedinquiry employing analogous models may bring us closer to representing the sense of self in cognitive neurorobots. Thefirst section quickly locates cognitive neurorobotics in the broad field of computational cognitive modeling. The secondsection introduces principles according to which cognition may be formalized, and reviews cognitive neuroroboticsexperiments employing such formalizations. The third section interprets the results of these and other experiments inthe context of different senses of self, both ‘‘minimal’’ and ‘‘narrative’’ self. The fourth section considers model validityand discusses what we may expect ongoing cognitive neurorobotics studies to contribute to scientific explanation ofcognitive phenomena including the senses of minimal and narrative self.

3.3 Investigation of the Sense of Agency in Social Cognition, based on frameworks of Predictive Coding and Active Inference: A simulation study on multimodal imitative interaction.

When agents interact socially with different intentions (or wills), conflicts are difficult to avoid. Although the means by which social agents can resolve such problems autonomously has not been determined, dynamic characteristics of agency may shed light on underlying mechanisms. Therefore, the current study focused on the sense of agency, a specific aspect of agency referring to congruence between the agent's intention in acting and the outcome, especially in social interaction contexts. Employing predictive coding and active inference as theoretical frameworks of perception and action generation, we hypothesize that regulation of complexity in the evidence lower bound of an agent's model should affect the strength of the agent's sense of agency and should have a significant impact on social interactions. To evaluate this hypothesis, we built a computational model of imitative interaction between a robot and a human via visuo-proprioceptive sensation with a variational Bayes recurrent neural network, and simulated the model in the form of pseudo-imitative interaction using recorded human body movement data, which serve as the counterpart in the interactions. A key feature of the model is that the complexity of each modality can be regulated differently by changing the values of a hyperparameter assigned to each local module of the model. We first searched for an optimal setting of hyperparameters that endow the model with appropriate coordination of multimodal sensation. These searches revealed that complexity of the vision module should be more tightly regulated than that of the proprioception module because of greater uncertainty in visual information flow. Using this optimally trained model as a default model, we investigated how changing the tightness of complexity regulation in the entire network after training affects the strength of the sense of agency during imitative interactions. The results showed that with looser regulation of complexity, an agent tends to act more egocentrically, without adapting to the other. In contrast, with tighter regulation, the agent tends to follow the other by adjusting its intention. We conclude that the tightness of complexity regulation significantly affects the strength of the sense of agency and the dynamics of interactions between agents in social settings.

Figure 2: Overall schematic of the proposed model. Blue and red bell curves represent prior and posterior distributions, respectively. Blue and red arrows illustrate information flows of the prediction with conditional prior and posterior inferences, respectively. (A) The training scheme of the proposed network model. (B) The cycle of prediction with conditional prior and posterior inferences during an interaction with a human. (C) A diagram of providing the configuration of a human counterpart to the network.

3.4 Self-Organization of Action Hierarchy and Compositionality by Reinforcement Learning with Recurrent Neural Networks.

Recurrent neural networks (RNNs) for reinforcement learning (RL) have shown distinct advantages, e.g., solving memory-dependent tasks and meta-learning. However, little effort has been spent on improving RNN architectures and on understanding the underlying neural mechanisms for performance gain. In this paper, we propose a novel, multiple-timescale, stochastic RNN for RL. Empirical results show that the network can autonomously learn to abstract sub-goals and can self-develop an action hierarchy using internal dynamics in a challenging continuous control task. Furthermore, we show that the self-developed compositionality of the network enhances faster re-learning when adapting to a new task that is a re-composition of previously learned sub-goals, than when starting from scratch. We also found that improved performance can be achieved when neural activities are subject to stochastic rather than deterministic dynamics.

Figure 3: The consecutive relearning task: (a) Illustration of the task. (b-d) Performance curves for all phases, plotted in the same way asFig. 2(b). ReMASTER-det. stands for the case in which all the neurons followed deterministic dynamics, and LSTM is the alternativeusing the same algorithm but the network was a single-layer LSTM.(e)Performance curve of phase 3 with the lower-level synapticweights frozen (Phase 3 - LF).

3.5 Goal-Directed Planning for Habituated Agents by Active Inference Using a Variational Recurrent Neural Network.

It is crucial to ask how agents can achieve goals by generating action plans using only partial models of the world acquired through habituated sensory-motor experiences. Although many existing robotics studies use a forward model framework, there are generalization issues with high degrees of freedom. The current study shows that the predictive coding (PC) and active inference (AIF) frameworks, which employ a generative model, can develop better generalization by learning a prior distribution in a low dimensional latent state space representing probabilistic structures extracted from well habituated sensory-motor trajectories. In our proposed model, learning is carried out by inferring optimal latent variables as well as synaptic weights for maximizing the evidence lower bound, while goal-directed planning is accomplished by inferring latent variables for maximizing the estimated lower bound. Our proposed model was evaluated with both simple and complex robotic tasks in simulation, which demonstrated sufficient generalization in learning with limited training data by setting an intermediate value for a regularization coefficient. Furthermore, comparative simulation results show that the proposed model outperforms a conventional forward model in goal-directed planning, due to the learned prior confining the search of motor plans within the range of habituated trajectories.

Figure 4: Simulated robot executing the grasp and place task. In the workspace in front of the robot, there are two graspable blocks and two goal circles. Crosshair markers show the predicted positions of the gripper and the two blocks.

3.6 Efficient decomposition of latent representation in generative models.

In designing self-organizing generative models of robot behaviour, it is important to address the issue of generalization for multiple patterns, while keeping latent representation in a low dimension. We are investigating the possibility of introducing local coordinates for samples of each pattern in shared latent representation. Moreover, recent advances in efficient, approximate computation of Jacobians allows us to introduce specific regularization that ensures directional robustness in introduced local coordinates.

4. Publications

4.1 Journals

Chame H. F., Ahmadi A., & Tani, J. (2020). A Hybrid Human-Neurorobotics Approach to Primary Intersubjectivity via Active Inference. Frontiers in Psychology. 11, 584869.
Tani, J., & White, J. (2020). Cognitive neurorobotics and self in the shared world, a focused review of ongoing research. Adaptive Behavior, 1–20.
Ohata, W., & Tani, J. (2020). Investigation of the Sense of Agency in Social Cognition, based on frameworks of Predictive Coding and Active Inference: A simulation study on multimodal imitative interaction. Frontiers in Neurorobotics, 14, 61.
Han, D., Doya, K., & Tani, J. (2020). Self-Organization of Action Hierarchy and Compositionality by Reinforcement Learning with Recurrent Neural Networks. Neural Networks, 129, 149-162.
Matsumoto, T., & Tani, J. (2020). Goal-Directed Planning for Habituated Agents by Active Inference Using a Variational Recurrent Neural Network. Entropy, 22(5), 564.

4.2 Books and refereed conference proceedings

Nikulin, V., & Tani, J. (2020). Efficient decomposition of latent representation in generative models. 2020 IEEE Symposium Series on Computational Intelligence (SSCI), Canberra, Australia, December 1-4.

4.3 Invited seminars and talks

Tani, J. Understanding Embodied Cognition through Free Energy Minimization. 2021 International Workshop on Embodied Intelligence (Online), March 24, 2021.
Invited talk, Tani, J. Cognitive Neurorobotics Study Using Predictive Coding and Active Inference. Neuroengineering Symposium (Online), The Technical University of Munich, Germany, March 11, 2021.
Tani, J. Exploring minds using predictive coding and active inference frameworks. 1st SMILES workshop 2020 (Online), November 2, 2020.
Tani, J. Cognitive Neurorobotics Study and Its Possible Applications to Rehabilitation. The 2nd RENEW Workshop: Integrating Sensor Information for Optimal Rehabilitation (Online), October 20, 2020.
Tani, J. An account of the development of cognitive minds using predictive coding and active inference frameworks. University of California Irvines CogSci colloquium (Online), October 2, 2020.
Tani, J. Exploring robotic minds. Mind & Life Europe summer research institute 2020, 'Grounding Knowledge in Uncertainty’ (Online), Aug.10 -15, 2020.
Tani, J. Cognitive Neurorobotics Study Using Predictive Coding and Active Inference Framework. icra2020 rain PIL Workshop – virtual, New advances in brain-inspired perception, interaction and learning, May 31, 2020.