When humans make a collaborative musical performance, the initiative exchange often takes place. Sometimes the initiative is exchanged with gestures such as sign and wink and sometimes with the control key which is embedded in the performed musical sounds. It is important to communicate with partner players through the musical sounds without ruining the performance coordination.

Current researches in musical robotics lie mainly on playing instruments with precise movements. Few researches address the abilities to listen to players’ performance, to analyze perceptual musical aspects and to utilize the analysis for playing. The humanoid robot developed by Toyota
Motors can play violin with its dexterous robot hands. Also, the humanoid robot ASIMO successfully conducted Detroit orchestra showing human-like behaviors. However, in spite of the mechanically high performance, it is still difficult to combine the physical ability with a social dimension which makes it possible for robots to interact naturally with humans and surroundings. There are some achievements enabling robots to play the rhythmic sound with sensory perceptions.

This research paid attention to the natural interactions between human and robot in terms of the initiative exchange during musical performance. The objective is to build a robot vibraphone player which is capable of giving and taking the initiative with a human partner by perceiving auditory and visual information. The robot understands the human player’s intentions for initiative exchange by listening to the decreased volume and detecting nodding actions. It then performs actions to transfer the initiative with the human player. For the goal, we have designed the initiative exchange algorithm by linking behavioral cues with auditory signs. We also have realized the fundamental algorithms such as the real-time beat time prediction for synchronization, the human player’s nodding detection and the mallet detection to produce natural initiative exchange situation. The cognitive understanding to initiative exchange would create a sense of interacting with life-like machine and drive the human’s affective responses. This is a comprehensive approach that robots imitates the human intelligence of associating the thought with the body. From this point of view, this study differs from many previous researches that have focuses on imitating the external body motions of humans.