Digito: A Fine-Grained, Gesturally Controlled Virtual Musical Instrument

 

Digito is a gesturally controlled virtual musical instrument, controlled through a number of intricate free-air hand gestures.  A performer controls Digito by 'tapping' a number of virtual notes with the index finger of their right hand.  The amplitude and timbre of the resulting note is directly related to the amplitude of the performer's tap.  Multiple notes can be played at once, or overlapped to form chords.  The sustain and vibrato of the triggered note(s) are controlled by the performer's continuous hand gestures.  Notes can either fade away naturally, in relation to their initial onset amplitude, or can be stopped at anytime by the performer making a closed hand gesture.

Hand and finger gestures are captured using a standard 3D depth sensor and recognized using a number of specifically designed computer vision and machine learning algorithms. Digito is currently being used to evaluate the possible strengths and limitations of gesturally controlled, free-air, virtual musical instruments and to assist in uncovering new questions regarding the design of gestural musical interfaces.

Digito was first presented at NIME 2012, you can read the related paper here or you can find it on the publications page of the site.

VisualFeedbackImage_01.jpg

[ Digito's main visual feedback illustrating: (a) the 12 notes of the current layer, with several layers visible in the distance, (b) the continuous gesture feedback, (c) the enlarged tracking status feedback ]