Pixel International Conferences

The use of digital technology and interactive games as educational tools, after a period of skepticism on behalf of the educational system itself, has been adopted by the latter, giving very positive results. One of the most important reasons for this acceptance has been the use of these tools in the area of special education, particularly in the areas of general knowledge or main school subjects through specialized software and peripheral computer devices. What happens however in music education that appeals to deaf students? Can technology be used to educate the deaf on music?

So far in this area, technology has established its presence through peripheral systems, especially analog, such as chairs or accessories transporting vibrations that focus on music pleasure and not on music education. Music education however is particularly important, since through this process someone is able to develop abilities such as synchronization and harmony, be developed emotionally and mentally, cultivate new skills, improvise, create, and be given the opportunity to leave his imagination free. Moreover, music helps in language skills development and in socialization.

This study aims to integrate digital technology, in particular low-cost virtual reality systems (HMD combined with hand movement tracking systems) in the process of music education for the deaf. The ultimate goal of this research is to offer the deaf a new or a different form of contact with sound through a ludic and interactive environment. An important feature of this effort is the use of technological aids as flexible and affordable solutions, which could easily be used by music institutions, schools, cultural places or even at home.

In addition, interactivity in such an application can result into two more benefits. On one hand due to the active participation of the user, the experience itself becomes more effective and interesting. On the other hand, the system of interaction is able to grant various statistics, such as recording the user's performance, the user's interaction timespan or the repeatability, aiming at further development and upgrading of the application.

The project presented in this paper (named “Immersence; step into the world of sound”) consists of a composite immersive visual and audio system. For its realization a HMD is used, combined with a hand movement tracking system, as well as a special construction consisting of specialized speakers (low frequency) which surrounds the user at a certain position and distance. The application constitutes of three main categories: a) visual performance of oscillations of the sound in order to make its structure clear, b) creation (by the user) of rhythmic patterns through default processes and c) representation through three-dimensional space of musical compositions specifically made for this purpose in order to describe the frequency range and the meaning of timbre.