School of ECM University of Surrey Guildford, Surrey GU2 5XH, UK

Tel: +44 (0)1483 259823 Fax: +44 (0)1483 876051

For example, in most electronic dictionaries, conceptual and contextual definitions are absent; in the processing of images, the distinction between the processing of faces and objects is ignored; and sequences of sound receive the same treatment whether they be music, speech or birdsong. Furthermore, current multimedia systems take little account of how these modes interact or are indeed packaged together, e.g. texts and sounds, sounds and pictures, so that the interdependence of modalities which can contribute so much to information processing is not capitalised upon.

In terms employed by some semioticians current multimedia systems can be described as doing well in recording the material vehicles via which communications take place in various modes. However the treatment of the underlying semiotic frameworks which are crucial in the production and interpretation of meaning in these modes remains unaddressed. This observation may also be phrased in terms of the 'physical' level which is captured, and captured well, in the process of digitisation and the 'conceptual' level which is lost in the very same process.

Currently our work is proceeding in 3 areas; (i) An investigation into the storage, retrieval and presentation of images, with particular respect to 'Virtual Art Galleries'; (ii) An inquiry into the interaction of modalities and hence information processing systems in domains such as Dance, and; (iii) The development of an on-line museum - a proposed project in collaboration with a major folk museum. In each of these areas we are concerned with the nature of the underlying semiotic frameworks and how knowledge representation schemata may be used to model them. Important aspects of this modelling include understanding the place of hierarchies in models of semiotic systems and how to proceed with the characterisation of metaphoric processes which are rife throughout these systems

Virtual Art Galleries

Traditionally art galleries have left the viewer with very little choice as to how he/she wishes to view the works of art which they house. Thus the advent of on-line art galleries promises the 'virtual viewer' previously untold powers to select, manipulate and order digital renderings of art pieces for viewing. In order for this interaction between the on-line gallery and the user to be affected successfully the machine must have more knowledge about the images it contains than simply their digital encodings and basic file labels.

We are seeking to re-apply ideas which we developed for the storage and manipulation of written texts with a view to the construction of a virtual art gallery system which has some knowledge about the paintings it stores and can therefore assist the viewer in 'making sense' of the multimedia resources available to them.

"The Rehearsal", c. 1873-78. H. G. Edgar Degas (1834-1917)

Dance and multimedia

Both the production and perception of dance involve the interplay of a variety of cognitive faculties, e.g. visual, auditory, kinaesthetic, and hence the combined processing of information from different modalities. This 'multi-modal' characteristic of dance is reflected by the artefacts of dance such as video recordings, costumes, movement notations and soundtracks. These artefacts present a rich source of data for multimedia computer systems and make the domain of dance an ideal testing ground for new theories of multimodal information processing.

Working in collaboration with the Dance Studies department at Surrey we are implementing a system, based on theories of dance analysis, which incorporates knowledge about dance in the construction of multimedia databases that will store digital renderings of various dance-related artefacts. We believe that our bi-disciplinary approach will lead to the enrichment of computational theories which in turn will drive 'intelligent' interfaces to repositories of multimedia information.

R Welch and E Chua in Ballet Omaha's 1991 production of Les Sylphides

On-line Museums

We have recently started a collaboration with a folklore museum in the Middle East. In the course of this project we will be creating a WWW-based 'Virtual Museum' to facilitate the remote browsing of multimedia renderings of artefacts from the museum.

Ancient stone relief of an Assyrian soldier