INTRODUCTION

Closed caption and sign language broadcasting are provided with the terrestrial digital television (DTV) services for deaf people in most countries.

Closed caption broadcasting can be ‘hidden’ by the DTV standard (1) and does not disturb the hearing (non-disabled) viewers’ watching, and therefore can be serviced for almost the whole broadcast time.

However, the sign language interpretation, aside from the cost problem, occupies some space in the TV screen and is hardly provided for more than 5% of the broadcast time in Korea.

Even if a TV program has a sign language interpretation in it, deaf viewers are still not satisfied because it is small and fixed in the screen.

If one can generate the sign language interpretation from the closed captions, the sign language interpretation can be provided for the remaining 95% of the broadcast time.

To this end, the real-time Korean-Korean sign language (KSL) translation and 3D avatar animation technologies are required.

Automatic 3D avatar-based interpretation systems have been proposed for a few sign languages. For example, Kaneko et al (2) suggested a Japanese sign language (JSL) interpretation system using TV program making language (TVML). Araújo et al (3) proposed an automatic translation and middleware structure for DTV for Brazilian sign language (LIBRAS).

We propose a system that translates the closed captions of weather forecast programs into KSL and present it with 3D avatar animation.

The translated sign language data are sent using the Internet to the receivers such as a personal computer (PC) and mobile devices to show the corresponding sign language animation.

The system consists of the Korean-KSL translator, the sign language avatar animation system, and the server system that provides the closed caption and video for the most recent weather forecast in an on- demand manner.

We extend the translation and the animation systems for all kind of TV programs other than the weather forecast.

Similarly to the existing systems, the purpose of the proposed system in this paper is to provide information even when human interpreters are not available, not to replace them.

CLOSED CAPTION TO SIGN LANGUAGE TRANSLATION

Table 1 shows an example of Korean-KSL translation in a couple of weather forecast sentences by a professional sign language interpreter. Our goal is to implement an automatic translator that outputs similar KSL results when the same Korean weather forecast input is given.

SIGN LANGUAGE DICTIONARY

There are about 12,000 words in KSL Dictionary. It is difficult to build the Korean-KSL dictionary and capture the motions for all these words.

Therefore we analysed the weather forecast scripts for the past three years, from KBS and a few other sources. After some preprocessing of the weather forecast scripts, the scripts are divided into part of speech (POS).

For this purpose we use the POS tagger of PNU with about 1 million registered words (4). The accuracy of the POS tagger is about 98%.

Since the basic word orders of Korean and KSL are similar as subject- object-verb (S-O-V), direct word-to-word translation rule is applied in the proposed system.

Synonyms are translated into same sign language words. For example, ‘house’, ‘housing’ and ‘abode’ are synonyms and all should be translated into one sign language word.

The synonym dictionary is built based on KorLex by Yoon et al (5), which is the Korean Wordnet.

Using this synonym dictionary, a word that is absent in the KSL dictionary still can be translated to a synonym, increasing the translation success rate. In Figure 1, if the word ‘truck’ is not registered in the dictionary, the word still can be translated to ‘car’ or ‘vehicle’.

Without the synonym dictionary, non-registered words would have been omitted or represented with finger spelling, making the system less useful.

DOWNLOAD THE FULL TECH PAPER BELOW