Technologies such as text-to-speech and braille provide an alternative to electronic text. It allows blind users to access most documents independently using a screen reader that reads a text out loud or displays it on a special hardware braille device.
However, non-textual graphical information in electronic documents, such as diagrams and equations, remains largely inaccessible to the blind population. This is because mathematical equations and diagrams are represented as images, not text that the screen reader can read.
This is according to Dr Rynhardt Kruger, a researcher at the Council for Scientific and Industrial Research, who recently obtained his doctorate in Electronic Engineering at Stellenbosch University. For his PhD, titled Technical Document Accessibility, Dr Kruger, under the supervision of Professors Thomas Niesler and Febe de Wet from the Digital Signal Processing Research Group in the Department of Electrical and Electronic Engineering, developed methods that could make it easier for blind or visually impaired people to read non-textual graphical information in electronic documents.
Background to the study
Dr Kruger explains his experience and motivation for his research as follows:
“As someone who was born blind, I know first-hand how difficult it is to read equations and diagrams in scientific documents. During my studies, it was not always easy to access the material in my textbooks. I realised that many technical documents do not adhere to existing accessibility standards for blind or visually impaired people.”
Dr Kruger became the first blind person at SU to obtain both his BSc honours and master’s degrees in computer science. For his Honours project, he developed a programme that allows blind musicians to study music notation. For his master’s degree, Kruger researched methods to allow blind people to access online virtual worlds.
In his research, Dr Kruger explored audio-visual sensory substitution to render the information represented as diagrams and equations in electronic documents to blind readers. For diagram accessibility, he extended the audio rendering approach used by the well-established vOICe algorithm to allow interactive and localised exploration by means of gestures and a touch screen.
The effectiveness of this approach was evaluated in a set of user trials that required six sighted and six blind subjects to identify elements of scenes consisting of a number of geometrical shapes and emoticons.
The research results
The results show that both groups of subjects were more successful at identifying shapes using the interactive algorithm than they were when using the baseline vOICe algorithm to a highly statistically significant degree. Furthermore, the results indicate that this improvement is greatest for the most complex scenes. Subsequently, we extended this approach to enable the reading of mathematical equations embedded in PDF documents by incorporating a navigational system that draws on the navigational mechanisms often used to explore the virtual worlds of text adventure games.
The relative spatial placement of the elements of an equation is represented as a virtual world so that the reader can navigate from element to element. Textual elements are announced as synthesised speech, while graphical elements are rendered using audio-visual sensory substitution. This approach was evaluated by eleven blind and fourteen sighted participants in a set of trials comprising the identification of several equations extracted from PDF documents.
Considering only completely correct responses, blind and sighted candidates attained an accuracy of 74% and 83%, respectively, with an overall average of 78%. When partial correctness is evaluated, these accuracies are even higher.
Conclusion
By enabling interactive exploration by means of gestures, a spatial model represented as a virtual world, and audio-visual sensory substitution, currently inaccessible diagrams and mathematical content in technical documents can be made accessible to visually impaired readers without the need for specialised equipment.
Dr Kruger says he would like to see his methods built into a book reader application that will allow readers to study diagrams and equations directly from books.
Download and read the complete research: Kruger, R. (2020). Technical Document Accessibility [PhD dissertation]. Stellenboch University. Available at https://scholar.sun.ac.za/bitstream/handle/10019.1/125922/kruger_technical_2022.pdf?sequence=1.
