This research is part of OpenVL: The Open Vision Language.

Computer vision is a complex field which can be challenging for those outside the research community to apply in the real world. The problem preventing widespread adoption is the lack of a formulation which separates the understanding of a task from the knowledge of which algorithm to use as a solution. Successful abstractions in computer graphics, such as OpenGL and DirectX, hide the algorithmic details of rendering behind a powerful interface. We propose a similar abstraction for computer vision, analogous to a shader language, which provides developers with access to sophisticated vision methods without requiring specialist knowledge. Our contribution is an interface which presents developers with mechanisms to describe their vision task; the description is interpreted to select an appropriate method and provide a solution.

Many computer vision problems can be divided up into smaller sub-problems and solved by providing solutions to each sub-problem. This applies conceptually as well as algorithmically and so we base our vision shader language on this principle. We allow the user to describe vision tasks by dividing the conceptual problem into sub-tasks, then the sequence of sub-tasks is analysed to select a suitable method to apply. Our language is made up of these sub-tasks, which we term operations, and we provide a core set to span the range of computer vision problems.

Each of these operations accepts descriptions of the conditions under which they must operate. For example, if requesting detection we provide the developer with a means to describe the object to detect (e.g. a "face", mostly front-facing, possible occlusion, multiple instances) and the means to describe the image (detailed, varied illumination). From this description we can infer which face-detection algorithm will work most effectively under these conditions. Behind the scenes, each algorithm which is incorporated into our framework is evaluated with respect to how it performs under the conditions descriptions; this allows us to select the best one when the developer describes their problem.

The power of the abstraction comes from the evaluation of the operations as a sequence instead of individually; this approach allows us to establish the higher-level problem to solve and possibly select a single method capable of solving it more effectively.

Presented at the SIGGRAPH poster session in Vancouver, August 2011.