CSE 126 Lecture 18

Content-Based Video Retrieval (CBVR)

Overview

How can we extend the idea of CBIR to video? The simplest way is to do the following:

1. Segment the video, breaking it up into distinct scenes.
2. QBIC extracts a representative frame (r-frame) for each scene, storing this as an image in the database.

Let's take a closer look at each of these:

Segmentation

In MPEG video, there are several properties that we can take advantage of to perform scene change detection.

1. There are likely to be large changes in color and intensity between two different scenes.
2. Objects may appear or disappear at scene boundaries
3. Macroblock prediction may change dramatically at scene change boundaries, macroblocks from one scene may not appear in another.

R-Frames

This brings up two important issues, however. First, when you are only storing one frame for an entire scene, you are losing all the important motion information contained between frames in the scene. Second, how do you choose the r-frame? You can choose the first frame of the scene, the last frame, or you can build a mosaic, combining several important features of the scene.

A complete CBVR system should able to use important information from motion, audio, subtitles, and more to really store videos according to their content… There is a lot of active research in these areas.

Other Information

What databases are well suited to handling the type of information in a CBR system? In databases, there are two primary camps:

• Relational Databases provide for very efficient querying on simple data such as text.
• Object-Oriented Databases provide support for a wide variety of data, but they are difficult to query.

As usual, when there are two different camps in an area, a hybrid approach works well. Following this trend, Object-Relational databases have been developed. These databases provide support for arbitrary data types and provide efficient query mechanisms. Two database vendors that provide these systems are Informix and Oracle. Informix has Data Blades and Oracle uses Data Cartridges. These allow a user to add functionality to their database. For example, there is a Scene Change Data Blade available, as well as several others.

While the database provides the back-end of the system where the processing takes place, the front-end of the system is another very active area of research. A Visual Asset Management System (VAMS) provides a query environment for a user, allowing him/her to define the query that will retrieve the information they are looking for in an intuitive user-friendly manner.

Notes about the Final Exam

The final will be:

• Cumulative, covering the entire course. This includes the required readings.
• Closed Book
• Closed Notes
• Closed Neighbor

You will not need to know equations, but you should be familiar with the concepts, block diagrams, and architecture diagrams of the systems we have discussed.

Good Luck!