Project - Active Learning
Hidden annotation has been proved to be a powerful tool to bridge the gap between low-level features and high-level semantic meanings in a retrieval system. We propose to use active learning to improve the efficiency of hidden annotation. Active learning has been studied in the machine learning literature. For many types of machine learning algorithms, one can find the statistically "optimal" way to select the training data. This was given the name active learning. Although in traditional machine learning research, the learner typically works as the recipient of data to do training, active learning enables the learner to use his own ability to respond, to collect data, and to influence the world he is trying to understand. To be more precise, what we are interested here is a specific form of active learning, i.e., selective sampling. The goal of selective sampling is to reduce the number of training examples that need to be labeled by examining unlabeled examples and selecting the most informative ones for the human to annotate.
We propose a general active learning framework for content-based information retrieval. We use this framework to guide hidden annotations in order to improve the retrieval performance. For each object in the database, we maintain a list of probabilities, each indicating the probability of this object having one of the attributes. During training, the learning algorithm samples objects in the database and presents them to the annotator to assign attributes to. For each sampled object, each probability is set to be one or zero depending on whether or not the corresponding attribute is assigned by the annotator. For objects that have not been annotated, the learning algorithm estimates their probabilities with kernel regression. Furthermore, the normal kernel regression algorithm is modified into a biased kernel regression, so that an object that is far from any annotated object will receive an estimate result of the prior probability. This is based on our basic assumption that any annotation should not propagate too far in the feature space if we cannot guarantee that the feature space is good. Knowledge gain is then defined to determine, among the objects that have not been annotated, which one the system is the most uncertain of, and present it as the next sample to the annotator to assign attributes to. During retrieval, the list of probabilities works as a feature vector for us to calculate the semantic distance between two objects, or between the user query and an object in the database. The overall distance between two objects is determined by a weighted sum of the semantic distance and the low-level feature distance. The algorithm is tested on both synthetic database and real database. In both cases the retrieval performance of the system improves rapidly with the number of annotated samples. Furthermore, we show that active learning outperforms learning based on random sampling.
Our algorithm is best illustrated with a synthetic example. The database we constructed have three categories, and one of them has two subcategories, as is shown in the next several figures. Around 2000 samples are included in this database.
The synthetic database.
At the initialization stage, we randomly sample 50 points to start, as shown below. Black boxes represent samples that have been annotated.
In the initialization step, 50 objects are chosen randomly and annotated
We can then compare the processes of annotation the two methods have:
The annotation processes. left: random sampling; right: active learning
(a) 200 objects annotated. (b) 400 objects annotated. (c) 600 objects annotated.
From the above figures we can see that random sampling wastes a lot of annotation on areas that the low-level feature is already very good for retrieval. As a comparison, the active learning algorithm focuses on annotating the confusing area and leaves the unconfusing area untouched. This is the major reason that active learning can outperform the random sampling algorithm.
For more details about our work, please refer to our technical report. .
Any suggestions or comments are welcome. Please send them to Cha Zhang.