Active Learning

An important Supervised Learning paradigm employed when obtaining labeled examples directly is costly.

1. Strategy

Out of the multiple strategies of active learning, exploring 2

post obtaining a current model trained on the labelled examples..
for each unlabeled feature vector x, generate a confidence score as follows:
- density(x)*uncertainty(x)
- density reflects how many examples surround x in its close neighborhood
  - can be obtained by averaging distance from the k nearest neighbors (k being a hyperparameter).
- uncertainty reflects how uncertain the prediction of the current model is for x. (obtained by prediction probabilities of x in case of classification)
  - in the case of binary classification, the closer it is to 0.5, the more uncertain the prediction is
  - in the case of Support Vector Machine, the closer the examples is to the decision boundary, the more uncertain the prediction is.
  - In the case of multiclass classification, Entropy can be used as a measure of uncertainty :- read up here : https://en.wikipedia.org/wiki/Entropy_(information_theory)
Once importances are obtained for all unlabeled feature vectors, a domain expert is asked to annotate the ones with the highest importances.
add these newly annotated examples to the training set
until a stopping criterion (max requests per annotation session for instance) has been achieved continue updating training set and finally build the model.
repeat the cycle according to desired frequencies

build an SVM model using labelled data
ask the expert to annotate unlabeled examples that lie closest to the hyperplane separating the two classes (binary classifiation)