How to predict the unknown by looking at its closest neighbors

How to predict the unknown by looking at its closest neighbors
Image: Intellect (generated)

In short: In machine learning, you don't always need a complex model to understand new data. Sometimes, the best way to classify an unknown entity is simply to look at the examples most similar to it. This intuitive approach, known as the k-nearest neighbors algorithm, relies on the power of local patterns.

Developed in 1951 by researchers Evelyn Fix and Joseph Hodges, and later expanded by Thomas Cover, the k-nearest neighbors (k-NN) algorithm operates on a simple principle: similarity implies shared characteristics. It is a non-parametric, supervised learning method that works for both classification and regression. In classification, a new data point is assigned a label based on a plurality vote from its 'k' closest neighbors. In regression, the prediction is instead calculated as the average value of those neighbors.

The 'k' in k-NN represents a user-defined integer. Choosing the right value is a delicate balance. A small k, such as k=1, makes the algorithm highly sensitive to the specific nearest neighbor, while larger values of k can reduce the impact of noise but may blur the boundaries between different classes. To avoid tied votes in binary classification, practitioners often select an odd number for k.

While powerful, the algorithm has notable sensitivities. Because it relies on distance metrics—such as Euclidean distance for continuous variables or Hamming distance for text—it can be easily misled by inconsistent scales. If features represent different physical units, the accuracy may drop unless the data is normalized. Furthermore, if one class is much more frequent than others, it can dominate predictions. To counter this, researchers use weighting schemes, where closer neighbors exert more influence on the final decision, often using a weight proportional to the inverse of the distance.

Opens the Intellect+ feed with this card first. Swipe up for the next one.

Sources

To cite this card, link to https://intellectplus.co.uk/app/technology-k-nearest-neighbors-algorithm.

How we source, check and correct cards: Editorial standards.

General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.

Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?

Related

Finding something worth knowing…