Colour Season Analysis
A Computer Vision tool that analyses a facial photo, predicts one of four seasonal colour types, and extracts the dominant colours from the user’s skin, eyes, and lips.
Project Details
About the Project
Colour Season Analysis is an image-classification tool that identifies a person’s seasonal colour type: Autumn, Summer, Winter, or Spring.
Seasonal colour analysis is a personal-styling framework that groups people according to characteristics such as their skin undertone, eye colour, and lip tone. Instead of leaving this entirely to subjective human judgement, we built a machine learning pipeline that extracts these features automatically and predicts a colour season using a trained neural network.
Users upload a frontal facial photo, and the application returns a predicted season, confidence breakdown, extracted facial colours, and a recommended colour palette.
Why We Built It
We wanted to explore whether a computer vision model could reproduce a task normally performed by a trained human stylist—one that depends on subtle differences in colour across facial features and skin tones.
The project allowed us to work through the complete machine learning pipeline: sourcing and augmenting a dataset, comparing different colour representations, building a facial-feature extraction system, and adapting a pretrained vision model for a niche classification task.
We also treated negative results as an important part of the project. Experiments that performed poorly helped us understand which information the model needed and which assumptions did not hold.
Preparing the Images
Before training, we masked the backgrounds of our images to isolate the face. We then applied random crops, horizontal flips, brightness and contrast changes, colour jitter, and sharpness adjustments.
These augmentations reduced the likelihood that the model would treat backgrounds or specific lighting conditions as shortcuts when predicting a colour season.
Extracting Facial Colours
We use FaRL, a Vision Transformer trained for facial understanding, to segment each image at the pixel level. This separates regions such as the skin, eyes, lips, eyebrows, nose, and hair.
We erode the edges of the resulting masks to remove contaminated pixels—for example, eyelid skin being included in the eye region.
For each selected region, we use K-Means clustering to identify its dominant colour. Selecting a valid colour cluster is more robust than averaging every pixel, which can be distorted by shadows, highlights, pupils, or reflections.
We apply filters in the LAB colour space when choosing valid clusters because its perceptual uniformity makes it better suited to comparing colours across different skin tones.
Predicting the Colour Season
Our classifier uses a CLIP ViT-B/16 visual backbone adapted with FaRL weights for facial understanding. It converts each image into a 512-dimensional feature representation.
A linear classification head maps these features to one of the four seasons. During our initial training setup, we kept the visual backbone frozen and trained the classification head on the seasonal-colour dataset.
The deployed application uses the resulting fixed model checkpoint for inference. It does not retrain itself when users upload new photos.
Takeaways
Colour-space choice mattered more than we initially expected.
HSV achieved the highest classification accuracy at 39.33%, compared with 38.76% for RGB and 38.20% for LAB. Separating hue and saturation from brightness appears to make the warm-versus-cool distinctions more accessible to the classifier.
LAB remained more useful for filtering K-Means clusters, where perceptual consistency across skin tones mattered more than class separability.
Feature Analysis
Our grayscale experiment achieved only 11% accuracy, below the 25% random baseline for four balanced classes.
This was our clearest negative result: facial shape and texture alone did not provide enough information. Chromatic features such as hue, saturation, and undertone are fundamental to seasonal colour classification.
Future Extensions
With more time, we would:
- collect a larger, more balanced, and more ethnically diverse dataset;
- fine-tune later layers of the visual backbone at a low learning rate;
- expand from four seasons to sub-types such as Soft Autumn and Deep Winter; and
- calibrate the model’s confidence scores so ambiguous predictions are represented more honestly.