Day and Night Image Classifier¶
The day/night image dataset consists of 200 RGB color images in two categories: day and night. There are equal numbers of each example: 100 day images and 100 night images.
We'd like to build a classifier that can accurately label these images as day or night, and that relies on finding distinguishing features between the two types of images!
Note: All images come from the AMOS dataset (Archive of Many Outdoor Scenes).
Import resources¶
Before you get started on the project code, import the libraries and resources that you'll need.
import cv2 # computer vision library
import helpers
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
%matplotlib inline
Training and Testing Data¶
The 200 day/night images are separated into training and testing datasets.
- 60% of these images are training images, for you to use as you create a classifier.
- 40% are test images, which will be used to test the accuracy of your classifier.
First, we set some variables to keep track of some where our images are stored:
image_dir_training: the directory where our training image data is stored
image_dir_test: the directory where our test image data is stored# Image data directories
image_dir_training = "day_night_images/training/"
image_dir_test = "day_night_images/test/"
Load the datasets¶
These first few lines of code will load the training day/night images and store all of them in a variable, IMAGE_LIST. This list contains the images and their associated label ("day" or "night").
For example, the first image-label pair in IMAGE_LIST can be accessed by index:
IMAGE_LIST[0][:].
# Using the load_dataset function in helpers.py
# Load training data
IMAGE_LIST = helpers.load_dataset(image_dir_training)
Construct a STANDARDIZED_LIST of input images and output labels.¶
This function takes in a list of image-label pairs and outputs a standardized list of resized images and numerical labels.
# Standardize all training images
STANDARDIZED_LIST = helpers.standardize(IMAGE_LIST)
Visualize the standardized data¶
Display a standardized image from STANDARDIZED_LIST.
# Display a standardized image and its label
# Select an image by index
image_num = 0
selected_image = STANDARDIZED_LIST[image_num][0]
selected_label = STANDARDIZED_LIST[image_num][1]
# Display image and data about it
plt.imshow(selected_image)
print("Shape: "+str(selected_image.shape))
print("Label [1 = day, 0 = night]: " + str(selected_label))
Feature Extraction¶
Create a feature that represents the brightness in an image. We'll be extracting the average brightness using HSV colorspace. Specifically, we'll use the V channel (a measure of brightness), add up the pixel values in the V channel, then divide that sum by the area of the image to get the average Value of the image.
Find the average brigtness using the V channel¶
This function takes in a standardized RGB image and returns a feature (a single value) that represent the average level of brightness in the image. We'll use this value to classify the image as day or night.
# Find the average Value or brightness of an image
def avg_brightness(rgb_image):
# Convert image to HSV
hsv = cv2.cvtColor(rgb_image, cv2.COLOR_RGB2HSV)
# Add up all the pixel values in the V channel
sum_brightness = np.sum(hsv[:,:,2])
area = rgb_image.shape[0]*rgb_image.shape[1] # pixels
# find the avg
avg = sum_brightness/area
return avg
# Testing average brightness levels
# Look at a number of different day and night images and think about
# what average brightness value separates the two types of images
# As an example, a "night" image is loaded in and its avg brightness is displayed
image_num = 190
test_im = STANDARDIZED_LIST[image_num][0]
avg = avg_brightness(test_im)
print('Avg brightness: ' + str(avg))
plt.imshow(test_im)
Classification and Visualizing Error¶
In this section, we'll turn our average brightness feature into a classifier that takes in a standardized image and returns a predicted_label for that image. This estimate_label function should return a value: 0 or 1 (night or day, respectively).
threshold = 120
# This function should take in RGB image input
def estimate_label(rgb_image):
# Extract average brightness feature from an RGB image
avg = avg_brightness(rgb_image)
# Use the avg brightness feature to predict a label (0, 1)
predicted_label = 0
if(avg > threshold):
# if the average brightness is above the threshold value, we classify it as "day"
predicted_label = 1
# else, the pred-cted_label can stay 0 (it is predicted to be "night")
return predicted_label
Testing the classifier¶
Here is where we test your classification algorithm using our test set of data that we set aside at the beginning of the notebook!
Since we are using a pretty simple brightess feature, we may not expect this classifier to be 100% accurate. We'll aim for around 75-85% accuracy usin this one feature.
Test dataset¶
Below, we load in the test dataset, standardize it using the standardize function you defined above, and then shuffle it; this ensures that order will not play a role in testing accuracy.
import random
# Using the load_dataset function in helpers.py
# Load test data
TEST_IMAGE_LIST = helpers.load_dataset(image_dir_test)
# Standardize the test data
STANDARDIZED_TEST_LIST = helpers.standardize(TEST_IMAGE_LIST)
# Shuffle the standardized test data
random.shuffle(STANDARDIZED_TEST_LIST)
Determine the Accuracy¶
Compare the output of your classification algorithm (a.k.a. your "model") with the true labels and determine the accuracy.
This code stores all the misclassified images, their predicted labels, and their true labels, in a list called misclassified.
# Constructs a list of misclassified images given a list of test images and their labels
def get_misclassified_images(test_images):
# Track misclassified images by placing them into a list
misclassified_images_labels = []
# Iterate through all the test images
# Classify each image and compare to the true label
for image in test_images:
# Get true data
im = image[0]
true_label = image[1]
# Get predicted label from your classifier
predicted_label = estimate_label(im)
# Compare true and predicted labels
if(predicted_label != true_label):
# If these labels are not equal, the image has been misclassified
misclassified_images_labels.append((im, predicted_label, true_label))
# Return the list of misclassified [image, predicted_label, true_label] values
return misclassified_images_labels
# Find all misclassified images in a given test set
MISCLASSIFIED = get_misclassified_images(STANDARDIZED_TEST_LIST)
# Accuracy calculations
total = len(STANDARDIZED_TEST_LIST)
num_correct = total - len(MISCLASSIFIED)
accuracy = num_correct/total
print('Accuracy: ' + str(accuracy))
print("Number of misclassified images = " + str(len(MISCLASSIFIED)) +' out of '+ str(total))
Visualize the misclassified images¶
Visualize some of the images you classified wrong (in the MISCLASSIFIED list) and note any qualities that make them difficult to classify. This will help you identify any weaknesses in your classification algorithm.
# Visualize misclassified example(s)
## TODO: Display an image in the `MISCLASSIFIED` list
## TODO: Print out its predicted label - to see what the image *was* incorrectly classified as5
num = 0
test_mis_im = MISCLASSIFIED[num][0]
fig = plt.figure(figsize=(16,16))
plt.title("Misclassified images")
for index in range(len(MISCLASSIFIED)):
ax = fig.add_subplot(4, 4, index + 1, xticks=[], yticks=[])
image = MISCLASSIFIED[index][0]
label_true = MISCLASSIFIED[index][1]
label_guess = MISCLASSIFIED[index][2]
bright = avg_brightness(image)
ax.imshow(image)
ax.set_title("{} {:0.0f} {}".format(label_true, bright, label_guess))
if index==15:
break
(Question): After visualizing these misclassifications, what weaknesses do you think your classification algorithm has?¶
Answer: Write your answer, here.
5. Improve your algorithm!¶
- (Optional) Tweak your threshold so that accuracy is better.
- (Optional) Add another feature that tackles a weakness you identified!
Adjustment part 1¶
In the followig block I just adjusted the threshold a little bit by which I could increase the accuracy from 86% to about 92%. Still we can't just consider brightness alone because day images with large shadows get misqualified as well as night images with many artificial lights.
threshold_improved = 102
# This function should take in RGB image input
def estimate_label_improved(rgb_image):
# Extract average brightness feature from an RGB image
avg = avg_brightness(rgb_image)
# Use the avg brightness feature to predict a label (0, 1)
predicted_label = 0
if(avg > threshold_improved):
# if the average brightness is above the threshold value, we classify it as "day"
predicted_label = 1
# else, the pred-cted_label can stay 0 (it is predicted to be "night")
return predicted_label
# Constructs a list of misclassified images given a list of test images and their labels
def get_misclassified_images_improved(test_images):
# Track misclassified images by placing them into a list
misclassified_images_labels = []
# Iterate through all the test images
# Classify each image and compare to the true label
for image in test_images:
# Get true data
im = image[0]
true_label = image[1]
# Get predicted label from your classifier
predicted_label = estimate_label_improved(im)
# Compare true and predicted labels
if(predicted_label != true_label):
# If these labels are not equal, the image has been misclassified
misclassified_images_labels.append((im, predicted_label, true_label))
# Return the list of misclassified [image, predicted_label, true_label] values
return misclassified_images_labels
# Find all misclassified images in a given test set
MISCLASSIFIED = get_misclassified_images_improved(STANDARDIZED_TEST_LIST)
# Accuracy calculations
total = len(STANDARDIZED_TEST_LIST)
num_correct = total - len(MISCLASSIFIED)
accuracy = num_correct/total
print('Accuracy: ' + str(accuracy))
print("Number of misclassified images = " + str(len(MISCLASSIFIED)) +' out of '+ str(total))
Visualization of misqualified images after threshold optimization¶
As shown in the code below the problem can't be solved by brightness alone.
num = 0
test_mis_im = MISCLASSIFIED[num][0]
fig = plt.figure(figsize=(16,16))
plt.title("Misclassified images")
for index in range(len(MISCLASSIFIED)):
ax = fig.add_subplot(4, 4, index + 1, xticks=[], yticks=[])
image = MISCLASSIFIED[index][0]
label_true = MISCLASSIFIED[index][1]
label_guess = MISCLASSIFIED[index][2]
bright = avg_brightness(image)
ax.imshow(image)
ax.set_title("{} {:0.0f} {}".format(label_true, bright, label_guess))
if index==15:
break
Analyzing HSV¶
When visualizing the HSV channels you can detect very fast that all misqualified night images have a very low hue value which is caused by the artificial yellow lights. The value lays about in the region of < 30.
Day images on the other hand which are a far more common mix of all sorts of colors basically land at an average value in general far above 50.
def display_artificial_lights(rgb_image):
hsv = cv2.cvtColor(rgb_image, cv2.COLOR_RGB2HSV)
cols = 2
rows = 2
h = hsv[:,:,0]
s = hsv[:,:,1]
v = hsv[:,:,2]
fig = plt.figure(figsize=(20,10))
ax = fig.add_subplot(rows, cols, 1)
ax.set_title("Original")
ax.imshow(rgb_image)
ax = fig.add_subplot(rows, cols, 2)
ax.set_title("H {}".format(np.mean(h)))
ax.imshow(h,cmap='gray', vmin=0, vmax=255)
ax = fig.add_subplot(rows, cols, 3)
ax.set_title("S")
ax.imshow(s,cmap='gray', vmin=0, vmax=255)
ax = fig.add_subplot(rows, cols, 4)
ax.set_title("V")
ax.imshow(v,cmap='gray', vmin=0, vmax=255)
plt.show()
display_artificial_lights(MISCLASSIFIED[10][0])
display_artificial_lights(MISCLASSIFIED[5][0])
100% accuracy¶
The following approach only looks at the upper part of the image and additionally takes the experience about hue values from above into account, so images with artificial bright lights will be flagged as night images.
threshold_with_hue = 130
# This function should take in RGB image input
def estimate_label_even_more_improved(rgb_image):
rgb_image = rgb_image[:rgb_image.shape[0]//4,:,:]
# Extract average brightness feature from an RGB image
avg = avg_brightness(rgb_image)
hsv = cv2.cvtColor(rgb_image, cv2.COLOR_RGB2HSV)
h = hsv[:,:,0]
avg_h = np.mean(h)
# Use the avg brightness feature to predict a label (0, 1)
predicted_label = 0
if(avg > threshold_improved):
# if the average brightness is above the threshold value, we classify it as "day"
predicted_label = 1
if(avg_h<40 and avg <= threshold_with_hue):
predicted_label = 0
# else, the pred-cted_label can stay 0 (it is predicted to be "night")
return predicted_label
# Constructs a list of misclassified images given a list of test images and their labels
def get_misclassified_images_even_more_improved(test_images):
# Track misclassified images by placing them into a list
misclassified_images_labels = []
# Iterate through all the test images
# Classify each image and compare to the true label
for image in test_images:
# Get true data
im = image[0]
true_label = image[1]
# Get predicted label from your classifier
predicted_label = estimate_label_even_more_improved(im)
# Compare true and predicted labels
if(predicted_label != true_label):
# If these labels are not equal, the image has been misclassified
misclassified_images_labels.append((im, predicted_label, true_label))
# Return the list of misclassified [image, predicted_label, true_label] values
return misclassified_images_labels
# Find all misclassified images in a given test set
MISCLASSIFIED = get_misclassified_images_even_more_improved(STANDARDIZED_TEST_LIST)
# Accuracy calculations
total = len(STANDARDIZED_TEST_LIST)
num_correct = total - len(MISCLASSIFIED)
accuracy = num_correct/total
print('Accuracy: {:0.0f}% '.format(accuracy*100))
print("Number of misclassified images = " + str(len(MISCLASSIFIED)) +' out of '+ str(total))