Measuring and enhancing image quality attributes
Discover the main attributes used to measure and enhance the perceived image quality
Before starting our discussion about measuring or enhancing image quality attributes, we have to first properly introduce them. For this, I’ve taken inspiration from the book Camera Image Quality Benchmarking, which describes in great detail the attributes that I will be speaking about here. It’s important to note that, although the attributes described in the book are camera attributes, our discussion is centered around image attributes. Fortunately, a couple of camera attributes can be used as image attributes as well.
Measuring attributes
1. Exposure
Usually referring to the time of exposure, which is a property of the camera that affects the amount of light in an image. The corresponding image attribute is actually the brightness. There are multiple ways to compute the brightness or an equivalent measure:
- I found that mapping from RGB to HSB(Hue, Saturation, Brightness) or HSL (Hue, Saturation, Luminance) and looking only at the last component (L or B) would be a possibility.
- A really nice measure of perceived brightness is the one proposed by Darel Rex Finley where:
If we do the average for all the pixels, we can obtain a measure of perceived brightness. Also, by splitting the resulting value into five pieces (because the min is 0 and the max is 255) we can define a scale: (Very dark, dark, Normal, Bright, Very Bright).
import cv
import mathimg = cv2.read(‘image.jpg’)def pixel_brightness(pixel):
assert 3 == len(pixel)
r, g, b = pixel
return math.sqrt(0.299 * r ** 2 + 0.587 * g ** 2 + 0.114 * b ** 2)def image_brightness(img):
nr_of_pixels = len(img) * len(img[0])
return sum(pixel_brightness(pixel) for pixel in row for row in img) / nr_of_pixels
2. Tone mapping
High-dynamic-range imaging (HDRI or HDR) is a technique used in imaging and photography to reproduce a greater dynamic range of luminosity than is possible with standard digital imaging or photographic techniques. While the human eye can adjust to a wide range of light conditions, most imaging devices use 8-bits per channel, so we are limited to only 256 levels. HDR imaging works with images that use more than 8 bits per channel (usually 32-bit float values), allowing a much wider dynamic range.
What is tone mapping?
There are different ways to obtain HDR images, but the most common one is to use photographs of the scene taken with different exposure values. To combine these exposures it is useful to know your camera’s response function and there are algorithms to estimate it. After the HDR image has been merged, it has to be converted back to 8-bit to view it on usual displays. This process is called tone mapping.
Measuring if an image is well tone-mapped
From the above definition, I propose (so it’s possible it’s totally wrong) the following procedure for measuring tone mapping. The intuition behind this comes from the way histograms look when the images are not properly tone mapped. Most of the time they look like this:
They are either too dark (shadow clipping), too bright (highlights clipping), or with both (for example a dark bathroom with the blitz visible in the mirror or a photo of a light pole in the middle of the night).
In contrast, a well tone mapped image looks like this:
Based on this, I propose (so take it with a grain of salt) a scoring method that tries to take into account the things described above. The score will be between [0, 1], 0 meaning the image is not correctly tone-mapped, and 1 that it is correctly tone-mapped. Besides the saturation effect, a poorly tone-mapped image might also be an image that has the majority of the brightness values in a tight interval (small variance => fewer available tones).
- For simplicity, to not work with distinct color channels, we can use the brightness (
pixel_brightness) from above. - We construct a brightness histogram (x is from [0, 255])
- We build a probability distribution from the histogram, in the [0, 1] range:
4. We define a parabolic penalizing probability distribution, that’s 0 in 0 and 1 with a maximum in 1/2 (This should be pretty fine as long as we penalize the extremes — thus low scores <=> the majority of the brightness was concentrated in the head and tail of the distribution).
(Note: This is actually a simple example of a Bernoulli distribution, a good thing to use as a prior probability distribution).
5. Next, we can define the “penalized” brightness probability distribution as . The only thing left is to properly limit this product…between 0 and 1. The first part is already solved…the minimum of this product, for all the values of is 0. That’s because we can define a black and white image that’s with the following probability distribution:
We can see because f is not 0 only at 0 and 255 the sum over all the pixels in the example image will be 0. Any other configuration would result in a sum that’s greater than 0.
To make the sum at most 1 we can use a high school trick, via the CBS inequality. In general:
In our case, that would be:
If we divide the left part with the right part we finally get a score that’s between 0 and 1. Thus the final form of the first term is:
I don’t know why, but it resembles a lot with Pearson’s correlation factor… 🤔
The next term I would simply define as:
In the end, we get the following tone mapping score:
Now, let’s see some code as well:
import math
import numpy as np
from scipy.stats import betaRED_SENSITIVITY = 0.299
GREEN_SENSITIVITY = 0.587
BLUE_SENSITIVITY = 0.114def convert_to_brightness_image(image: np.ndarray) -> np.ndarray:
if image.dtype == np.uint8:
raise ValueError("uint8 is not a good dtype for the image")
return np.sqrt(
image[..., 0] ** 2 * RED_SENSITIVITY
+ image[..., 1] ** 2 * GREEN_SENSITIVITY
+ image[..., 2] ** 2 * BLUE_SENSITIVITY
)
def get_resolution(image: np.ndarray):
height, width = image.shape[:2]
return height * width
def brightness_histogram(image: np.ndarray) -> np.ndarray:
nr_of_pixels = get_resolution(image)
brightness_image = convert_to_brightness_image(image)
hist, _ = np.histogram(brightness_image, bins=256, range=(0, 255))
return hist / nr_of_pixels
def distribution_pmf(dist: Any, start: float, stop: float, nr_of_steps: int):
xs = np.linspace(start, stop, nr_of_steps)
ys = dist.pdf(xs)
# divide by the sum to make a probability mass function
return ys / np.sum(ys)
def correlation_distance(
distribution_a: np.ndarray, distribution_b: np.ndarray
) -> float:
dot_product = np.dot(distribution_a, distribution_b)
squared_dist_a = np.sum(distribution_a ** 2)
squared_dist_b = np.sum(distribution_b ** 2)
return dot_product / math.sqrt(squared_dist_a * squared_dist_b)
def compute_hdr(cv_image: np.ndarray):
img_brightness_pmf = brightness_histogram(np.float32(cv_image))
ref_pmf = distribution_pmf(beta(2, 2), 0, 1, 256)
return correlation_distance(ref_pmf, img_brightness_pmf)
3. Texture Blur
Because, the blurred image’s edge is smoothed, so the variance is small. It's a one-liner in OpenCV, simply code 🎨: (https://stackoverflow.com/questions/48319918/whats-the-theory-behind-computing-variance-of-an-image).
import cv2def blurry(image, threshold=100):
return cv2.Laplacian(image, cv2.CV_64F).var() < threshold
Enhancing attributes
- HDR with multiple images
The OpenCV docs have a nice tutorial on this, High Dynamic Range (HDR).
For brevity’s sake, I’m putting here only the results obtained with Debevec’s algorithm (http://www.pauldebevec.com/Research/HDR/debevec-siggraph97.pdf).
- First, multiple pictures are taken with different exposure times (the exposure times are known, and the camera is not moving).
import cv2 as cv
import numpy as np# Loading exposure images into a list
img_fn = [“img0.jpg”, “img1.jpg”, “img2.jpg”, “img3.jpg”]
img_list = [cv.imread(fn) for fn in img_fn]
exposure_times = np.array([15.0, 2.5, 0.25, 0.0333], dtype=np.float32)# Merge exposures to HDR image
merge_debevec = cv.createMergeDebevec()
hdr_debevec = merge_debevec.process(img_list, times=exposure_times.copy())# Tonemap HDR image (i.e. map the 32-bit float HDR data into the range [0..1])
tonemap1 = cv.createTonemap(gamma=2.2)
res_debevec = tonemap1.process(hdr_debevec.copy())# Convert datatype to 8-bit and save (! 8-bit per channel)
res_debevec_8bit = np.clip(res_debevec*255, 0, 255).astype(‘uint8’)
cv.imwrite(“ldr_debevec.jpg”, res_debevec_8bit)
The end result:
2. Flare
Finding flares reduces to the problem of finding very bright regions in the image. I haven’t found a specific method of finding if an image has a flare, only for correcting one: The method is called CLAHE (Contrast Limited Adaptive Histogram Equalization).
import numpy as np
import cv2
img = cv2.imread('statue.jpg',0)
res = cv2.equalizeHist(img)
cv2.imwrite('global_hist_eq_statue.jpg',res)Before speaking about CLAHE, it’s good to know why Histogram Equalization does NOT work:
While the background contrast has improved after histogram equalization, the face of the statue became too bright. Because of this, a local version is preferred and thus, adaptive histogram equalization is used. In this, the image is divided into small blocks called “tiles” (tile size is 8x8 by default in OpenCV). Then each of these blocks is histogram equalized as usual. So in a small area, a histogram would confine to a small region (unless there is noise). If the noise is there, it will be amplified. To avoid this, contrast limiting is applied.
import numpy as np
import cv2
img = cv2.imread('statue.jpg',0)# create a CLAHE object (Arguments are optional).
clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))
cl1 = clahe.apply(img)
cv2.imwrite('clahe_statue.jpg',cl1)
More on histogram equalization on the OpenCV docs (https://docs.opencv.org/3.1.0/d5/daf/tutorial_py_histogram_equalization.html).
