1. Pixels and Linear Filters

Image Formation Process

clipboard.png

  1. Light comes out from the sources
  2. Light bounces off the elements in the scene
  3. Some of that light is reflected through the lens into the camera onto the imaging plane

Note

Films and Analog Camera:

  • Light is recorded as a continuous signal on that film

Digital Camera

  • Light is recorded onto a CCD
  • Cells in CCD are photo-recepters converting the photrons of light into electrons

Sensor Array

  • Continuous signals recorded are discretized based on space and intensity

clipboard.png

The raster image (pixel matrix)

clipboard.png

  • Raster Image: the matrix representation of an image
  • Difficult Part for image analysis:
  • We perceive the contrast of a black character with a white background
  • In reality, it’s a mid-range character with a noisy background

Perception of Intensity

clipboard.png

  • Intensity: Overall, how much light is coming over the sensor.
  • The darkness of A and B is actually the same.

Digital Color Images

clipboard.png

  • Color images are all generated with 3 different colors with different intensity

Images in Python

# read image (order: BGR)
im = cv2.imread(filename)

# order channels as RGB
im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)
im = im / 255
  • RGB image im is H x W x 3 matrix (np.ndarray)
  • H: height
  • W: width
  • 3: # of channels (RGB)
  • im[0, 0, 0]:
  • top-left pixel value in the R-channel
  • im[y, x, c]:
  • y + 1 pixels down
  • x + 1 pixels right
  • cth channel

Image Filtering

What is it?

  • compute function of local neighborhood at each position

Example: Box Filter

  • Blurring image
  • Calculating moving average
Formula
h[m,n]=∑k,jg[k,l]×f[m+k,n+l]h[m, n] = \sum_{k,j} g[k, l] \times f[m + k, n + l]
How it works
  • Let’s say you have a 3 x 3 box filter

clipboard.png

  • For every 3 x 3 window in the image,
  1. multiply the corresponding elements of that patch of the image with the elements in the filters
  2. sum up all of products

clipboard.png

clipboard.png

Types of Filters

Box Filter

clipboard.png

  • smooth the image
  • create some edges (will be explained later in the course)

Calculate by hand

Note: convolution

[0110122000010112]∗[100010001]=[0110124002410112]\begin{bmatrix} 0&1&1&0 \\ 1&2&2&0 \\ 0&0&0&1 \\ 0&1&1&2 \\ \end{bmatrix} * \begin{bmatrix} 1&0&0 \\ 0&1&0 \\ 0&0&1 \\ \end{bmatrix} = \begin{bmatrix} 0&1&1&0 \\ 1&2&4&0 \\ 0&2&4&1 \\ 0&1&1&2 \\ \end{bmatrix}

clipboard.png

Practice with Linear Filters

Identity Filter

  • no change
[000010000]\begin{bmatrix} 0&0&0 \\ 0&1&0 \\ 0&0&0 \\ \end{bmatrix}

clipboard.png

Shifted Left

  • every pixel would be replaced by its right neighbor
[000001000]\begin{bmatrix} 0&0&0 \\ 0&0&1 \\ 0&0&0 \\ \end{bmatrix}

Screenshot 2025-08-27 at 08.51.46

Sharpening

  • accentuates differences with local average
[000020000]−19[111111111]Box Filter\begin{bmatrix} 0&0&0 \\ 0&2&0 \\ 0&0&0 \\ \end{bmatrix} - \underset{\text{Box Filter}}{ \frac{1}{9} \begin{bmatrix} 1&1&1 \\ 1&1&1 \\ 1&1&1 \\ \end{bmatrix}}

clipboard.png

Tip

You are calculating the original matrix + the difference of the center pixel from the average of the neighbors, which makes the differences stronger.

Edge Filter

  • Subtracting sum of pixels to the left to sum of the pixels to the right
  • Stronger left pixels -> positive result
  • Stronger right pixels -> negative result
19[10−120−210−1]Sobel\underset{\text{Sobel}}{ \frac{1}{9} \begin{bmatrix} 1&0&-1 \\ 2&0&-2 \\ 1&0&-1 \\ \end{bmatrix}}

Screenshot 2025-08-27 at 09.17.14

  • white pixels: high magnitude
  • black pixels: low magnitude

How to synthesize motion blur?

  • Shift the picture by multiple positions and then average those out
im = cv2.imread(im_fn)

# invert to gray scale
im = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY) / 255

# degree motion blur
theta = 45
# theta = 0 # vertical filter
# theta = 90 # horizontal filter
len = 15
mid = (len - 1) / 2

fil = np.zeros((len, len))
fil[:, int(mid)] = 1 / len # sum of the values is 1
R = cv2.getRotationMatrix2D((mid, mid), theta, 1)
fil = cv2.wrapAffine(fil, R, (len, len))

im_fil = cv2.filter2D(im, -1, fil)
fig, axes = plt.subplot(3, 1, figsize=(50, 50))
axes[0].imshow(im, cmap='gray')
axes[1].imshow(fil, cmap='gray)

Screenshot 2025-08-27 at 09.30.55

Warning

This cannot be used to anti-blur the image as there are several ways tThat may lead to the same blurred image.

If we change the len

  • The amount of blur would be less
len = 7

clipboard.png

If we do not include the all rows

alpha = 45
len = 25
fil[12:, int(mid)] = 1 / len # synthesizing blur on the right half only
R = cv2.getRotationMatrix2D((mid, mid), theta, 1)
fil = cv2.wrapAffine(fil, R, (len, len))
  • The pixels on the upper-left will be replaced by the pixels on the lower-right
  • The image would be shifted to the upper-left and blurred clipboard.png clipboard.png

Correlation vs. Convolution

2d correlation

  • im_fil = cv2.filter2d(im, -1, fil)
  • When you takes a window over the image, you multiply corresponding elements of that window with the elements of the kernel (filter matrix) or the filter.
im_fil[m,n]=∑k,lfil[k,l]⋅im[m+k,n+l]im\_fil[m, n] = \sum_{k, l} fil[k, l] \cdot im[m + k, n + l]

2d convolution

  • im_fil = scipy.signal.convolve2d(im, fil, [opts])
  • Convolution flip the kernel by 180 degrees first and then apply correlation
im_fil[m,n]=∑k,lfil[k,l]⋅im[m−k,n−l]im\_fil[m, n] = \sum_{k, l} fil[k, l] \cdot im[m - k, n - l]

Important

“convolve” mirrors the kernel, while “filter” doesn’t. The following two expressions are the same

  • cv2.filter2D(im, -1, cv2.flip(fil, -1))
  • siginal.convolve2d(im, fil, mode='same', boundary='symm')

Properties of Linear Filters

Linearity

filter(f1+f2)=filter(f1)+filter(f2)filter(f_1 + f_2) = filter(f_1) + filter(f_2)

Shift Invariance:

  • Same behavior regardless of pixel location
filter(shift(f))=shift(filter(f))filter(shift(f)) = shift(filter(f))

Note

Any linear, shift-invariant filter’s operations can be represented as convolutions

Commutative

a∗b=b∗aa * b = b * a

Associative

a∗(b∗c)=(a∗b)∗ca * (b * c) = (a * b) * c

Distributes over addition

a∗(b+c)=(a∗b)+(a∗c)a * (b + c) = (a * b) + (a * c)

Scalars factor out:

ka∗b=a∗kb=k(a∗b)ka * b = a * kb = k(a * b)

Identity

e=[0,0,1,0,0]a∗e=a\begin{align*} &e = [0, 0, 1, 0, 0] \\ &a * e = a \end{align*}

Gaussian Filter

  • Weight contributions of neighboring pixels by nearness
  • The value in the center is the highest

clipboard.png

  • It is a very effective smoother

clipboard.png

  • Remove the high-frequency components from the image (low-pass filter)

Convolution with self is another Gaussian

  • convolving two times with Gaussian kernel of width σ\sigma is the same as convolving once with kernel of width σ2\sigma\sqrt{2}
  • The Gaussian kernel is defined as:
Gσ(x)=12πσ2e−x22σ2G_\sigma(x) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{x^2}{2\sigma^2}}
  • Convolving two Gaussian functions = Multiplying their Fourier transforms.
  • The Fourier transform of a Gaussian is also a Gaussian:
F[Gσ(x)]=e−ω2σ22\mathcal{F}[G_\sigma(x)] = e^{-\frac{\omega^2\sigma^2}{2}}

Convolve two Gaussians with widths σ1\sigma_1 and σ2\sigma_2:

  • In frequency domain:
e−ω2σ122⋅e−ω2σ222=e−ω2(σ12+σ22)2e^{-\frac{\omega^2\sigma_1^2}{2}} \cdot e^{-\frac{\omega^2\sigma_2^2}{2}} = e^{-\frac{\omega^2(\sigma_1^2 + \sigma_2^2)}{2}}
  • This corresponds to a Gaussian with variance
σresult2=σ12+σ22\sigma_{result}^2 = \sigma_1^2 + \sigma_2^2
  • For two identical Gaussians with width σ:
σresult=σ2+σ2=2σ2=σ2\sigma_{result} = \sqrt{\sigma^2 + \sigma^2} = \sqrt{2\sigma^2} = \sigma\sqrt{2}

Separability of the Gaussian Filter

Gσ=12πσ2exp−x2+y22σ2=12πσ2exp−x22σ2+12πσ2exp−y22σ2\begin{align*} G_\sigma &= \frac{1}{2\pi\sigma^2} exp ^{-\frac{x^2 + y^2}{2\sigma^2}}\\ &= \frac{1}{2\pi\sigma^2} exp ^{-\frac{x^2}{2\sigma^2}} + \frac{1}{2\pi\sigma^2} exp ^{-\frac{y^2}{2\sigma^2}} \end{align*}

Separability Example

  • Faster to computer
  • method 1 takes k2k^2 times of caluculation
  • method 2 takes 2k2k times of calculation
1.[121242121]∗[233355446]=2+6+3+6+20+10+4+8+6=652.[121242121]=[121]⋅[121][121]∗[233355446]=[111818][121]∗[111818]=65\begin{align*} % 2D filtering (center location only) 1. &\begin{bmatrix} 1 & 2 & 1 \\ 2 & 4 & 2 \\ 1 & 2 & 1 \end{bmatrix} * \begin{bmatrix} 2 & 3 & 3 \\ 3 & 5 & 5 \\ 4 & 4 & 6 \end{bmatrix} \\ &= 2 + 6 + 3 + 6 + 20 + 10 + 4 + 8 + 6 = 65 \\ % The filter factors into a product of 1D filters: 2. &\begin{bmatrix} 1 & 2 & 1 \\ 2 & 4 & 2 \\ 1 & 2 & 1 \end{bmatrix} = \begin{bmatrix} 1 \\ 2 \\ 1 \end{bmatrix} \cdot \begin{bmatrix} 1 & 2 & 1 \end{bmatrix} \\ % Perform filtering along rows: &\begin{bmatrix} 1 & 2 & 1 \end{bmatrix} * \begin{bmatrix} 2 & 3 & 3 \\ 3 & 5 & 5 \\ 4 & 4 & 6 \end{bmatrix} = \begin{bmatrix} 11 \\ 18 \\ 18 \end{bmatrix}\\ % Followed by filtering along columns: &\begin{bmatrix} 1 \\ 2 \\ 1 \end{bmatrix} * \begin{bmatrix} 11 \\ 18 \\ 18 \end{bmatrix} = 65 \end{align*}

Practical Matters

Proper size for kernel

  • For kenel, with a proper size, the values at edges should be near zero
  • For Gaussian:
kernel half_width>=3σkernel\ half\_width >= 3\sigma

Near the edges

  • the filter window falls off the edge of the image

Extrapolation

clip filter

clipboard.png

  • apply smoothing filter

clipboard.png

  • Crop the image back to the original size

clipboard.png

wrap around

clipboard.png

copy edge

clipboard.png

Caution

For templating matching, the copied edges may cause some extra problems

reflect across edge

clipboard.png

In Python

Methods

convolve2d(g, f, boundary='fill', 0) # clip filter
convolve2d(g, f, boundary='wrap') # wrap around
convolve2d(g, f, boundary='symm') # reflect across edge

Size of output

  • convolve2d(g, f, mode)

  • mode='full':

  • output size is sum of sizes of f and g

clipboard.png

  • mode='same':

  • output size is the same as f

  • What we usually wanna do

clipboard.png

  • mode='valid'

  • output size is the difference of sizes of f and g

clipboard.png

What is texture?

  • Regular or stochastic patterns caused by bumps, grooves, and markings
  • We can represent textures by computing responses of different filters

clipboard.png

Filter Bank

  1. Process image with each filter and keep resposnes
  • for example, sum of the responses may be stored to tell how strongly the image responds the filter
  1. Measure responses of blobs and edges at various orientations and scales
  2. Record simple stats of absolute filter responses