4.1 Texture Synthesis

Goal

  • Create new samples of a given texture
  • Simplest: tiling

clipboard.png

The Challenge

clipboard.png

  • Need to model the whole spectrum from repeated to stochastic texture

One idea: Build Probability Distributions

  • Compute the stats of the input texture and generate a new texture that keeps the same stats

clipboard.png

Another Idea: Sample from the Image

  • Assume Markov property
  • A given pixel p is independent of the entire picture given the neighboring pixels around it.

clipboard.png

  • What are we doing?
  • Find patches on the input image that are similar to the neighboring pixels around p
  • Sample intensities around these patches P(p∣Neighbor(p))P(p \mid Neighbor(p))

Note

Idea from Shannon

  • Generate English-sounding sentences by modeling the probability of each word given the previous words (n-grams)
  • Large “n” will give more structured sentences

How to match patches?

  • Gaussian-weighted SSD

clipboard.png

SSD(src,template,mask)=∑i,j∑x,y∣template(x,y)⊙mask(x,y)−src(i+x,j+y)⊙mask(x,y)∣2=∑i,j∑x,y[template(x,y)⊙mask(x,y)]2−2∑i,j∑x,y[template(x,y)⊙mask(x,y)⊙src(i+x,j+y)]+∑i,j∑x,y[src(i+x,j+y)⊙mask(x,y)]2=∑i,j[∑x,y[template(x,y)⊙mask(x,y)]2]−2∑i,j∑x,y[template(x,y)⊙mask(x,y)⊙src(i+x,j+y)]+∑i,j∑x,y[src(i+x,j+y)]2⊙mask(x,y)=∣template⊙mask∣2⊙Nij−2[(template⊙mask)⋆src]+[src2⋆mask]\begin{align*} &SSD(src, template, mask) \\ &= \sum_{i, j} \sum_{x, y} |template(x, y) \odot mask(x, y) - src(i+x, j+y) \odot mask(x, y)|^2 \\ &= \sum_{i, j} \sum_{x, y} [template(x, y) \odot mask(x, y)]^2 - 2\sum_{i, j}\sum_{x, y} [template(x, y) \odot mask(x, y) \odot src(i+x, j+y)] + \sum_{i, j}\sum_{x, y} [src(i+x, j+y) \odot mask(x, y)]^2 \\ &= \sum_{i, j} \left[\sum_{x, y} [template(x, y) \odot mask(x, y)]^2\right] - 2\sum_{i, j}\sum_{x, y} [template(x, y) \odot mask(x, y) \odot src(i+x, j+y)] + \sum_{i, j}\sum_{x, y} [src(i+x, j+y)]^2 \odot mask(x, y) \\ &= |template \odot mask|^2 \odot N_{ij} - 2[(template \odot mask) \star src] + [src^2 \star mask] \end{align*}
  • The pixels with most neighbors are synthesized first (propagate outwards from the center of an image)
  • To synthesize from scratch, start with a randomly selected small patch from the source texture

Size of Neighborhood Window

clipboard.png clipboard.png

Algorithm

  • While image not filled
  1. Get unfilled pixels with filled neighbors, sorted by the number of filled neighbors
  2. For each pixel, get top N matches based on visible neighbors
  • SSD can be used for patch match
  1. Randomly select one of the matches and copy pixel from it