Hello Reader,

Welcome to another edition of the AIFEE newsletter!

Another week, another Machine Learning topic!

Image style transfer

Image style transfer is an approach that you can apply for image painting, where the goal is to extract the style from one image and apply it to a second image while preserving the content of this second image.

In order to achieve this, the style of an image needs to be defined.

In fact, the style of an image is very much related to its texture. So by using texture extraction methods, we are essentially extracting the style of an image. These methods are called Visual Texture Modeling methods. There are 2 types of these methods :

• Parametric Texture Modelling with Summary Statistics
• Non-parametric Texture Modelling with Markov Random Fields

In order to apply this style on an image, we can use image reconstruction techniques. There are 2 types of these techniques as well:

• Image-Optimisation-Based Online Image Reconstruction.
• Model-Optimisation-Based Offline Image Reconstruction.

Convolutional neural networks can be used for both Visual Texture Modeling and Image Reconstruction.

Neural style transfer (NST)

One of the fairly simple approaches to do style transfer in images is by using the approach described in the paper “A Neural Algorithm of Artistic Style”.

What you need :

• A CNN (VGG, Inception, ...etc) that’s pretrained on a large dataset (ImageNet).
• An image that you want to style but keep its content, let’s call it “content image”.
• An image that you want to extract its style, let’s call it “style image”.

What to do:

• Pass both images through the CNN.
• Extract feature maps from both the content image and the style image.
• To get the style from the style image, you need to use what’s called a “Gram matrix”. It’s a fancy way of saying : compute the means and correlations across the different feature maps.
• So the feature maps will represent the content of an image and the Gram matrix will represent the style of an image.
• Next, you need to optimize a loss function. The goal of this loss function is to make the style of the content image as close as possible to the style of the style image.
• At the end you would have basically transferred the style of one image to another image while preserving the content of the latter one.

​

The image above I created using open source code from a tutorial provided by Tensorflow.

Real time neural style transfer

The previous approach gives some great results from an artistic perspective, but the downside is that it’s too long.

Every image has to go through a set of iterations where there is extraction of the content and the style of two images and then there is an optimization process to make one image’s style close to another one’s.

Because of this hurdle, another approach has been proposed in the literature. With this method, there are two neural networks working together.

One network extracts the style from the style image. The other network has two inputs : the content image and the output of the previous network.

With this approach, the full system learns representations of painting styles from a large dataset during the training phase.

At test time, the network directly extracts the style from the style images and applies it to the content image. No optimization needed!

Reader Are you receiving this newsletter in your main inbox?

If you're not receiving this newsletter in your primary inbox, then there is a quick fix that you can apply so that you don't miss any emails from me about machine learning and computer vision topics (like the current email!).

You just need to drag and drop the email from whatever tab in your gmail account to the main tab, like it's shown in the Gif below:

​