20 | 天马行空的DeepDream

简介

介绍DeepDream的原理并用TensorFlow实现

效果

先来看一下DeepDream的效果,本来是这样一张图片

经过DeepDream处理之后就有可能变成这样

有点奇特和梦幻,也有点不明所以、精神污染

原理

大多时候我们是根据给定的数据和标签,去训练和调整网络的参数

不过也有时候,我们是固定网络的参数,根据某个损失函数调整输入数据,例如在图像风格迁移里,根据内容损失函数和风格损失函数调整合成的图片

对于常见的图片分类模型,输入一张图片,网络中的每个tensor会输出相应的响应值,值越大说明这个tensor越“喜欢”这张图片

比如输入一张狗的图片,网络中用于识别和分类狗的tensor就会输出较大的响应值

把优化目标设为最大化某个tensor的响应值,以此来调整输入图片,这就是DeepDream的原理

举例来说,为了满足一个喜欢狗的tensor,我们将原始图片中像狗的一些蛛丝马迹进行调整和放大,从而使得这一tensor的响应值更大

实现

加载库

# -*- coding: utf-8 -*-

import tensorflow as tf
import numpy as np
import cv2
from imageio import imread, imsave, mimsave
import matplotlib.pyplot as plt
%matplotlib inline
from scipy.ndimage.filters import gaussian_filter

加载图片分类模型,这里使用inception5h

layer_names = ['conv2d0', 'conv2d1', 'conv2d2', 
               'mixed3a', 'mixed3b', 'mixed4a', 'mixed4b', 'mixed4c', 'mixed4d', 'mixed4e',
               'mixed5a', 'mixed5b']

graph = tf.Graph()
with graph.as_default():
    with tf.gfile.FastGFile('inception5h.pb', 'rb') as f:
        graph_def = tf.GraphDef()
        graph_def.ParseFromString(f.read())
        tf.import_graph_def(graph_def, name='')
    X = graph.get_tensor_by_name('input:0')
    layers = [graph.get_tensor_by_name(name + ':0') for name in layer_names]

    all_layers_names = [tensor.name for tensor in tf.get_default_graph().as_graph_def().node]
    print(all_layers_names)

sess = tf.Session(graph=graph)

定义获取梯度tensor的函数、对原始图片按块计算梯度的函数

def get_gradient(tensor):
    with graph.as_default():
        return tf.gradients(tf.reduce_mean(tf.square(tensor)), X)[0]

def get_tile_size(num_pixels, tile_size=400):
    num_tiles = max(1, int(round(num_pixels / tile_size)))
    return int(np.ceil(num_pixels / num_tiles))

def tiled_gradient(gradient, image, tile_size=400):
    grad = np.zeros_like(image)
    H, W, _ = image.shape

    h = get_tile_size(H, tile_size)
    h_4 = h // 4
    w = get_tile_size(W, tile_size)
    w_4 = w // 4

    h_start = np.random.randint(-3 * h_4, -h_4)
    while h_start < H:
        h_end = h_start + h
        h_start_lim = max(h_start, 0)
        h_end_lim = min(h_end, H)

        w_start = np.random.randint(-3 * w_4, -w_4)
        while w_start < W:
            w_end = w_start + w
            w_start_lim = max(w_start, 0)
            w_end_lim = min(w_end, W)

            g = sess.run(gradient, feed_dict={X: [image[h_start_lim: h_end_lim, w_start_lim: w_end_lim, :]]})[0]
            g /= (np.std(g) + 1e-8)

            grad[h_start_lim: h_end_lim, w_start_lim: w_end_lim, :] = g

            w_start = w_end

        h_start = h_end

    return grad

根据梯度调整输入图片,即DeepDream

def dream(layer_tensor, image, iteration=10, step=3.0, tile_size=400):
    img = image.copy()
    gradient = get_gradient(layer_tensor)

    for i in range(iteration):
        grad = tiled_gradient(gradient, img)

        sigma = (i * 4.0) / iteration + 0.5
        grad = gaussian_filter(grad, 0.5 * sigma) + gaussian_filter(grad, sigma) + gaussian_filter(grad, 2 * sigma)

        scaled_step = step / (np.std(grad) + 1e-8)
        img += grad * scaled_step
        img = np.clip(img, 0, 255)

    return img

将原始图片进行缩放,对多个尺度进行DeepDream处理并叠加

def recursive_dream(layer_tensor, image, repeat=3, scale=0.7, blend=0.2, iteration=10, step=3.0, tile_size=400):
    if repeat > 0:
        sigma = 0.5
        img_blur = gaussian_filter(image, (sigma, sigma, 0.0))

        h0 = img_blur.shape[0]
        w0 = img_blur.shape[1]
        h1 = int(scale * h0)
        w1 = int(scale * w0)
        img_downscaled = cv2.resize(img_blur, (w1, h1))

        img_dream = recursive_dream(layer_tensor, img_downscaled, repeat - 1, scale, blend, iteration, step, tile_size)
        img_upscaled = cv2.resize(img_dream, (w0, h0))

        image = blend * image + (1.0 - blend) * img_upscaled
        image = np.clip(image, 0, 255)

    return dream(layer_tensor, image, iteration, step, tile_size)

读取一张图片

image = imread('mountain.jpg')
image = image.astype(np.float32)

分别以12个tensor的响应值作为优化目标,对原始图片进行处理

for i in range(len(layers)):
    print(layer_names[i])
    result = recursive_dream(layers[i], image)
    plt.figure(figsize=(10, 15))
    plt.imshow(result / 255.)
    plt.show()
    imsave('imgs/%s.jpg' % layer_names[i], result)

conv2d2的DeepDream结果

mixed3a的DeepDream结果

mixed4c的DeepDream结果

mixed5a的DeepDream结果

随着tensor所在的层数变深,DeepDream优化出来的图形也更加复杂

除了将某个tensor整个作为目标,也可以仅选择一个filter的响应值进行优化

例如选择mixed4c的某个filter,可以看到不同的filter偏好的图形是不一样的

for i in range(10):
    print('Filter %d of mixed4c' % i)
    result = recursive_dream(layers[7][:, :, :, i], image)
    plt.figure(figsize=(10, 15))
    plt.imshow(result / 255.)
    plt.show()
    imsave('imgs/mixed4c_filter_%d.jpg' % i, result)

mixed4c的filter0对应结果

mixed4c的filter8对应结果

当然,也可以对一张图片反复执行DeepDream,优化出来的图形会变得越来越明显

img = image.copy()
imgs = []
for i in range(20):
    print('Iteration %d of mixed4c' % i)
    img = recursive_dream(layers[7], img)
    plt.figure(figsize=(10, 15))
    plt.imshow(img / 255.)
    plt.show()
    imgs.append(img)
mimsave('imgs/mixed4c多轮迭代结果.gif', imgs, fps=5)

结果有点鬼畜,可能是mixed4c比较喜欢狗吧……

参考

github.com/Hvass-Labs/…

results matching ""

    No results matching ""