I had some time over the summer to have a bit of a play with pix2pix - the conditional adversarial network behind edges2cats by Christopher Hesse, which I think is probably where I first saw it. Having already used tensorflow to explore features within deepdream, and have fun with image style transfer, pix2pix was immediately interesting to me. It seemed quite flexible and I wanted to see how it could be used, or misused, for creative purposes.

With a bit of work, a fair amount of trial and error, and a lot of waiting for models to train, I have ended up with a workflow that can generate things like this…

Horizontal bands of creamy white and dark brown undulate across the frame like rows of waves or mountain ridges, flecked with blue, orange, and red.
Above and below
Diagonal streaks of cream, yellow, deep red, and near-black run across the frame like painted fabric or rock strata.
Stripe

Though they require some manual post-processing to tidy them up, I’m happy with what I can now create using pix2pix, so am documenting some of the ideas behind the process here.

what2what?

I could attempt to describe what pix2pix is and what it does… but I’m more coder/tinkerer than machine learning expert. Instead, if you’re curious, this episode of Two Minute Papers gives a good overview.

In essence, pix2pix can be trained with any set of (carefully prepared) A-to-B image pairs, will learn a relationship between the two, and should then be able to transform inputs similar to A-images into outputs that resemble B-images.

It’s flexible enough to have success at changing day to night, or indeed, edges to cats, but is limited in the size of the images it works with; the implementations I’ve seen mostly run at 256x256 pixels. Other than the time taken to train the network, there’s a reason that the examples are using that size - it’s a fairly memory hungry process.

I don’t have a particularly amazing GPU setup, so also have to work at 256x256, but want to be able to create more detailed images… Perhaps if trained with pairs of pixelated and normal images, pix2pix could be put to work as an upscaler?

It seemed worth a try, and after a few days of training, I had an answer.

Input, output, and target panels of a potted plant, from blocky pixels to a painterly reconstruction to the original painting.
Input, output, and target panels of a portrait of a moustached man, the output's face smeared and distorted.
Input, output, and target panels of a still life with a skull on a shelf.
Input, output, and target panels of a sunlit landscape with trees under a cloudy blue sky.
Input, output, and target panels of a house behind trees in a green garden.
Input, output, and target panels of cliffs along a beach under a golden evening sky.
Input, output, and target panels of a field of red flowers with tall trees on the horizon.
Input, output, and target panels of a tall tree beside a row of houses on a riverside path.

I wanted to encourage the network to reimagine lost detail, so decided to give it a challenge by training an 8x upscale, using pairs at 32x32 and 256x256 like the examples above. It worked surprisingly well; I could feed the trained network with a variety of images at 32x32 and, in most cases, get something interesting out the other end.

Blocky pixels of red, black, and pale grey.
Input - 32x32
The red, black, and grey pixels reimagined as soft, painterly red blooms over pale rock-like forms.
Output - 256x256
Blocky pixels of white, black, pink, and cyan.
Input - 32x32
The white, black, and cyan pixels reimagined as smooth, swirling white shapes against black.
Output - 256x256

With that working, and a bit more code on top, I could generate larger outputs by iteratively rendering a grid of overlapping 256x256 cells. The slides below show the first few stages of the process for two different inputs.

A coarse grid of black, white, and lilac pixels arranged in rough horizontal rows.
Abstracts 006 (source)
Blurry rows of wavy black and white bands with a fuzzy, noisy texture.
Abstracts 006 (1x)
Wavy black and white bands, slightly sharper, with hints of yellow and orange.
Abstracts 006 (2x)
Crisper wavy bands of dark navy and white with painterly edges.
Abstracts 006 (4x)
Sharply defined wavy bands of dark navy and white with fine painted detail.
Abstracts 006 (8x)
A coarse grid of grey, pink, and maroon pixels with a pale strip along the top.
Abstracts 020 (source)
A blurry texture of vertical grey stripes with diagonal pink and maroon streaks in the middle.
Abstracts 020 (1x)
Vertical grey columns and diagonal pink forms, now with painterly outlines.
Abstracts 020 (2x)
Rock-like grey columns and folded pink shapes with more defined edges and green tints.
Abstracts 020 (4x)
Detailed rock-like columns in grey, green, and pink with dark outlines throughout.
Abstracts 020 (8x)

The final versions of these examples, along with the rest of the Abstracts collection, can be found in the project gallery.

Using pix2pix in this way is something I’ll likely continue experimenting with, but it’s slow going. The time between making a change to getting a result from a trained network does limit the amount of experimentation I do with the core code, so I have tended to focus on trying different types of image pairs to see what works, what doesn’t, and what ends up making the weird, otherworldly scenes I seem drawn to.

Wedding Present

Not long after working on the abstracts I thought I could use a similar process to generate some interesting, and hopefully not too horrific, portraits as presents for my brother’s upcoming wedding.

Using images of their faces from social media I only had a couple of hundred sources to work with and wasn’t sure there would be enough variation to train decent models. On the upside, training would only take a couple of days to complete.

I used three models in the end; one designed to help generate new mixes of their faces and a further two to upscale, or ‘up-imagine’, those outputs. I’m sparing you from the cavalcade of ever-so-nightmarishly malformed faces that were generated along the way. Suffice to say, there was a lot of manual picking and re-generating to whittle a few thousand down to the four I ended up selecting.

A painterly, generated face with one clear eye on the left dissolving into dark green textures, between bands of vertical glitch streaks.
Wedding Present - A
A generated portrait in reds and pinks where several overlapping faces merge, with one dark eye and lips near the centre.
Wedding Present - B
A generated close-up in pink and teal blending a woman's smiling face with a bearded man's face.
Wedding Present - C
A dark, generated portrait of a face with dark hair, its features folded and doubled.
Wedding Present - D

After a little editing they were ready to be printed and framed before handing over to the happy couple at the wedding.