Differentiable Vector Graphics Rasterization for Editing and Learning

 Tzu-Mao Li Michal Lukáč Michaël Gharbi Jonathan Ragan-Kelley MIT CSAIL Adobe Research Adobe Research MIT CSAIL

We introduce a differentiable rasterizer for vector graphics that bridges the raster and vector domains through backpropagation. Differentiable rasterization enables many novel vector graphics applications. (a) Interactive editing that locally optimizes for image-space metrics, such as opacity, under geometric constraints. (b) A new painterly rendering technique by fitting random Bézier curves to a target image. (c) Improving state of art image vectorization result. (d) Editing vector graphics using potentially non-differentiable raster image processing operators, such as seam carving [Avidan et al. 2007] for image retargeting. (e) Training a variational autoencoder to generate vector MNIST digits and adding stylized strokes as postprocessing.

Abstract

We introduce a differentiable rasterizer that bridges the vector graphics and raster image domains, enabling powerful raster-based loss functions, optimization procedures, and machine learning techniques to edit and generate vector content. We observe that vector graphics rasterization is differentiable after pixel prefiltering. Our differentiable rasterizer offers two prefiltering options: an analytical prefiltering technique and a multisampling anti-aliasing technique. The analytical variant is faster but can suffer from artifacts such as conflation. The multisampling variant is still efficient, and can render high-quality images while computing unbiased gradients for each pixel with respect to curve parameters.

We demonstrate that our rasterizer enables new applications, including a vector graphics editor guided by image metrics, a painterly rendering algorithm that fits vector primitives to an image by minimizing a deep perceptual loss function, new vector graphics editing algorithms that exploit well-known image processing methods such as seam carving, and deep generative models that generate vector content from raster-only supervision under a VAE or GAN training objective.

Introduction Video

SIGGRAPH Asia Presentation

Publication

Tzu-Mao Li, Michal Lukáč, Michaël Gharbi, Jonathan Ragan-Kelley
Differentiable Vector Graphics Rasterization for Editing and Learning.
ACM Transactions on Graphics 39(6) (Proceedings of ACM SIGGRAPH Asia 2020)
BibTeX

Errata

In Equation 6 in the paper, the line integral should have measure dp(t) instead of dt (or if we are keeping the dt measure, we should include the Jacobian ||p'(t)||). In Equation 7, we should use function g instead of function f.

More precisely, the line integral at Equation 6 should read: $$\int_{\partial A_i(\Theta)} \left(\frac{\partial p(t)}{\partial \Theta} \cdot n(t)\right) g(p(t)) d{\color{Red}p(t)}$$

And Equation 7 should read: $$\int_{\partial A_{i}(\Theta)} \left( \nabla_{\Theta} p \cdot n \right) {\color{Red}g(p(t))} d{\color{Red}p(t)} \approx \\ \frac{1}{N} \sum_{j} \frac{\left(\nabla_{\Theta} p_j \cdot n_j \right) \left( {\color{Red}g}(p_j + \epsilon n_j) - {\color{Red}g}(p_j - \epsilon n_j) \right) }{P(p_j|c)P(c)}$$