FFmpeg, SOX, Pandoc and RSVG for AWS Lambda

Gojko Adzic in Serverless 3 minutes

You can now use all the power of FFmpeg, SOX, Pandoc and RSVG to manipulate video, sound files, SVG images and text documents in Lambda functions, with just a few lines of code. We’ve pre-packaged four commonly used file conversion utilities into Lambda layers, which you can use with any serverless framework or deployment utility.

With low on-demand cost and scalability, cloud functions are ideal for file conversions. But for computationally intensive tasks, such as transcoding video, compiled code still rocks, and in most cases the best way of converting files is to just call into a standard Unix utility such as FFmpeg. The basic AWS Lambda container is quite constrained, and until recently it was relatively difficult to include additional binaries into Lambda functions. Lambda Layers make that easy.

A Layer is a common piece of code that is attached to your Lambda runtime in the /opt directory. You can reuse it in many functions, and deploy it only once. Individual functions do not need to include the layer code in their deployment packages, which means that the resulting functions are smaller and deploy faster. For example, at MindMup, we use Pandoc to convert markdown files into Word documents. The actual lambda function code is only a few dozen lines of JavaScript, but before layers, each deployment of the function had to include the whole Pandoc binary, larger than 100 MB. With a layer, we can publish Pandoc only once, so we use significantly less overall space for Lambda function versions. Each code change now requires just a quick redeployment.

And the best part of this is that you can also use Layers published by other people. Here are the four common Unix utility layers: you can build and deploy your own versions easily from Github, or just use the Layers we published directly:

  • FFmpeg: arn:aws:lambda:us-east-1:145266761615:layer:ffmpeg:4 installs /opt/bin/ffpmeg and /opt/bin/ffprobe (Source on GitHub).
  • Pandoc: arn:aws:lambda:us-east-1:145266761615:layer:pandoc:1 installs /opt/bin/pandoc (Source on GitHub)
  • RSVG: arn:aws:lambda:us-east-1:145266761615:layer:rsvg-convert:2 installs /opt/bin/rsvg-convert (Source on GitHub)
  • SOX: arn:aws:lambda:us-east-1:145266761615:layer:sox:1 installs /opt/bin/sox, /opt/bin/lame and /opt/bin/soxi (Source on GitHub)

The layers are published according to the original licenses from the Unix utilities, GPL2. For more information on those binaries and how to use them, check out the original project pages: https://ffmpeg.org/, http://pandoc.org, http://sox.sourceforge.net and https://wiki.gnome.org/Projects/LibRsvg.

For some nice examples of these layers in action, check out these projects:

You can also use some ready-made components in the Serverless Application Repository that convert files using S3 bucket uploads:

How to use Layers in your applications

You can easily attach these layers to your functions using CloudFormation. Just include the Layers property into AWS::Lambda::Function

ConvertFileFunction:
  Type: AWS::Lambda::Function
  Properties:
    Handler: index.handler
    Runtime: nodejs8.10
    CodeUri: src/
    Layers:
      - !Ref LambdaLayerArn

With AWS SAM, you can also use the AWS::Serverless::Function resource

ConvertFileFunction:
  Type: AWS::Serverless::Function
  Properties:
    Handler: index.handler
    Runtime: nodejs8.10
    CodeUri: src/
    Layers:
      - !Ref LambdaLayerArn

With claudia, use the --layers <LambdaLayerArn> option with claudia create or claudia update to attach a layer to a function.

With the Serverless Framework, use the Layers property to link a layer to your service.