Transcoding Media Files with AWS

A media file usually consists of a number of pieces - called tracks - which are packed into a container along with metadata information. Each of the tracks is encoded into a specific format. Transcoding is done by extracting those tracks from the container, then decoding, filtering, and encoding again each of them one after another, and finally packing them into a new container.

Why transcode media files

Converting media into different formats has some disadvantages - it almost always lowers the quality and can also be time-consuming. However, there are some reasons why it should be done. If your application lets users upload music or movies, you would probably want to have some control over the uploaded content. Transcoding will help you standardize those files, save their storage space, and ensure that their format is compatible with every specific device or service targeted by your application.

Ways to do transcoding in Rails

The most popular tool for converting media files is FFmpeg - a free and open-source framework which provides several libraries and command line tools allowing you to easily deal with transcoding. There are some gems built on top of it - kind of 'wrappers' for FFmpeg commands, like streamio-ffmpeg, paperclip-av-transcoder, or carrierwave-video. For converting only audio files you can use SoX - another free software for processing media. It is also easy to use with wrapper gems like carrierwave-audio or ruby-sox.

Thanks to these solutions you do not need to use any third party services. They let you transcode data for free, enable many options and provide numerous formats. However, this way of dealing with transcoding can have a lot of drawbacks and will not always be sufficient. Firstly, even if the friendly gems make communicating with conversion software easier, you still have to set up a lot of things by yourself. You'll need to do some research, choose the most optimal options for your case and set them up properly. Secondly, the gems are not always compatible with the most up-to-date versions of underlying conversion software, what can sometimes result in inapplicable usage of the command line. And, probably the most important issue, transcoding on your own machine may have poor performance which can cause some trouble, especially when transcoding a lot of data.

If you don't want to face all these problems and are willing to save time and server resources, then you should try an easier and more reliable solution - Amazon Elastic Transcoder.

Amazon Elastic Transcoder

Amazon Elastic Transcoder is a product which is claimed to be highly secure, cost-effective and elastically scalable. Combined with two more AWS services - S3 for storage and Lambda for computing - it becomes a very comfortable tool for converting media files, easy to use and integrate with a Rails application.

How it works

Amazon offers a vast documentation carefully describing the transcoding workflow, pricing, and setup process. You can find a lot of options and ways of using the Transcoder. Let's see a simple example of how it works and how a Rails application can take advantage of it.

Let's say that your application allows users to upload music, and then listen to it with a built-in player. However, the player requires a specific file format, so you have to somehow get control of your users' uploaded content. To have everything working fine, you'll have to convert every uploaded file to a required format. Using AWS Transcoder along with S3 and Lambda services will limit your application's responsibility only to communicating with the buckets where you'll store the music files. The rest will be handled in the cloud.

You'll need to create two S3 buckets - one for input files, where the original files uploaded by users will be stored, and the second one for transcoded files, ready to be used by your player. Every upload to the input bucket will trigger a lambda function. Lambda will launch a specific transcoding job, which will end up generating a new file in the output bucket.

After having the buckets created, you'll need to set up a transcoding pipeline - a queue-like structure which will be used to run transcoding jobs. Go to Elastic Transcoder service in your AWS console and create a new pipeline - this will require defining the pipeline name, and the input and output buckets.

Next step is creating a function in AWS Lambda service. You'll need to add an S3 trigger, choose your input bucket as the source event and select the 'Object created' option as the event type. In your function code, you'll have to define which pipeline to use and specify the required output format. Here is a very simple example of such a function in Node.js:

'use strict';

var AWS = require('aws-sdk');
var transcoder = new AWS.ElasticTranscoder();

exports.handler = (event, context, callback) => {
  let key = event.Records[0].s3.object.key;
  transcoder.createJob({
    PipelineId: '<your_pipeline_id>',
    Input: {
      Key: key
    },
    Output: {    
      Key: key,
      PresetId: '<your_preset_id>',
    }
  }, function() {
    callback();
  });
};

This function creates a new job for your transcoding pipeline, passing all necessary information - which files to transcode, which preset to use and how to name the output file. Your pipeline ID you'll find in the previously created pipeline details. And the preset ID defines the format of the transcoded file. A list of provided presets is available in Elastic Transcoder documentation.

And that's it! Now your lambda should trigger a transcoding job every time a file is uploaded, and after a while, you should be able to see the newly generated transcoded file in the output bucket.

Summary

If your application needs converting media files, AWS Elastic Transcoder seems to be the best option. Although it is not a free service, the pricing is reasonable - you pay only for what you use, it depends on the duration of the transcoded content. And in return, you get reliable, easy to set up solution. You don't need to worry about performance or keeping codecs up to date - Amazon handles all that perfectly.