Introduction

As we continue our project in creating a serverless Video on Demand (VoD) system, this segment, titled "Converting to Stream Format," plays a pivotal role. Here, we will delve into the mechanics of managing events triggered by uploads to Amazon S3. Specifically, our focus will be on initiating conversion jobs that transform these uploaded videos into a streaming format. This process is integral to ensuring that our VoD system not only stores videos but also prepares them for seamless streaming to end users.

This part of the series is essential as it bridges the gap between simple video storage and the actual streaming of video content, a core functionality in any VoD platform. By automating the conversion process, we ensure efficiency and readiness of content for users, fulfilling the promise of a robust and responsive VoD service.

The series is structured as follows:

• Part 1 - Architecture Overview: We lay the groundwork, detailing the architecture and AWS services involved in our serverless VoD solution.

• Part 2 - Secured File Upload: We'll cover the video upload process, how to securely upload to s3 bucket using signed url

• Part 3 - Converting to Stream Format: This part will explain how to manage events from s3 upload to trigger conversion job to transform them into streaming format

• Part 4 - Saving, Retrieving, and Playing: We will go through the saving and retrieval of video links, then play them using HLS Player on Chrome

• Part 5 - Securing the Streaming: The final part will focus on securing the application to ensure that only authenticated users can access the videos.

Process Overview

In our approach, we employ AWS services effectively to automate the process. When a video is uploaded to an S3 bucket, it triggers an event that activates a Lambda function. This function is then responsible for invoking AWS Media Convert, which undertakes the critical task of converting the video file into a format suitable for streaming. The converted files are subsequently saved back into an S3 folder, with the folder name mirroring the original file name, thereby maintaining a coherent structure.

The process is shown in the diagram and in summary:

1. Newly uploaded files to s3 will trigger an event
2. This event will invoke a lambda function
3. The lambda function will invoke aws media convert
4. AWS Media convert will read the file from s3 and convert it
5. Once conversion is done, the converted files will be saved into s3 folder where the folder name is the original file name

Create lambda function

1. Start by Creating a New Function in AWS Lambda. (If you are unfamiliar with creating lambda functions then please visit Part 2 of this series)
2. Name this function "start-media-convert".
3. Choose Python as the Runtime, Python is suitable for handling the scripting required in this process.
4. Insert the Python Code, this code will handle the event from S3 and start the media conversion process.

import boto3
import json


MEDIA_CONVERT_ENDPOINT_URL = 'Set this up through tutorial below'
ROLE_ARN = 'Set this up through tutorial below'
QUEUE_ARN = 'Set this up through tutorial below'


mediaconvert_client = boto3.client(
    'mediaconvert',
    region_name='us-east-1',
    endpoint_url=MEDIA_CONVERT_ENDPOINT_URL
)


def generate_parameters(bucket, key):
    s3_output_object_path = key.split('.')[0]
    s3_input_location = f's3://{bucket}/{key}'
    s3_output_location = f's3://{bucket}/{s3_output_object_path}/'
    
    base_input = {
        'VideoSelector': {
            'ColorSpace': 'FOLLOW',
            'Rotate': 'AUTO',
        },
        'FilterEnable': 'AUTO',
        'PsiControl': 'USE_PSI',
        'FilterStrength': 0,
        'DeblockFilter': 'DISABLED',
        'DenoiseFilter': 'DISABLED',
        'TimecodeSource': 'EMBEDDED',
        'FileInput': s3_input_location,
    }
    
    base_output = {
        'ContainerSettings': {
            'Container': 'M3U8',
            'M3u8Settings': {},
        },
        'VideoDescription': {
            'CodecSettings': {
                'Codec': 'H_264',
                'H264Settings': {
                    'MaxBitrate': 5000000,
                    'RateControlMode': 'QVBR',
                    'SceneChangeDetect': 'TRANSITION_DETECTION',
                },
            },
        },
        'OutputSettings': {
            'HlsSettings': {},
        },
        'NameModifier': 'stream',
    }
    
    job_settings = {
        'Queue': QUEUE_ARN,
        'UserMetadata': {
          'Customer': 'Amazon',
        },
        'BillingTagsSource': 'QUEUE',
        'Role': ROLE_ARN,
        'Settings': {
            'OutputGroups': [
                {
                    'CustomName': 'hls',
                    'Name': 'Apple HLS',
                    'Outputs': [base_output],
                    'OutputGroupSettings': {
                        'Type': 'HLS_GROUP_SETTINGS',
                        'HlsGroupSettings': {
                            'SegmentLength': 10,
                            'Destination': s3_output_location,
                            'MinSegmentLength': 0,
                        },
                    },
                },
                {
                    'CustomName': 'poster-frame',
                    'Name': 'File Group',
                    'Outputs': [
                        {
                            'ContainerSettings': {
                                'Container': 'RAW',
                            },
                            'VideoDescription': {
                                'CodecSettings': {
                                    'Codec': 'FRAME_CAPTURE',
                                    'FrameCaptureSettings': {
                                        'MaxCaptures': 1,
                                        'Quality': 20,
                                    },
                                },
                            },
                            'Extension': 'jpg',
                        },
                    ],
                    'OutputGroupSettings': {
                        'Type': 'FILE_GROUP_SETTINGS',
                        'FileGroupSettings': {
                            'Destination': s3_output_location,
                        },
                    },
                },
                {
                    'CustomName': 'poster-thumbnail',
                    'Name': 'File Group',
                    'Outputs': [
                        {
                            'ContainerSettings': {
                                'Container': 'RAW',
                            },
                            'VideoDescription': {
                                'CodecSettings': {
                                    'Codec': 'FRAME_CAPTURE',
                                    'FrameCaptureSettings': {
                                        'MaxCaptures': 1,
                                        'Quality': 5,
                                    },
                                },
                            },
                            'Extension': 'jpg',
                        },
                    ],
                    'OutputGroupSettings': {
                        'Type': 'FILE_GROUP_SETTINGS',
                        'FileGroupSettings': {
                            'Destination': f'{s3_output_location}thumbnail',
                        },
                    },
                },
            ],
            'AdAvailOffset': 0,
            'Inputs': [base_input],
            'TimecodeConfig': {
                'Source': 'EMBEDDED',
            },
        },
    }
    
    return job_settings

def lambda_handler(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']
    print(bucket)
    print(key)
    job_settings = generate_parameters(bucket, key)
    response = mediaconvert_client.create_job(**job_settings)
    return response['ResponseMetadata']

5. After ensuring the code is correctly set, deploy the function.
6. Note that we need to replace the values of the following variables

MEDIA_CONVERT_ENDPOINT_URL = ''
ROLE_ARN = ''
QUEUE_ARN = ''

Update Lambda variables - Media Convert

Note that this is better placed in the environment variables section or using aws secrets, but for simplicity sake, we will simply add it to the function itself

1. Go to media convert -> Queues, there should be a "Default" queue under the On-Demand queues, if it is missing then simply "Create Queue".
2. Click on the default queue and copy the ARN under Queue information (e.g. arn:aws:mediaconvert:us-east-1:[aws-account-id]:queues/Default). and copy it under the variable QUEUE_ARN
3. Now on the left panel, click Account and copy the API endpoint (e.g. https://vasjpylpa.mediaconvert.us-east-1.amazonaws.com).

Update Lambda variables - Iam Role

1. Let's create an IAM role that allows Media Convert to access API Gateway and S3.
2. Go to IAM -> Roles -> Create role
3. Select AWS Service as entity Type and Media Convert as Use Case
4. Click next, this will show the permissions screen and simply click next again as we will just leave it as default
5. Under Role name, put media-convert-role then click "Create role"
6. Now find that role again and click it.
7. On the role details page, copy the role arn (e.g. arn:aws:iam::[aws-account-id]:role/media-convert-role)
8. Go back to the start-media-convert lambda function and paste the arn copied above to the ROLE_ARN variable of our lambda function
9. Click Deploy to update the function

Update Lambda Permission

1. Now go back to IAM -> Roles, we will need to make sure that our lambda function has access to aws media convert
2. If you keep everything as default when creating the lambda function above, then you should find a role with the name "start-media-convert-[random-chars]"
3. Click that role and click on Add permissions -> Attach policies.
4. Find and check AWSElementalMediaConvertFullAccess then click "Add Permissions".

We are now done with everything that this lambda function will need to execute

Triggering the Conversion

To trigger the conversion, we will setup S3 Event Notifications to filter .mp4 files and for each new .mp4 file, it will notify the lambda function to run.

1. Go back to s3 and click on the bucket we used on part 2 (this tutorial is using "cloudfleet-vod-bucket")
2. Go to Properties -> Event Notifications -> Create event notification
3. Fill out the details with the following information and leave the rest to default:
   • Event name - Start Media Conversion
   • Suffix - .mp4
   • Event types - Check "All object create events"
   • Destination - Lambda function
   • Choose from your Lambda functions -> Find the "start-media-convert" from the lambda dropdown
4. Save Changes

Verification

To check if everything is working so far then:

1. Get an upload url through the api gateway endpoint shown in part 2 (in my case i will use: https://es8m6m6ezd.execute-api.us-east-1.amazonaws.com/dev/upload)
2. Copy the upload url from step 1 and create PUT request with binary format through that url.
3. Go to AWS Elemental MediaConvert -> Jobs
4. You should see your first conversion to have "IN PROGRESS" status, once it is done then it will be "COMPLETE". If there is an error then make sure you only uploaded .mp4 and you followed all permissions setup above.
5. Once you see "COMPLETE", go back to the bucket your bucket. There will be a folder that is named the same as the filename ([uuid].mp4 filename will have a [uuid]/ folder counterpart).

Conclusion

That's it, we just finished the conversion part of the series. On the next part, we will show how we can save the file links and fetch the data from dynamo db and how to see our video playing on Chrome through an HLS Player Extension