Introduction

Have you ever been curious about the technology behind streaming videos on platforms like YouTube, Facebook, and TikTok? These services allow video playback to begin instantly, employing a process known as "buffering" which is a fundamental feature of video on demand (VoD) streaming. In this comprehensive five-part series, we're going to design and implement a serverless architecture for a VoD service. Our system will enable users to upload videos, which we will then convert to HTTP Live Streaming (HLS) format, ensuring compatibility across all HLS-supported video players.

The series is structured as follows:

• Part 1 - Architecture Overview: We lay the groundwork, detailing the architecture and AWS services involved in our serverless VoD solution.

• Part 2 - Secured File Upload: We'll cover the video upload process, how to securely upload to s3 bucket using signed url

• Part 3 - Converting to Stream Format: This part will explain how to manage events from s3 upload to trigger conversion job to transform them into streaming format

• Part 4 - Saving, Retrieving, and Playing: We will go through the saving and retrieval of video links, then play them using HLS Player on Chrome

• Part 5 - Securing the Streaming: The final part will focus on securing the application to ensure that only authenticated users can access the videos.

Part 1 - Architecture Overview and Explanation

We begin by laying the architectural foundation and examining the AWS services that make serverless VoD possible. At the heart of VoD lies the principle of breaking down video content into segments, allowing immediate playback and providing a smooth streaming experience, even on fluctuating internet connections.

AWS Services

Below are the AWS Services we are going to use to create our serverless VOD service

• API Gateway: Acts as the interface for our serverless application, handling client requests securely and efficiently.

• S3 (Simple Storage Service): Stores raw and converted video files, leveraging its vast storage capacity and durability.

• Lambda: Serves as our serverless compute service, executing code in response to various triggers without provisioning or managing servers.

• AWS Elemental MediaConvert: Converts video files into the required HLS format, which is essential for streaming across different platforms.

• AWS CloudFront: Distributes our video content globally, reducing latency and improving the viewer's experience by caching content at edge locations.

• DynamoDB: Provides a managed NoSQL database with seamless scalability and speed, ideal for storing and retrieving metadata associated with our video content.

• Amazon EventBridge: Orchestrates event-driven workflows by connecting application data from various sources and routing them to the appropriate destinations.

System Explanation: The Architectural Workflow for Video on Demand

Here is a breakdown on how the system works

1. Initiating Upload: The process starts when the client requests an upload URL from the API Gateway. This step circumvents the size limitations of direct uploads.

2. Lambda Function & S3 URL: The API Gateway triggers a Lambda function, which creates a pre-signed S3 URL. This URL, granting limited-time bucket access, is returned to the user for the file upload.

3. Uploading via Pre-signed URL: Users proceed to upload their video content to S3 using the pre-signed URL.

4. Media Conversion Trigger: Post-upload, S3 activates another Lambda function to start a media conversion job with AWS Elemental MediaConvert and logs the video metadata in DynamoDB.

5. Conversion Completion Event Handling: AWS EventBridge, with a preset rule, detects the 'conversion complete' event and triggers a Lambda function to update DynamoDB, flagging the media as ready for streaming. The media conversion will also write to the same bucket but under the /video-streams prefix

6. Streaming Request Handling: When a user requests video streaming, the API Gateway fetches all data marked as 'ready for streaming' and provides the manifest file's S3 link.

7. Initial Public Access Setup: Initially, to facilitate streaming, the '/video-streams' folder in S3 is made public.

8. Manifest File-Based Streaming: The media player streams the video by loading the manifest file from the public S3 link.

9. Restricting Access: The S3 bucket is made private again to prevent unauthorized access to the videos.

10. CloudFront Distribution: The API Gateway is updated to return a CloudFront URL for the manifest file, utilizing CloudFront's CDN capabilities for faster content delivery.

11. Signed Manifest File Requests: The client requests the signed manifest file, which now ensures that only authenticated users can access it, maintaining the privacy of the video files listed.

12. Lambda@Edge for URL Signing: When a manifest file is requested, a CloudFront rule triggers a Lambda@Edge function. This function adds signing parameters to each file link in the manifest, allowing access to private content.

13. Buffering and Streaming: Finally, the client's media player buffers and streams the video content, providing a continuous and secure streaming experience.

Conclusion

This article has outlined the first steps in building a serverless architecture for a secure and efficient VoD streaming service. As we progress through the series, we will delve into the nitty-gritty of uploading videos, automatic conversion, handling metadata, and securing content. By the end, you will have a comprehensive understanding of how to implement a serverless VoD solution, tailored to deliver high-quality streaming services with the agility and scalability that modern users demand. Stay with us as we navigate through this technical journey, leveraging cloud technologies to deliver exceptional media streaming experiences.

Stay tuned for the next part of our series to learn how to securely upload files and convert to streaming format!