Fully-Automated AWS S3 Video Redaction¶

This example covers the steps needed to set up a fully-automated redaction workflow where videos that are placed in an "input" AWS S3 bucket will be automatically redacted and sent to an "output" S3 bucket. This example is fairly high-level and doesn't cover error cases or retries, but it should be a good starting point.

Prerequisites¶

You must be familiar with AWS and its Management Console.
You must have an AWS account with the ability to create new S3 buckets, IAM Policies and Users, Lambda functions, and an EC2 instance to run the Redactor service.
You must have a Sighthound Redactor serial number to activate the service. Contact redactor@sighthound.com if you need this.

Create S3 Buckets¶

It is highly recommended to create separate S3 buckets for the input and output files. Only store input media in the input bucket (no logs, miscellaneous files, etc.). Since we'll be using Object Created events to trigger a Lambda function, you'll need to be careful NOT to write files back to the input bucket and cause an infinite loop. That could be a very expensive mistake.

For the purpose of this example, I'm going to refer to the input S3 bucket name as INPUT_BUCKET and the output as OUTPUT_BUCKET. Those names do not meet AWS's naming convention and don't point to real buckets, so create your own and replace the example names with yours.

Log in to the AWS Console and follow these steps:

Go to the S3 page and click the Create Bucket button
Enter bucket name. I suggest appending -input or -output to make it clear which is which
Confirm the AWS Region is the same as where your Redactor server will be running
Keep all other default settings
Click Create bucket
Repeat for the other bucket

Create IAM Policy for S3 Access¶

We will create an IAM Policy that grants read-only access to INPUT_BUCKET and read/write to OUTPUT_BUCKET.

Go to IAM page on AWS
Click Policies on left nav
Click Create Policy
Select the JSON tab
Paste the JSON block below into the Policy Editor JSON field
Replace INPUT_BUCKET and OUTPUT_BUCKET in the JSON with your actual input and output S3 bucket names
Click Next button
Give your policy a name (E.g. sighthound-redactor-s3-bucket-access) and a description (E.g. Redactor S3 Bucket Access) .

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowReadAccessToInputBucket",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::INPUT_BUCKET",
                "arn:aws:s3:::INPUT_BUCKET/*"
            ]
        },
        {
            "Sid": "AllowReadWriteAccessToOutputBucket",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectVersion",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::OUTPUT_BUCKET",
                "arn:aws:s3:::OUTPUT_BUCKET/*"
            ]
        }
    ]
}

Create IAM User for Redactor Service¶

We will need to create a new IAM user in order to generate AWS Access Keys for the Redactor service to use. The IAM Policy we previously we created will be assigned to this user so it can access the S3 buckets.

Go to the IAM page on AWS
Click Users on left navigation
Click Add Users at top right
Create a username (e.g sighthound-redactor-s3-demo)
Do NOT click the "provide user access to the AWS Management console. Keep it unchecked.
Click the Next button
Under Permissions Options, select the "Attach Policies Directly" option
Under Permissions policies, click the search field and type the IAM Policy name you created earlier (e.g. sighthound-redactor-s3-bucket-access)
Check the box to the left of that policy when it appears in the search results
Click the Next button at the bottom right
Review your changes and click the Create User button at the bottom right.

Generate AWS Access Keys for IAM User¶

The Redactor service will need to be configured with the IAM User's access keys so that it can access the S3 buckets as allowed in the IAM Policy.

On the IAM Users page, search for the user you just created (e.g. sighthound-redactor-s3-demo) and click on it when it appears in search results
Click the Security Credentials tab
Scroll down to the Access Keys section and click "Create access key" button
On the Access Key best practices page, scroll down and click the "Other" option and click the "Next" button
Type a short description for the key's use (e.g. Redactor S3 Access)
Click the Create Access Key button
Click the "Download .csv file" button to download the credentials. This will be your only chance to see the secret access key.
Click the Done button

Create an EC2 instance for Redactor¶

(This section is under construction and left as an exercise for the reader.)

Redaction is a very resource-intensive operation, so choose the best instance type that your budget allows. We highly recommend running Redactor on an instance with an NVIDIA GPU and have found that AWS's g4dn.2xlarge offers the best price-to-performance ratio. You can choose an instance that does not have a GPU, but the auto-detection speed will be drastically slower.

When creating the instance, we recommend choosing Ubuntu 18 or newer. Redactor can run on Windows, but there are additional licensing costs on AWS for it. This guide will assume you're on Linux.

Redactor runs on port 9000 by default, so you'll need to make sure the EC2 Security Group assigned to that instance allows access to this port number.

Once the instance is created follow the instructions to set up Docker and install the NVIDIA drivers (if you chose a GPU instance).

Docker¶

Ensure your system has a recent version of Docker Engine and Docker Compose. You can either manually install it for your specific operating system by following the directions on Docker's site, or you can run the convenience script that Docker provides:

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

NVIDIA (optional)¶

If you have an NVIDIA GPU, install the latest drivers and install the NVIDIA Container Toolkit to give the Docker containers access to the GPU.

Edit docker-compose.yml and uncomment the runtime: nvidia line for both the videoapi and imageapi services to enable GPU acceleration.

Sighthound Redactor Installation¶

You will be provided with a sighthound-redactor-s3-demo.zip file. Contact redactor@sighthound.com if you need this.

Copy the zip file to the EC2 instance and extract it on the instance. This will create a folder named sighthound-redactor-s3-demo which contains a docker-compose.yml file.
Open docker-compose.yml in an editor (e.g. nano, vim, etc.)
Change the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY values to match the IAM Access Keys you created and downloaded above.
Make sure the AWS_DEFAULT_REGION value matches the region you selected when creating the S3 buckets.
Save the changes.

Initial Redactor Setup¶

In a terminal connected to your Redactor instance, cd into the sighthound-redactor-s3-demo folder
Start Redactor by typing: docker-compose up -d
Open a web browser and go to http://redactor-instance-ip:9000 to be redirected to the Redactor Admin page. (Replace redactor-instance-ip with your instance's actual ip address)
Check the "enabled" box for Local Authentication and create your first admin user
The service will automatically restart after you save the user
Log back in with the newly created user
Click the Licensing tab, enter your Redactor serial number, and click the Add button
When you receive the "license activated" message, the server is now ready for use

Test Redactor Server¶

To make sure things are working, run a quick test with the browser:

If you're still on the Redactor Admin page, click the blue "Redactor" button at the top right. Otherwise, open a browser and go to http://redactor-instance-ip:9000
Click the "Add Media" button at the top right and select a short video from your system. If you need one: https://sh-io-downloads.s3.amazonaws.com/videos/garage-top.mp4
Once the video is loaded, click the Auto Detect button at the top right, select Faces and License Plates, and click the Submit button
The processing will take between 10 - 20 seconds, depending on if your EC2 has a properly configured GPU or not.
Once the processing completes, you should see boxes over the license plates and the face in the video.
Click the Render and Export button at the top right to create a redacted video
If everything looks good, the system is working as expected.

Test API with S3 output bucket¶

The curl command below uses a video that's hosted by Sighthound for the inputUri. Make sure to replace OUTPUT_BUCKET in the snippet with your actual output bucket's name. If the redacted video does not appear in the output S3 bucket, review the Troubleshooting section at the bottom of this document for tips on where to find the logs.

curl --location --request POST 'http://redactor-instance-ip:9000/api/v1/videos:process' \
--header 'Content-Type: application/json'  \
--data-raw '{
  "inputUri": "https://sh-io-downloads.s3.amazonaws.com/videos/garage-top.mp4",
  "outputUri": "s3://OUTPUT_BUCKET/test1/garage-top-redacted.mp4",
  "features": ["FACE_DETECTION", "LICENSE_PLATE_DETECTION", "MEDIA_RENDERING"]
}'

Create an AWS Lambda Function¶

This Lambda function will be triggered every time a new file is uploaded to your input bucket. It will parse the event and then make an API call to the Redactor API to auto-detect faces and license plates, and then export a redacted version of the video to your output bucket.

Open the AWS Lamda page
Click the Create Function button at the top right
Keep the "Author from scratch" selected
Give your function a name (e.g. sighthound-redactor-api-on-object-created)
Select your runtime. For this example, we'll use Node.js 18.x
Keep everything else set to defaults and click "Create Function" button
Modify the code below (REDACTOR_API_URI and OUTPUT_BUCKET_NAME) and paste it into the index.mjs block, overwriting the default code
Click the Deploy button.
Click the Add Trigger in the "Function Overview" section
Under "Select a source", type S3 and pick it.
In the Bucket search field, enter the name of your INPUT bucket and select it.
Ensure that "All object create events" is added to the Event Types
Optional: enter a prefix (folder name) in the INPUT bucket that will trigger the events. If you keep this blank, any files put into any folders (or the root) of the S3 bucket will trigger the Lambda function, which may be fine for most use cases.
IMPORTANT: Read the Recursive Invocation notice and check the box that you understand that it's possible to create a recursion with the Lambda function.
Click the Deploy button again if changes were made.

const REDACTOR_API_URI = 'http://redactor-instance-ip:9000/api/v1/videos:process'; // Update with your Redactor server IP address
const OUTPUT_BUCKET_NAME = 'YourOutputBucketName'; // Replace with your output bucket name

export const handler = async (event) => {
    console.log('Received S3 event:', JSON.stringify(event, null, 2));
    const s3Event = event.Records[0].s3;
    const inputBucketName = s3Event.bucket.name;
    // Object key may have spaces or unicode non-ASCII characters. Must decode.
    const objectKey = decodeURIComponent(s3Event.object.key.replace(/\+/g, " "));

    // Check if the event is for a folder
    if (objectKey.endsWith('/')) {
        console.log('Event is for a folder. Skipping POST request.');
        return;
    }

    const timestamp = Date.now();

    const requestBody = JSON.stringify({
        inputUri: `s3://${inputBucketName}/${objectKey}`,
        outputUri: `s3://${OUTPUT_BUCKET_NAME}/${objectKey}-${timestamp}-redacted.mp4`,
        features: ['FACE_DETECTION', 'LICENSE_PLATE_DETECTION', 'MEDIA_RENDERING']
    });

    const options = {
        method: 'POST',
        body: requestBody,
        headers: {
            'Content-Type': 'application/json'
        }
    };

    try {
        /*global fetch*/
        const response = await fetch(REDACTOR_API_URI, options);
        const responseBody = await response.text();
        console.log('Redactor API Response:', responseBody);
        return;
    } catch (error) {
        console.error('Error:', error);
        throw error;
    }
};

After this is done, you should be able to drop a video into your input bucket and wait a few moments for it to complete.

Troubleshooting¶

Since there are quite a few moving parts to this example, it may be helpful to review the various logs to make sure things are working (or not) as expected.

The first place I would look is in Lambda's execution logs. You can find these by opening the Lambda function you created above in the AWS Management Console:

Click the Monitor tab in the Lambda function's detail page.
Click the View Cloudwatch Logs button
Select the first item you see listed in the Log Streams section
You should see some entries in the Log Events table. About 4-5 entries are created every time the Lambda is invoked.
Click on each row to expand them for more detail. If any have errors, that means the Lambda failed and didn't send the request to Redactor.

Next, you can monitor the Redactor logs:

Using a terminal, ssh into the Redactor instance.
cd into the sighthound-redactor-s3-demo folder
"tail" the logs by running sudo tail -f volumes/redactor/logs/main.log
Make another API request or drop a file into your input bucket to see if anything updates in the logs

If you run into any issues, send an email to support@sighthound.com