# Ensure S3 buckets have custom backup schedule configured

- Source: https://support.icompaas.com/support/solutions/articles/62000208416-ensure-s3-buckets-have-custom-backup-schedule-configured
- Section: [Solutions](https://support.icompaas.com/support/solutions) › [SOC2 Readiness](https://support.icompaas.com/support/solutions/62000111733) › [SOC2 Controls](https://support.icompaas.com/support/solutions/folders/62000166025)
- Last updated: 2026-05-29
- Publisher: iCompaas, Inc. (https://www.icompaas.com)

****Description******:**

Amazon S3 will store and retrieve any amount of data at any time, from anywhere on the web. The policy checks if S3 has controlled storage access and its backups are custom configured.

  

****Remediation:****

**Sync Your S3 Bucket to an EC2 Server Periodically**

This can be easily achieved by utilizing multiple command-line utilities that make it possible to sync a remote S3 bucket to the local filesystem.

[s3cmd](http://s3tools.org/s3cmd)  
At first, `s3cmd` looked extremely promising. However, after trying it on my enormous S3 bucket -- it failed to scale, erroring out with a `Segmentation fault`. It did work fine on small buckets, though. Since it did not work for huge buckets, I set out to find an alternative.

[s4cmd](https://github.com/bloomreach/s4cmd)  
The newer, multi-threaded alternative to `s3cmd`. Looked even more promising, however, I noticed that it kept re-downloading files that were already present on the local filesystem. That is not the kind of behavior I was expecting from the sync command. It should check whether the remote file already exists locally (hash/filesize checking would be neat) and skip it in the next sync run on the same target directory. I opened an issue ([bloomreach/s4cmd/#46](https://github.com/bloomreach/s4cmd/issues/46)) to report this strange behavior. In the meantime, I set out to find another alternative.

[awscli](https://aws.amazon.com/cli/)  
And then I found `awscli`. This is Amazon's official command-line interface for interacting with their different cloud services, S3 included.

![AWSCLI](https://support.icompaas.com/assets/kb/5ddb643abd1f3f33.png)

It provides a useful sync command that quickly and easily **downloads the remote bucket files to your local filesystem**.

$ aws s3 sync s3://your-bucket-name /home/ubuntu/s3/your-bucket-name/

### Benefits:

-   Scalable - supports huge S3 buckets
-   Multi-threaded - syncs the files faster by utilizing multiple threads
-   Smart - only syncs new or updated files
-   Fast - thanks to its multi-threaded nature and smart sync algorithm

### Accidental Deletion

Conveniently, the `sync` command won't delete files in the destination folder (local filesystem) if they are missing from the source (S3 bucket), and vice-versa. This is perfect for backing up S3 -- in case files get deleted from the bucket, re-syncing it will not delete them locally. And in case you delete a local file, it won't be deleted from the source bucket either.

# Setting up awscli on Ubuntu 14.04 LTS

Let's begin by installing `awscli`. There are [several ways](http://docs.aws.amazon.com/cli/latest/userguide/installing.html) to do this, however, I found it easiest to install it via `apt-get`.

$ sudo apt-get install awscli

### Configuration

Next, we need to configure `awscli` with our Access Key ID & Secret Key, which you must obtain from [IAM](https://console.aws.amazon.com/iam/home), by creating a user and attaching the **AmazonS3ReadOnlyAccess** policy. This will also prevent you or anyone who gains access to these credentials from deleting your S3 files. Make sure to enter your S3 region, such as `us-east-1`.

$ aws configure

                ![aws configure](https://support.icompaas.com/assets/kb/26d00c47d414b7d0.png)

### Preparation

Let's prepare the local S3 backup directory, preferably in `/home/ubuntu/s3/{BUCKET_NAME}`. Make sure to replace `{BUCKET_NAME}` with your actual bucket name.

$ mkdir -p /home/ubuntu/s3/{BUCKET\_NAME}

### Initial Sync

Let's go ahead and sync the bucket for the first time with the following command:

$ aws s3 sync s3://{BUCKET\_NAME} /home/ubuntu/s3/{BUCKET\_NAME}/

Assuming the bucket exists, the AWS credentials and region are correct, and the destination folder is valid, `awscli` will start to download the entire bucket to the local filesystem.

Depending on the size of the bucket and your Internet connection, it could take anywhere from a few seconds to hours. When that's done, we'll go ahead and set up an automatic cron job to keep the local copy of the bucket up to date.

### Setting up a Cron Job

Go ahead and create a `sync.sh` file in `/home/ubuntu/s3`:

$ nano /home/ubuntu/s3/sync.sh

Copy and paste the following code into `sync.sh`:

#!/bin/sh

# Echo the current date and time

echo '-----------------------------'
date
echo '-----------------------------'
echo ''

# Echo script initialization
echo 'Syncing remote S3 bucket...'

# Actually run the sync command (replace {BUCKET\_NAME} with your S3 bucket name)
/usr/bin/aws s3 sync s3://{BUCKET\_NAME} /home/ubuntu/s3/{BUCKET\_NAME}/

# Echo script completion
echo 'Sync complete'

Make sure to replace **{BUCKET\_NAME}** with your S3 bucket name, twice throughout the script.

> **Pro tip:** You should use `/usr/bin/aws` to link to the `aws` binary, as `crontab` executes commands in a limited shell environment and won't be able to find the executable on its own.

Next, make sure to `chmod` the script so it can be executed by `crontab`.

$ sudo chmod +x /home/ubuntu/s3/sync.sh

Let's try running the script to make sure it actually works:

$ /home/ubuntu/s3/sync.sh

The output should be similar to this:

                                                    ![sync.sh output](https://support.icompaas.com/assets/kb/768b1a3969df86ab.png)

Next, let's edit the current user's `crontab` by executing the following command:

$ crontab -e

If this is your first time executing `crontab -e`, you'll need to select a preferred editor. I'd recommend selecting `nano` as it's the easiest for beginners to work with.

### Sync Frequency

We need to tell `crontab` how often to run our script and where the script resides on the local filesystem by writing a command. The format for this command is as follows:

m h  dom mon dow   command

The following command configures `crontab` to run the `sync.sh` script every hour (specified via the minute:0 and hour:\* parameters) and to have it pipe the script's output to a `sync.log` file in our `s3` directory:

0 \* \* \* \* /home/ubuntu/s3/sync.sh > /home/ubuntu/s3/sync.log

You should add this line to the bottom of the `crontab` the file you are editing. Then, go ahead and save the file to disk by pressing **Ctrl + W** and then **Enter**. You can then exit `nano` by pressing **Ctrl + X**. `crontab` will now run the sync task every hour.

> **Pro tip:** You can verify that the hourly cron job is being executed successfully by inspecting `/home/ubuntu/s3/sync.log`, checking its contents for the execution date & time, and inspecting the logs to see which new files have been synced.

All set! Your S3 bucket will now get synced to your EC2 server every hour automatically, and you should be good to go. Do note that over time, as your S3 bucket gets bigger, you may have to increase your EC2 server's EBS volume size to accommodate new files. You can always increase your EBS volume size by following [this guide](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-expand-volume.html).

  

****Reference:****

    [](https://aws.amazon.com/getting-started/hands-on/backup-files-to-amazon-s3/)[https://aws.amazon.com/getting-started/hands-on/backup-to-s3-cli/](https://aws.amazon.com/getting-started/hands-on/backup-to-s3-cli/)

## Related compliance guidance

- [SOC 2 compliance: a practical implementation guide for cloud-native SaaS teams](https://support.icompaas.com/compliance/soc-2-trust-services-criteria): The full SOC 2 implementation guide: scoping the Trust Services Criteria, the roadmap to a Type 2 report, and what the auditor samples.
- [ISO/IEC 27001:2022 compliance: a practical implementation guide](https://support.icompaas.com/compliance/iso-27001-2022): Teams that finish SOC 2 usually add ISO 27001 next; the control overlap is large and the guide explains what is genuinely extra.
