Getting Started: How to Use AWS S3 Sync for File/Directory Syncing from Local Servers

Aws s3 sync command is a command line utility which helps us to sync folders and files between our local system with AWS s3 bucket using Aws CLI. We are all familiar with the Rsync command line utility, which syncs folders and files between Linux servers.
If you are not familiar with rsync read our blog How to use the Rsync command in Linux.

Using this utility we can store our website files and database backup copies in an s3 bucket and resync it to our local system/server whenever in case of recovery

What is AWS s3?
  • Aws s3 is called a simple storage service it’s a well-liked storage service that’s scalable, provides high-speed solutions, and is a web-based cloud storage service
  • Aws s3 has many features and it’s mainly used as a storage service where we can store large backups it works as a pay-as-you-go service.
  • Aws s3 stores objects inside the s3 bucket. We can create s3 buckets and storage backup and other objects inside them.

To store an object in Amazon S3, you create a bucket and then upload the object to a bucket. When the object is in the bucket, you can open it, download it, and move it. When you no longer need an object or a bucket, you can clean up your resources.

What is AWS CLI?

Aws is a CLI tool which helps to manage services inside the AWS cloud.
We can download Aws CLI in our local system using package managers and interact with various AWS services using various AWS CLI commands.

Now we got a basic understanding of AWS s3 and AWS CLI next we are going with an example and learn how to sync a file/folder from the local system to the AWS s3 bucket using AWS CLI.

Steps involved

Create an S3 bucket in the AWS account.
Create an AWS IAM user with administrative access to the AWS account.
Install the AWS CLI on your local system and configure the AWS IAM profile.
Sync the files/folder to the bucket using the AWS s3 sync command line utility.

We are going through each step one by one

1:Create an S3 bucket in the AWS account.

Step1: Login to your AWS account and search for s3 in the search bar

Step 2: Select buckets and click on create a bucket on the right side of the dashboard

Under the general configurations

  • Type a required bucket name in my case in choose s3testfors3sync.
  • AWS Region–Selection your region.
  • Object Ownership--ACLs disabled (recommended).
  • Block Public Access settings for this bucket–block all public access.
  • Bucket Versioning-choose as you wish.
  • Tags(0) – optional.
  • Default encryption-enable it you need encryption.

Finally, click on the create bucket option in order to create the s3 bucket.

2: Create an AWS IAM user with administrative access to the AWS account.

We need to setup s3 sync permission to our IAM account in order to sync files to s3

  • Search IAM in the AWS search console.
  • Select the users and click on add user.
  • Under user details select any user name enable programmatically and AWS Management Console access.
  • Attach permission for this account you can either create a policy or directly attach any existing AWS permission for this demo I am attaching administrative access.
    It is not mandatory to attach administrative permission you need to only attach s3 full access to do this operation
  • Add any tags that are optional
  • Review and create the user download the access and secret key from the download .csv option and save it in your local system.
3: Install the AWS CLI and Configure IAM profile

Here I am using the ubuntu system hence installing AWS CLI using apt package management.

sudo apt install awscli

Configure your IAM profile in the machine using AWS configure command

aws configure

Enter the IAM access,secret key for the IAM account which is saved before and select the region.

In order to list your s3 bucket run this command this will list the bucket which you created earlier.

aws s3 ls

If you have done all the previous steps correctly your bucket will be listed here.

4:Sync files from local to s3 bucket

Step 1: Create a backup.tar file which needs to push to AWS s3.

Step 2: Here I have a backup file in the folder name backup so I am syncing my backup folder with the s3 bucket.

Syntax

aws s3 sync <localpath> s3://<name of your s3 bucket>

aws s3 sync backup s3://s3testfors3sync

Verifying the backup.tar.gz on the AWS s3 bucket.

That is super easy also we can resync the objects from AWS s3 to our local machine.

syncing objects from s3 to the local server

Suppose I have deleted the backup file from my local system I can resync it from the AWS s3 bucket using the command.

Syntax

aws s3 sync s3://<bucketname> <localpath>
aws s3 sync s3://s3testfors3sync backup

Here I have deleted the backup from local system and resynced it using AWS s3 sync.

This is how we can store the backup in AWS s3 and use it whenever we require.

Summary:
In this blog, we have learned how to store backups of files and folders in AWS s3 using aws s3 sync CLI.

Leave a Reply

Your email address will not be published. Required fields are marked *