Aws s3 sync command is a command line utility which helps us to sync folders and files between our local system with AWS s3 bucket using Aws CLI. We are all familiar with the Rsync command line utility, which syncs folders and files between Linux servers.
If you are not familiar with rsync read our blog How to use the Rsync command in Linux.
Using this utility we can store our website files and database backup copies in an s3 bucket and resync it to our local system/server whenever in case of recovery
What is AWS s3?
- Aws s3 is called a simple storage service it’s a well-liked storage service that’s scalable, provides high-speed solutions, and is a web-based cloud storage service
- Aws s3 has many features and it’s mainly used as a storage service where we can store large backups it works as a pay-as-you-go service.
- Aws s3 stores objects inside the s3 bucket. We can create s3 buckets and storage backup and other objects inside them.
To store an object in Amazon S3, you create a bucket and then upload the object to a bucket. When the object is in the bucket, you can open it, download it, and move it. When you no longer need an object or a bucket, you can clean up your resources.
What is AWS CLI?
Aws is a CLI tool which helps to manage services inside the AWS cloud.
We can download Aws CLI in our local system using package managers and interact with various AWS services using various AWS CLI commands.
Now we got a basic understanding of AWS s3 and AWS CLI next we are going with an example and learn how to sync a file/folder from the local system to the AWS s3 bucket using AWS CLI.
Steps involved
We are going through each step one by one
1:Create an S3 bucket in the AWS account.
Step1: Login to your AWS account and search for s3 in the search bar
Step 2: Select buckets and click on create a bucket on the right side of the dashboard
Under the general configurations
- Type a required bucket name in my case in choose s3testfors3sync.
- AWS Region–Selection your region.
- Object Ownership--ACLs disabled (recommended).
- Block Public Access settings for this bucket–block all public access.
- Bucket Versioning-choose as you wish.
- Tags(0) – optional.
- Default encryption-enable it you need encryption.
Finally, click on the create bucket option in order to create the s3 bucket.
2: Create an AWS IAM user with administrative access to the AWS account.
We need to setup s3 sync permission to our IAM account in order to sync files to s3
- Search IAM in the AWS search console.
- Select the users and click on add user.
- Under user details select any user name enable programmatically and AWS Management Console access.
- Attach permission for this account you can either create a policy or directly attach any existing AWS permission for this demo I am attaching administrative access.
It is not mandatory to attach administrative permission you need to only attach s3 full access to do this operation
- Add any tags that are optional
- Review and create the user download the access and secret key from the download .csv option and save it in your local system.
3: Install the AWS CLI and Configure IAM profile
Here I am using the ubuntu system hence installing AWS CLI using apt package management.
sudo apt install awscli
Configure your IAM profile in the machine using AWS configure command
aws configure
Enter the IAM access,secret key for the IAM account which is saved before and select the region.
In order to list your s3 bucket run this command this will list the bucket which you created earlier.
aws s3 ls
If you have done all the previous steps correctly your bucket will be listed here.
4:Sync files from local to s3 bucket
Step 1: Create a backup.tar file which needs to push to AWS s3.
Step 2: Here I have a backup file in the folder name backup so I am syncing my backup folder with the s3 bucket.
Syntax
aws s3 sync <localpath> s3://<name of your s3 bucket>
aws s3 sync backup s3://s3testfors3sync
Verifying the backup.tar.gz on the AWS s3 bucket.
That is super easy also we can resync the objects from AWS s3 to our local machine.
syncing objects from s3 to the local server
Suppose I have deleted the backup file from my local system I can resync it from the AWS s3 bucket using the command.
Syntax
aws s3 sync s3://<bucketname> <localpath>
aws s3 sync s3://s3testfors3sync backup
Here I have deleted the backup from local system and resynced it using AWS s3 sync.
This is how we can store the backup in AWS s3 and use it whenever we require.
Summary:
In this blog, we have learned how to store backups of files and folders in AWS s3 using aws s3 sync CLI.
