How to copy|sync data between s3 to s3 or local using AWS CLI

Aws Cli is a very powerful tool for working with AWS services. If you have configured the Access key and secrete key on your working machine then you can easily play with AWS services from your local machine.

In this article, we are going to cover the following topics

  • How to Copying data between the s3 bucket
  • In addition to How to Copy the file to destination bucket with a different name
  • How to Copy all/multiple files from source to destination bucket
  • And How to Copying data from local machine to s3 bucket
  • How to Copy multiple files from local Machine to destination bucket
  • How to Copying data from S3 bucket to local Machine
  • What is the diffrence between sync and cp copy command
  • How to Syncing Data between Two s3 bucket
  • How to Syncing Data between Local to s3 bucket
  • Also How to Syncing Data between S3 to Local
  • How to Move Data between Local to s3 bucket
  • How to Move Data between S3 to Local

But before working/accessing the AWS services you need to configure the access key and secrete key. if you have configured the keys then you can skip the following first section and directly jump to the next section but if you haven’t configured then you can use the following section to configure it

How to configure access key and secrete key

To configure the access key and secrete key use the following step

Step1: Open the command prompt

Step2: Type the following command

aws configure

Once execute you need to enter the access key and secrete key

Aws configure

And Congratulation you have successfully completed the AWS sdk setup.

The syntax for working with AWS s3 service from local system or vice versa

To do any operation on s3 or local you need to keep the following syntax in mind

aws s3 action source_location Destination_Location

The explanation of the above command is as follows

AWS s3 stands for you are using the AWS and in that, you want to access the s3 (simple storage service) to perform the operation

Action stand for which operation you want to perform like cp(copy data), mv(move data), sync(sync data)

Source_location means your source location form where you are planning to copy data or move data or sync data

Destination_Location means your destination location where you want to paste data or move data or sync data

How to Copying data between the s3 bucket

The use case here is to copy data between the two s3 buckets. You have the data into the first bucket and now you want to migrate that data into another bucket say the second bucket so in this scenario first_bucket will be your source bucket and second_bucket is the destination bucket. in other words
Source_bucket = first_bucket
Destination_bucket = second_bucket

The syntax will be as follows

aws s3 copy s3://Source_bucket s3://destination bucket

And the command will be

aws s3 copy s3://first_bucket/filename.txt s3://second_bucket/filename.txt

The above command will copy the filename.txt file form the first_bucket and past it into the second_bucket by name filename.txt

How to Copy the file to destination bucket with a different name

If you want to change the file name in destination location after copying data then you just need to put the destination name in your copy command you can use the following command

aws s3 copy s3://first_bucket/filename.txt s3://second_bucket/newfileName.txt

In the above command source file name is filename.txt but once you copied on the destination location file name will be newfileName.txt in this way you can easily change the file name

How to Copy all/multiple files from source to destination bucket

To copy the file from the source to the destination location you need to use the keyword –recursive keyword. this keyword iterates over all the files on a source location and copies the item to destination location. it copy object/file one by one and move to destination lcoation. To do this use the following command

aws s3 copy s3://first_bucket s3://second_bucket/ --recursive

The above command will pick up all files and folders from the first_bucket and paste them into the second_bucket. It will replicate the same structure of file and folder into the destination location

How to Copying data from local machine to s3 bucket

Now here the use-case or requirement is that you have stored some data on your local machine and now you want to copy that data from the local machine to the s3 bucket then you need to use the following command

aws s3 copy C:/user/unixcop.txt s3://second_bucket/unixcop.txt

The above file will copy the file unixcop.txt from the c:/user directory to the second_bucket in s3.

How to Copy multiple files from local Machine to destination bucket

Here if you want to copy all the files from C:/user directory to the s3 bucket say bucket name is second bucket then you need to use the following command

step1: Go to the C:/user directory using the following command

cd C:/user

step2: once you move to the directory it will look something as follows

Step3: Now use the following command to copy all files from the user directory to the s3 bucket

aws s3 cp . s3://second_bucket

once you put the above command and hit the enter it will start copying the file from local to the destination

the dot(.) in the above command tell to the command copy all file from the current directory

How to Copying data from S3 bucket to local Machine

Now If you want to copy data from the s3 bucket to your local machine you need to use the following command

AWS s3 cp s3://second_bucket/unixcop.txt C:/downloads

The above command will copy the unixcop.txt file from the second s3 bucket to the local machine. on local machine it will put data in download folder

What is the difference between sync and cp command

The main difference between this command is sync will only copy files which is new. It will never copy the existing command. sync command will be useful in the perspective of time and cost optimization as it only copies the new file it will save lots of cost but if you use cp instead of sync it will copy all the file which is not the cost optimise.

How to Syncing Data between Two s3 bucket

To sync the data between two buckets use the following command

aws s3 syncs3://fist_bucket/ s3://second_bucket/

Here all data from the first bucket will get synchronized with the second bucket if there are few files which are available in both bucket or we can say the common files they will not get copied only those file are not present in first bucket will get copied to the second file

How to Data between Local to s3 bucket

Now we want to sync the file from the local machine to the s3 bucket then use the following command first you need to give your local path and then provide the destination path i.e s3 bucket path where you are planning to copy to the file

AWS s3 syncC:/downloads s3://second_bucket/

The above command will sync copy all files from the local download folder to the s3 bucket. so destination bucket name will be second_bucket where all our file will go

How to Syncing Data between S3 to Local

you have some data on s3 and you want to download newly added data from the s3 bucket to the local machine for some analysis purpose you can use the following command to download it

aws s3 sync s3://second_bucket/ C:/analytics_unixcop

In the above command it will sync all files from second_bucket to the local machine. In local machine, all data will get the store in analytics_unixcop folder

How to Move Data between Local to s3 bucket

Now you want to move the files from your local system to the s3 bucket use the following command

aws s3 mv C:/unixcopfiles s3://unixcop_bucket

The above command will move all data from unixcopfiles folder to s3 bucket and will delete all file from the local

How to Move Data between S3 to Local

Now you want to move the files from your s3 bucket system to the local system use the following command

aws s3 mv s3://unixcop_bucket C:/localFolder

The above command will move all data from unixcopfiles s3 bucket to the localFolder

NOTE: Be careful when you use the mv(move) command as you will lose all data from the source bucket or source location


Conclusion

In this article, we have covered the most used as command with AWS s3(simple storage service) service.

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Latest articles

x