Aws Cli is a very powerful tool for working with AWS services. If you have configured the Access key and secrete key on your working machine then you can easily play with AWS services from your local machine.
In this article, we are going to cover the following topics
- How to Copying data between the s3 bucket
- In addition to How to Copy the file to destination bucket with a different name
- How to Copy all/multiple files from source to destination bucket
- And How to Copying data from local machine to s3 bucket
- How to Copy multiple files from local Machine to destination bucket
- How to Copying data from S3 bucket to local Machine
- What is the diffrence between sync and cp copy command
- How to Syncing Data between Two s3 bucket
- How to Syncing Data between Local to s3 bucket
- Also How to Syncing Data between S3 to Local
- How to Move Data between Local to s3 bucket
- How to Move Data between S3 to Local
But before working/accessing the AWS services you need to configure the access key and secrete key. if you have configured the keys then you can skip the following first section and directly jump to the next section but if you haven’t configured then you can use the following section to configure it
How to configure access key and secrete key
To configure the access key and secrete key use the following step
Step1: Open the command prompt
Step2: Type the following command
aws configure
Once execute you need to enter the access key and secrete key
And Congratulation you have successfully completed the AWS sdk setup.
The syntax for working with AWS s3 service from local system or vice versa
To do any operation on s3 or local you need to keep the following syntax in mind
aws s3 action source_location Destination_Location
The explanation of the above command is as follows
AWS s3 stands for you are using the AWS and in that, you want to access the s3 (simple storage service) to perform the operation
Action stand for which operation you want to perform like cp(copy data), mv(move data), sync(sync data)
Source_location means your source location form where you are planning to copy data or move data or sync data
Destination_Location means your destination location where you want to paste data or move data or sync data
How to Copying data between the s3 bucket
The use case here is to copy data between the two s3 buckets. You have the data into the first bucket and now you want to migrate that data into another bucket say the second bucket so in this scenario first_bucket will be your source bucket and second_bucket is the destination bucket. in other words
Source_bucket = first_bucket
Destination_bucket = second_bucket
The syntax will be as follows
aws s3 copy s3://Source_bucket s3://destination bucket
And the command will be
aws s3 copy s3://first_bucket/filename.txt s3://second_bucket/filename.txt
The above command will copy the filename.txt file form the first_bucket and past it into the second_bucket by name filename.txt
How to Copy the file to destination bucket with a different name
If you want to change the file name in destination location after copying data then you just need to put the destination name in your copy command you can use the following command
aws s3 copy s3://first_bucket/filename.txt s3://second_bucket/newfileName.txt
In the above command source file name is filename.txt but once you copied on the destination location file name will be newfileName.txt in this way you can easily change the file name
How to Copy all/multiple files from source to destination bucket
To copy the file from the source to the destination location you need to use the keyword –recursive keyword. this keyword iterates over all the files on a source location and copies the item to destination location. it copy object/file one by one and move to destination lcoation. To do this use the following command
aws s3 copy s3://first_bucket s3://second_bucket/ --recursive
The above command will pick up all files and folders from the first_bucket and paste them into the second_bucket. It will replicate the same structure of file and folder into the destination location
How to Copying data from local machine to s3 bucket
Now here the use-case or requirement is that you have stored some data on your local machine and now you want to copy that data from the local machine to the s3 bucket then you need to use the following command
aws s3 copy C:/user/unixcop.txt s3://second_bucket/unixcop.txt
The above file will copy the file unixcop.txt from the c:/user directory to the second_bucket in s3.
How to Copy multiple files from local Machine to destination bucket
Here if you want to copy all the files from C:/user directory to the s3 bucket say bucket name is second bucket then you need to use the following command
step1: Go to the C:/user directory using the following command
cd C:/user
step2: once you move to the directory it will look something as follows
Step3: Now use the following command to copy all files from the user directory to the s3 bucket
aws s3 cp . s3://second_bucket
once you put the above command and hit the enter it will start copying the file from local to the destination
the dot(.) in the above command tell to the command copy all file from the current directory
How to Copying data from S3 bucket to local Machine
Now If you want to copy data from the s3 bucket to your local machine you need to use the following command
AWS s3 cp s3://second_bucket/unixcop.txt C:/downloads
The above command will copy the unixcop.txt file from the second s3 bucket to the local machine. on local machine it will put data in download folder
What is the difference between sync and cp command
The main difference between this command is sync will only copy files which is new. It will never copy the existing command. sync command will be useful in the perspective of time and cost optimization as it only copies the new file it will save lots of cost but if you use cp instead of sync it will copy all the file which is not the cost optimise.
How to Syncing Data between Two s3 bucket
To sync the data between two buckets use the following command
aws s3 sync s3://fist_bucket/ s3://second_bucket/
Here all data from the first bucket will get synchronized with the second bucket if there are few files which are available in both bucket or we can say the common files they will not get copied only those file are not present in first bucket will get copied to the second file
How to Data between Local to s3 bucket
Now we want to sync the file from the local machine to the s3 bucket then use the following command first you need to give your local path and then provide the destination path i.e s3 bucket path where you are planning to copy to the file
AWS s3 syncC:/downloads s3://second_bucket/
The above command will sync copy all files from the local download folder to the s3 bucket. so destination bucket name will be second_bucket where all our file will go
How to Syncing Data between S3 to Local
you have some data on s3 and you want to download newly added data from the s3 bucket to the local machine for some analysis purpose you can use the following command to download it
aws s3 sync s3://second_bucket/ C:/analytics_unixcop
In the above command it will sync all files from second_bucket to the local machine. In local machine, all data will get the store in analytics_unixcop folder
How to Move Data between Local to s3 bucket
Now you want to move the files from your local system to the s3 bucket use the following command
aws s3 mv C:/unixcopfiles s3://unixcop_bucket
The above command will move all data from unixcopfiles folder to s3 bucket and will delete all file from the local
How to Move Data between S3 to Local
Now you want to move the files from your s3 bucket system to the local system use the following command
aws s3 mv s3://unixcop_bucket C:/localFolder
The above command will move all data from unixcopfiles s3 bucket to the localFolder
NOTE: Be careful when you use the mv(move) command as you will lose all data from the source bucket or source location
Conclusion
In this article, we have covered the most used as command with AWS s3(simple storage service) service.
