CloudWatch Logs
Amazon CloudWatch Logs let you allow to store, monitor, and access your log files from Amazon EC2 instances, VPC flow logs, Lambda functions, AWS CloudTrail, or other resources.
CloudWatch logs provide several features such as searching by patterns, keywords, date ranges, etc., and CloudWatch logs are classified as “hot storage” as we would be using it for debugging live issues. Hence it provides a lot of features. As compared to other storage options such as S3, it is much costlier. To get the most out of the CloudWatch log keeping the cost under control, it would be ideal for moving the older records to S3 for long term retention.
CloudWatch organizes logs in a log group, and its retention period is set never to expire when a new log group is created, which means it will retain records forever.
What is an accomplishment by using this automation?
Creating a new CloudWatch log group will set the desired retention time for that log group. After that retention time, it will delete all log stream data of the group automatically.
Requirments
- AWS Account
- IAM user of that AWS account (Used IAM user, not from the root account)
- IAM users should be authorized to access services for creating this automation task
How does automation work?
CloudWatch event rule triggers a lambda function whenever a new CloudWatch log group is created. Then the lambda function sets a desired retention time for the CloudWatch log group.
How to create an IAM role?
Here we learned how to set CloudWatch log groups’ retention time automatically. If you have observed, after deploying this automation into an AWS account, it will only set retention time for newly created log groups. We need to set retention time for already existing log groups. You can use our open-sourced NodeJS script to set the retention for all the log groups at once.
How to create a Lambda function?
Create an AWS Lambda function and place the code in that. While creating a lambda function, the Runtime should be on Node.js 8.10 and choose a previously created role in the current position.
How to create CloudWatch event rule?
- Go to services → CloudWatch → Rules → click on Create rule.
- Event Source → choose Event Pattern → select CloudWatch Logs in Service Name, AWS API Call via CloudTrail in Event Type (If you don’t have Trail setup in CloudTrail, do first.
- Create LogGroup in Specific operation(s)→ Targets → select Lambda function → select previously created lambda function → click on Configure details.
- Give Name, Description, and should enable state→ click on Create rule.
The process to create a Lambda function with default function code.
Here create a Node.js Lambda function using the Lambda console. Lambda automatically creates the default code for the function. Now, using sample event data, manually invoke the Lambda function. Lambda runs the function and returns the results. Now, then verify the results, including the logs that your Lambda function created and various Amazon CloudWatch metrics.
Steps to create a Lambda function with the console
Step1: Open the Functions page on the Lambda console.
Step2: Choose the Create function.
Step3: Under Basic Information, do the below steps:
- For Function name, enter my-LambdaFunction.
- Now, for Runtime, confirm that Node.js 12.x is selected.
Step4: Choose the Create function.
Here Lambda creates a Node.js function and executes the role that grants the function permission to upload the logs. The Lambda function assumes the execution role when you invoke your function and uses the execution role to create the AWS SDK credentials and to read data from event sources.
Conclusion
Here we learned how to set CloudWatch log groups’ retention time automatically. If you have observed, after deploying this automation into an AWS account, it will only set retention time for newly created log groups. We need to set retention time for already existing log groups. You can use our open-sourced NodeJS script to set the retention for all the log groups at once.
by Zehntech
0 Comments