AWS Cost Optimization – Cut Down Your AWS Cloud Spendings

Introduction

AWS is value-based pricing that reduces the barrier to enter the cloud. We will introduce the framework for optimizing and managing AWS spend by focusing on paying what you need.
Use Cost Optimization practice in a small environment and as you grow. Establishing these practices in your environment will help you to develop the right process and behavior.
More AWS usage drives more significant infrastructure growth in the innovation, which gives AWS a substantial economy of scale. Thereby lowering the Infrastructure cost and allowing AWS cost saving will continue the renovation to customers. 

Pillars of AWS Cost Optimization

  • Right Size of your Instance 
  • Increase Elasticity 
  • Choose the right pricing model
  • Match storage to usage 
  • Measure and monitor 
  • How to Reduce Cost

Right size of the instance

Right-Sizing: Amazon EC2 instances

Right-sizing is the process of matching the instances types and size of your demand and workloads and making sure you have the correct performance and capacity requirements at the lowest cost.
AWS says Right-sizing is the most effective way to control the cloud cost.
Select the cheapest instance available. Ensure you meet performance requirements, Select the right instance family (virtualization type, network, platform) Newer instance family generation typically have better price/performance profiles if your application supports it-beware of dependencies, legacy applications.

    Analyze using CloudWatch metrics 

    • Monitor CPU, RAM, Storage, and network utilization 
    • Identify potential instances that can be downsized 
    • Setup custom metrics (i.e., RAM) where needed 
    Right-Sizing Amazon-S3 

    Amazon S3 storage offers a range of storage classes: 

    • Standard 
    • Standard-Infrequent Access 
    • One Zone – Infrequent Access 
    • Glacier 
    • Glacier Deep Archive 
    • Reduced redundancy (which is no longer recommended) 

    Key points: 

    • Standard, Standard-IA and One Zone-IA are the real-time storage tiers 
    • Glacier and Glacier Deep Archive are the near-line storage tiers. They are rarely accessed storage. 
    • If you are retrieving an object once per month or more, Standard is a more cost-effective storage class than Infrequent Access. 
    • Storage classes can be set on not just per bucket but set as a per-object basis.  
    • To managing, storage classes on a per-object basis can be complex and time-consuming 
    • Consider using S3 Intelligent-Tiering if you have unknown or changing access patterns 
    • Automatically moves your data between Standard and Infrequent-Access tiers  
    Right-Sizing AWS Lambda function  

    Do performance testing of your Lambda function. Lambda allocates the CPU power in proportion to the amount of memory configured. Memory usage is determined per-invoke can be viewed in AWS CloudWatch logs.
    Identify if your function is CPU-bound or Memory-bound. Various benchmarking tools available on GitHub to identify it.
    Minimize your deployment package size and complexity. 

    • Minimizing size and complexity will reduce the time that it takes for your deployment package to be downloaded 
    • Prefer simpler a framework that can load quickly 

    Right-Sizing Tips: 

    • Schedule monthly right-sizing analysis reports. 
    • Use the Cost Optimization monitor to visualize reports. 
    • Enforce tagging so that their attribute can identify instances. 
    • Right sizing your databases by adjusting memory and compute power up and down as performance and capacity requirement change.  

    Tools for Right sizing and finding unused resources 

    • AWS CloudWatch 
    • AWS Trusted Advisor 
    Amazon DynamoDB pricing 

    Reserved capacity pricing:
    Reserved capacity pricing is a great way to reduce your AWS DynamoDB costs by reserving your read and write capacity units in advance, and you can obtain significant cost saving compared to the On-Demand pricing model

    Amazon DynamoDB pricing: 

    • Pricing for On-Demand capacity mode
      $1.25 (1 WCU = 1*1 KB write) per 1m Write Capacity Unit (WCU) per hour
      $0.25 (1 RCU = 1*1 KB read) per 1m Read Capacity Unit (RCU) per hour  
      • Pricing for provisioned capacity mode
        $0.00065 (1 WCU = 1*1 KB write per second) per Write Capacity Unit (WCU) per hour $0.00013 (1 RCU = 1*1 KB read per second) per Read Capacity Unit (RCU) per hour  

      Dynamo DB free tier:
      DynamoDB feature included in the AWS free tier calculated each month on pre-region, per-payer account basis: 

      • 25 WCUs and 25 RCUs of provisioned capacity 
      • 25GB of data storage 
      • 25 WCUs for global tables deployed in 2 AWS region 
      • 2.5 million streams read request from DynamoDB stream 

      Increase Elasticity

      Increase Elasticity to Always Meet Business Demands. In AWS Cloud, you can optimize costs to meet your needs and turn off the resources when they are not in use. You can turn off non-production instances for 70% or more of any given week. Also, you can scale the service for short term usage and helping you to reduce your billing. Since spot instances are available on discount as compared to on-demand pricing, you can reduce 90% of the operating cost of your running application compared to on-demand instances.
      Tools for Increase Elasticity 

      • Autoscaling 
      • AWS Instance schedule 
      • Instance purchasing option 

      Choose the Right Pricing Model

      AWS provides the following purchasing option to optimize your cost based on your actual needs: 

      • On-demand Instances 
      • Reserved Instances 
      • Standard RI  
      • Convertible RI 
      • Schedule Instance 
      • Spot Instances 
      • Dedicated Instance 
      • Dedicated Host 
      • Capacity
      • Reservation 

          Purchasing Options

          On-demand pay per unit only for what you used
          Example: Amazon S3, Amazon EC2
          Provisioned as pay per unit of capacity
          Examples: Amazon Kinesis Data Streams, Amazon DynamoDB
          Reserved discounted pricing in return for a fixed-time commitment.
          Examples: AmazonEC2, Amazon Elasticsearch Service
          Many services support multiple billing options, auto-scaling.


          Reserved Instance Pricing

          In AWS, each instance’s type offers different compute, memory, and storage capabilities. Each instance type is grouped into instance families based on the capabilities. When we right-size our Instances, select on Instance type based on requirements, and the demand of the application that you need to run AWS ensures that each instance type provided will have a consistent amount of CPU capacity. 

          Dedicated Host can help you address compliance requirements and reduce costs by allowing you to use your existing server-bound software license.
          Purchasing and Billing:  When you use a dedicated Host to your account, On-Demand billing is automatically activated. You must have a dedicated host in your account purchasing reservation. 

                Dedicated Instance

                In comparison to On-demand Instance, pay for dedicated Instances save 70% by purchasing Reserve Instance or save 90% by purchasing Spot Instances.  

                Spot Instances 

                Spot Instances are the EC2 capacity that can save up to 90% of On-Demand prices that AWS can interrupt with in a notification. It uses the same underlying EC2 instances as On-Demand and Reserved Instances and is best suited for the flexible workload. 

                Capacity Reservation purchasing option 

                If you require a capacity reservation, it is AWS best practice to purchase Reserved Instances or capacity reservation for a specific Availability zone or purchase schedule Instances. Spot Instances are a cost-effective choice, but you must be flexible with them since your application can be interrupted, including when your application run. Dedicated Host and dedicated Instances can help you reduce your costs because they allow you to use your existing already paid server-bound software license. 

                Dedicated Host

                Dedicated Host an help you address compliance requirements and reduce costs by allowing you to use your existing server-bound software license.
                Purchasing and Billing:  When you use a dedicated Host to your account, On-Demand billing is automatically activated. You must have a dedicated host in your account purchasing reservation. 

                      Dedicated Instance

                      In comparison to On-demand Instance, pay for dedicated Instances save 70% by purchasing Reserve Instance or save 90% by purchasing Spot Instances.  

                      Spot Instances 

                      Spot Instances are the EC2 capacity that can save up to 90% of On-Demand prices that AWS can interrupt with in a notification. It uses the same underlying EC2 instances as On-Demand and Reserved Instances and is best suited for the flexible workload. 

                      Capacity Reservation purchasing option 

                      If you require a capacity reservation, it is AWS best practice to purchase Reserved Instances or capacity reservation for a specific Availability zone or purchase schedule Instances. Spot Instances are a cost-effective choice, but you must be flexible with them since your application can be interrupted, including when your application run. Dedicated Host and dedicated Instances can help you reduce your costs because they allow you to use your existing already paid server-bound software license. 

                      Measure and Monitor

                      • AWS TCO 
                      • Monthly AWS Invoice 
                      • AWS Billing console  
                      • AWS Cost Explorer 
                      • AWS Budget 
                      • 3rd party tool AWS Billing file analysis, DIY dashboard 

                        AWS TCO 

                        AWS helps you reduce a comparative Total Cost of Ownership (TCO). TCO is the comparative total cost analysis, the operating cost, running in your infrastructure on the AWS environment, and running your infrastructure in the traditional on-premises environment.
                        AWS helps you to reduce TCO by reducing the need to invest in large capital expenditure and providing a pay-as-you-go model that empowers users to invest in the capital you need. TCO provides a detailed set of reports. TCO is a projection of the direct and indirect cost of purchasing a product or services
                         TCO calculators allow users to estimate the total cost savings when using AWS and provide a detailed set of reports. The TCO calculator also gives you the option to modify assumptions that best meet your business needs.
                        The TCO Calculator guides on possible realized savings when using AWS. This tool is built on a calculation model that generates a fair cost of value that a customer may achieve. This tool is for approximation purposes only.
                        Lowering TCO 

                        • Reduce over-provisioning – Pay only for what you use 
                        • Choose the right pricing model 

                        The TCO Calculator automates the task of selecting the right AWS instance type based on the information you provide. You can describe your physical or virtual infrastructure in detail, and the calculator will provide the equivalent AWS instance types that meet requirements.
                        You can download a cost breakdown report or store in Amazon S3 and share it with others. It provides a comprehensive & detailed cost breakdown. 

                        Monthly AWS Invoice 

                        Viewing your monthly charges:
                        When you select a one-time fee to incur AWS charges the credit card and issues your invoice as pdf, you can also download the pdf from Billing and Management console using the below measure.
                        Note: 
                        IAM users need explicit permission to see some of the pages in the Billing and Cost Management console.
                        To view your monthly charges: 

                        • In AWS Management Console, open the Billing and Cost Management console at the below link https://console.aws.amazon.com/billing/home#/
                        • choose Bills In the navigation pane,  
                        • For the Date, select a month. 
                        • The Summary section shows a summary and detailed information of the charges for that month.  
                        • If you are using the consolidated billing feature in AWS Organizations, In the Bills page, it shows the lists for totals for all accounts on the Consolidated Bill Details Tab. Select the Bill Details by Account tab to watch the activity for each user account in the organization. 

                        To view your Billing charges for a different month: 

                        • Go on the Bills page, select the month you want from the Date list. 

                        To download a copy of your billing charges as a PDF document: 

                        • From the Bills page, select a month from the Date list for which all activity is closed. 
                        • From the Summary tab, select Usage Charges and Recurring Fees. 
                        • Choose the Invoice <invoiceID> link. 

                        To download a monthly Billing report: 

                        • Choose the Download CSV button and then select the appropriate option. 

                        Getting an Invoice Mailed to you
                        Follow the below steps to have a PDF copy of your monthly invoice sent to the email address associated with your account.
                        To edit your email recipients, see Editing Contact Information.

                        • Log in to the AWS Management Console and open the Billing and Cost Management console at https://console.aws.amazon.com/billing/home#/
                        • Choose Billing preferences on the navigation pane. 
                        • Select the Receive PDF Invoice by an Email checkbox. 
                        • Choose Save preferences

                        Note: The PDF invoice is difficult from a tax invoice. The Tax invoice is specially for your tax charges, whereas PDF invoices show your monthly AWS charges.
                        When AWS bill will be ready, and an invoice/bill will be charged?
                        Your AWS bill/Invoice for the previous month is finalized and ready at the beginning of each month. Shortly after, the bill/Invoice is charged to your default payment method, usually between the 3rd and the 5th day of the month.
                        If you are using the Cost and Usage Report, check the bill/InvoiceID column there. The bill/InvoiceID column is blank until your bill is finalized. 

                        AWS Billing Console 

                        AWS Billing and Management console provide features and a dashboard to view and manage your costs and usage.
                        The following are the steps to discuss the most common tasks that you’re likely to perform when using the Billing and Cost Management console. 

                        AWS Cost Explorer 

                        AWS Billing and management includes Cost Explorer for a deeper dive into AWS Cost and usage. AWS Cost Explorer lets you visualize, understand, and manage your AWS costs and usage over time. If you have multiple AWS accounts within an AWS organization, cost will be consolidated in the master account. 

                        • Use Forecasting to get an idea of future costs. 
                        • Choose if you want to verify your data at a monthly or daily level of granularity. It helps you to visualize your day to day costs in AWS. 
                        • Use filter and grouping functionalities to dig even deeper into your data. 

                        Enable Cost Explorer
                        You can enable Cost Explorer for your AWS account using the Billing and Cost Management console, but you can’t enable Cost Explorer using the API. When you allow Cost Explorer, AWS prepares the report about your costs for the current month and the last three months and calculates the forecast for the next three months. The current month’s data is available for you for viewing in about 24 hours. The rest of the data will take a few days longer. Cost Explorer updates your cost data report at least once every 24 hours.
                        By default, you can launch Cost Explorer if your account is a member account in an organization. The Master or root account can, however, block your access.
                        Note:
                        An account’s status within an organization affects what cost and usage data is visible: 

                        • When a single account joins an organization, the account no longer has access to price and data usage from the time range, when the user account is standalone. 
                        • If a member account left the organization A to join organization B, the member account will no longer has access permission to cost and usage data from the time range when the user account was a member of organization A. The member has access only to the data that is generated as a member account of organization B. 
                        • If an account rejoins an AWS organization that it previously belonged to, the account regains access to its previous(historical) cost and data usage. 

                        Signing up to Cost Explorer and receive the AWS Cost and Usage data Report or the Detailed Billing Data doesn’t automatically enable Cost Explorer. You must enable it by performing the below procedure.

                        To Enable Cost Explorer 

                        • In AWS Management Console, sign in and open the Billing and Cost Management console at https://console.aws.amazon.com/billing/home#/
                        • On the navigation pane, select Cost Explorer
                        • On the Cost Explorer page, choose Enable Cost Explorer

                        AWS Budget 

                        AWS Budget creates a custom cost usage budget to track AWS cost, and usage. Use of AWS Budget to set the custom budget that alerts you when your costs or usage exceed (or a forecasted to exceed) your budgeted amount. Also, use the AWS budget to set reservation utilization or coverage target and receive alerts when your utilization drops below the defined threshold. AWS Budgets give you the ability to set up the alert if you exceed or are approaching your defined budget. 

                        • Create Cost, Usage or Reservation Budget, can be tracked at the monthly quarterly or yearly levels, with customizable start and end dates 
                        • Alerts support EC2, RDS, Redshift, and Elastic Cache reservations. 
                        • Budget based on fixed cost or plan your upfront based on your chosen level 

                        Feature: You can create up to 5 alerts each budget and each alert can notify up to 10 email recipients as well as publish to SNS.
                        AWS Budgets information is updated three times a day. Budgets track your unblended costs, refunds, subscriptions, and RIs. You can create the types of budgets given below: 

                        • Cost budgets –Plan how much you want to spend on a service. 
                        • Usage budgets–Plan how much service you want to use one or more. 
                        • RI utilization budgets –Define a utilization threshold and receive budget alerts when your RI usage drop below that threshold. This will show if your RIs are under-utilized or unused. 
                        • RI coverage budgets –Define a coverage threshold and receive budget alerts when the number of your instance hours that are covered by RIs drop below that threshold. This shows how much of your instance usage is covered by a reservation. 
                        • Savings Plans utilization budgets –Define a utilization threshold and receive budget alerts when the usage of your Savings Plans drops below that threshold. This shows if your Savings Plans are under-utilized or unused. 

                        For per master account you can create up to 20,000 budgets or AWS Organizations standalone account. The first 62 days of budget are free of charge for each month. For regular accounts, each additional budget day costs $0.02.

                        AWS Trusted Advisor

                        AWS says that AWS Trusted Advisor is an online tool that provides you real-time guidance to help you to provide your resources AWS best practices and can reduce Instance performance Improve security.
                        Five categories and seven cores Trusted Advisor checks
                        Optimized your AWS infrastructure with the five types of trusted advisor 

                        • Cost Optimization 
                        • Performance 
                        • Fault Tolerance 
                        • Security 
                        • Service limit 

                        All AWS customer gets the access of seven cores Trusted Advisor checks:
                        Business support and Enterprise support customers get full access to the complete set of Trusted Advisor checks and recommendations. These checks help optimize your entire AWS infrastructure, security, and performance, reduce overall cost and monitor service limits.

                        AWS Tagging

                        AWS-generated tags are tagging that AWS defines, and used to supported AWS resources for cost allocation purposes. To use an AWS generated tags, a master account owner must activate it in cost and management console for cost allocation tagging.

                        User-Defined tags are the tags that you define, create, and apply to resources. After you have created and applied it, you can activate them in the Billing and cost management console for cost allocation tagging.
                        AWS makes it easy to modify the tags to meet your changing business requirements. Remember to include future-plans for billing reports, automation, and access control in your tagging strategy
                        Tagging provides several benefits 

                        • Automation (Autoscaling, Scheduling) 
                        • Control and compliance (IAM policies) 
                        • Cost allocation (reporting and chargebacks)

                        How to Reduce Cost

                        • Ways to reduce cost:
                          • Shutdown unused AWS resources 
                          • Delete zombie resources 
                          • Instance scheduling 
                          • Reserved instances and spot instances 
                          • Move cold data to cheaper storage tiers 
                          • Autoscaling 
                          • Consolidated Billing 
                          • Select right Instance type and enable Tagging 
                          • Monitor, Track and analyze your service usage 
                          • Stop spending money on data center 
                          • Adopt a consumption model 
                          • Ask AWS Credits and save up to $100,000 
                          • Use Private IP instead of Elastic IP and stop paying inter AZ data transfer 
                          • Use new AWS saving plans
                        • Design Principles:
                          • Adopt a consumption model 
                          • Measure overall efficiency 
                          • Stop spending money on data center operations 
                          • Analyze and attribute expenditure 
                          • Use managed services 

                        Summary

                        Keep usage and expenditure awareness, Tag all the resources to enable the cost attribution. Ensure business owners have visibility into their workload cost. Always create reports, budgets and alerts. Optimize over time and give responsibility (and authority) for AWS cost optimization within your organization.