What is the best time to start optimizing your cloud spend?

Three months ago… but the second-best time is today. Quickly implementing Turbot’s prebuilt automated controls today will allow you to achieve far more savings in 2023 than a built-from-scratch or cross-team manual effort that won’t start realizing value until mid-year (or later). 

Your cloud costs are increasing daily, but controlling costs can be difficult and the time to execute on these initiatives competes with other priorities for the operations team.  This post describes Turbot automations that you can deploy immediately to reduce your cloud spend. The examples will focus on AWS, but the logic and tools apply to all major cloud providers.

Tagging strategies for charge-back and show-back

The adage — You can't control what you don't see — very much applies to cloud cost management. We have seen customers achieve massive cost savings of 30% or more by making sure each of their business units receive an accounting of their current spend each month. This will have even more impact if that business unit is charged for their usage, and it comes out of their budget. 

Tagging resources correctly is critical to segmenting your cloud spend by business unit. Turbot's automated tagging solutions allow you to retroactively tag resources with additional metadata (like cost center) and ensure all newly created resources are tagged correctly. Some example tagging strategies that can be automated using Turbot:

• Tag all resources with who created the resource and the resource created time.

• Propagate tagging metadata from AWS organizations to all resources in an account.

• Tag all resources with a cost center based on the account deployed to or the individual that created it.

• Propagate metadata from Service Now or custom databases to all resources.

• Auto remediate common misspellings and abbreviations in existing tag keys and values.

• Remediate competing tag strategies to one enterprise standard:

  • Env:Dev -> environment:development

  • env:Development -> environment:development

• Propagate tags from key resources to dependent resources:

  • Tag volumes attached to a compute instance with the instance’s tags. 

  • Tag snapshots created from a volume with the volume’s tags.

  • Tag all resources that are associated with a VPC with the VPC’s metadata.

• Remediate when new resources are created without correct tags:

  • Add missing tags to the resource.

  • Create alarms and alert on the resource tagging issue.

  • Terminate newly created resources without correct tags.

[Turbot On]

[Turbot On] Tagging with Context

Tagging is a crucial component for cloud operations, security and compliance. The most common tagging methodologies rely on owner-assigned resource tags to add external context to resources; however, additional deep context can be added to resources via automation…

Read more

5 years ago · 1 like · Bob Tordella

Stopping resources when not in use

One of the best things about cloud services is that you only need to pay for what you use. For workloads that don’t need to be running 24/7 (think development, testing, QA environments) scheduling resources to stop off hours can save over 70% of your cloud usage. Turbot has some great built-in controls to help you automate this across hundreds of cloud service accounts.

[Turbot On]

[Turbot On] Cost Savings

Ensure instances are stopped before they overrun your cloud budget. For workloads that don’t need to be running 24/7 (think development, testing, qa environments) scheduling resources to stop off hours can save over 70% of your cloud usage. This week we will look at some scheduling best practices and see how Turbot can help enforce them…

Read more

5 years ago · 2 likes · Bob Tordella

In Turbot, scheduling guardrails are readily available to control your cloud resource usage. You can choose to shutdown instances every night, every weekend, or on a custom schedule with just a few clicks. Turbot currently supports the following cloud services to be automatically scheduled for start/stop:

• AWS EC2 Instances

• AWS RDS Clusters/Instances

• AWS Redshift Clusters

• AWS WorkSpaces

• Azure Virtual Machines

• GCP Compute Instances

Finding and removing unused resources

Lack of visibility to cost factors and rapid development in the cloud can mean that teams lose track of infrastructure that is no longer needed over time. Turbot can help your operations team hunt down and destroy these unwanted pests.

• Delete unattached storage volumes

• Delete old snapshots

• Delete load balancers without targets

• Delete unattached elastic IP addresses

• Delete unattached NAT gateways

• Delete unused internet gateways

• Unused Secret Manger Secrets

[Turbot On]

[Turbot On] Unattached Storage Volume Cleanup

The ability to easily create, attach and unattach disk volumes is one of the key benefits of working with IaaS, but it can also become a source of unchecked cost if not watched closely. Even if an Amazon EBS Volume, Azure Compute Disk, or GCP Compute Engine Disk is unattached, you are still billed for their provisioned storage. Surprisingly this adds u…

Read more

5 years ago · 6 likes · Bob Tordella

[Turbot On]

[Turbot On] Cleanup Unwanted Internet Gateways

An Internet Gateway (IGW) attached to an Amazon VPC is a highly available network component that allows Internet connectivity from (or to) your VPC. However, for many Network topologies IGWs are either unnecessary or unwanted. For example, if you are routing all VPC traffic back to your on-premise network, having an IGW present may create risk of unappr…

Read more

5 years ago · 1 like · 2 comments · Bob Tordella

[Turbot On]

[Turbot On] Automated Snapshot Cleanup

One of the key benefits of working with IaaS services Amazon EBS, Amazon RDS, etc. is the ability to programmatically create backups and snapshots, but it can also become a source of unchecked cost if not watched closely. The AWS Backup service, first released in 2019, has the ability to automate backup scheduling and enforce retention policies, but man…

Read more

5 years ago · 2 likes · Bob Tordella

Delete default VPCs in unused regions

Did you know that AWS creates 17 VPCs in every new AWS account. Yes, a VPC is created automatically in every AWS region with multiple subnets and public accessibility via an internet gateway. Deleting these VPCs can create significant cost savings and cost avoidance.

• If you use AWS Config or similar products that catalog cloud resources, removing these VPCs can reduce your usage of those services by lowering the number of resources you are tracking.

• These VPCs are often mistakenly used to deploy resources to when using the AWS console, generating waste until discovered. Removing them early eliminates the technical debt and cost of cleaning them up later due to overlapping IP ranges.

• These VPCs are also frequently used by malicious insiders to hide crypto mining or other personal projects, and we have seen these VPCs used by external bad actors when an account is compromised. With Turbot it is easy to delete them automatically:

[Turbot On]

[Turbot On] Automatic deletion of default VPCs

For new AWS accounts, default VPCs with subnets in every AWS region are created automatically. AWS does this to make it easy for new users to get started quickly; however it’s a nuisance to manage for your next account and every one after that. Most enterprise customers have dozens, if not hundreds of accounts, and default VPCs add complexity as they a…

Read more

5 years ago · 2 likes · Bob Tordella

Remove multiple enabled CloudTrails per account

Did you know that your first AWS CloudTrail is free in each account, but you pay if you create multiple trails? Turbot can ensure that you have one global CloudTrail configured in each account and automatically disable additional trails. This setting will continuously monitor this and remove trails created by third party tools or misunderstanding developers.

Route53 TTL

If you configure a higher TTL for your DNS records, intermediate resolvers cache the records for longer time. As a result, there are fewer queries received by the name servers. Turbot can help you identify records with low TTL values, we suggest increasing the value to one day (86,400s) for production environments with static DNS, one hour (3,600s) for environments with more frequent changes.

Instance and volume type savings

There are two key strategies at play when considering automating optimization of instance and volume sizes. 

1.     Identifying existing non-optimized instance types

2.     Preventing future use of non-optimized instance types.

Turbot can help with both; our automation can immediately stop (or terminate) usage of instance/volume types that are not approved for use in your cloud and reporting against our CMDB can help you identify where existing infrastructure could be optimized.

Key strategies here that Turbot can implement:

1.     Enforce use of gp3 instead of gp2/io1 volume types. AWS's newer gp3 volumes are lower cost and higher performance for every price point against the older gp2 volume type, but the AWS console is still defaulting to gp2 instead. Use Turbot to ensure your developers are not leaving money on the table here and creating more technical debt. See: https://turbot.com/blog/2020/12/aws-ebs-cost-savings/

2.     If your workloads support running on ARM processors, you can save up to 40% by switching instance types from power hungry legacy x86 workloads. Turbot can help you identify candidates for switching and automatically stop (or terminate) new instances created using older instance types.

3.     Has your organization purchased reserved instances of a certain class? Use Turbot to ensure dev teams are using the correct class and size of instance to maximize usage of your reserved instances.

4.     Prevent usage of non-standard high-cost instance types by mistake. All three cloud providers have introduced a bevy of massive scale instance types for high performance computing needs (at eye-watering price points). Make sure your dev teams don’t accidentally spin up that $100/hr instance using Turbot's guardrails to limit provisioning to only approved instance types. This prevents accidental use, and it is super simple to create an exception when a team has budget and need for the extra horse power.

Account-based budgeting

Turbot can monitor the month-to-date spend for AWS accounts and take actions when accounts are trending to overspend. This can take the form of creating an over-budget alert, preventing new resources from being created or even deleting existing resources from sandbox or playground accounts once a threshold has been exceeded.

Why real-time cost remediation?

No matter your organizations maturity level with cloud, reducing costs is something that will always be a priority. Using automation to enforce cost controls allows your operations team to focus on the next opportunity while your current automations continuously save you money. When cost savings initiatives are manual, you only get a one-time improvement and the amount of savings will always be limited by the number of people you can have executing those processes.

If you need any assistance getting this configured in your environment, please reach out to Turbot Support!

It’s been a busy 2022 at Turbot. We launched Turbot Quick Actions, support for new compliance frameworks, added support for new cloud services, and hundreds of new policies and controls. We ended the year with over 250 updates to Turbot Mods, and over 35 releases of Turbot Enterprise (TE stack). In addition to all the great work we accomplished on Turbot in 2022, we also had a lot to celebrate with our open source project Steampipe.

We’re constantly iterating on Turbot based on your voice of customer feedback and while we’re excited to bring you even more innovation in 2023, it’s worth reflecting on all of the new capabilities and use cases that have been unlocked over the last 12 months. 

Here are some of the highlights over the year.

Quick Actions

Our customers love using Turbot automation to find and fix problems in real-time across hundreds of cloud service accounts. However, there are many situations where the cloud team wants to quickly take a specific one-time action on a resource while remaining in the context of their multi-cloud compliance dashboard.

Quick Actions enable DevOps engineers to instantly remediate cloud configuration issues (e.g. enable encryption on a resource), snooze compliance alarms, or take operational actions (e.g. tag a resource, start/stop an instance) from the Turbot Compliance Dashboard.

Cloud Compliance Frameworks

Customers often map Turbot controls to their own Governance Risk Compliance (GRC) tools, and use the resulting evidence to prove continuous adherence to internal controls or external standards. Throughout the year we started to port over control frameworks from Steampipe into Turbot.  Along with AWS, Azure & GCP CIS benchmarks already supported by Turbot, this year we added support Payment Card Industry Data Security Standard (PCI DSS), Health Insurance Portability and Accountability (HIPAA), and NIST 800-53 controls.

Custom Approved Controls

The Approved guardrail checks the status of the defined Approved sub-policies for the resource. If the resource is not approved according to any of these policies, an alarm is raised and takes the defined enforcement action (e.g. stops the resource, deletes the resource, etc).

Custom checks were added to be part of the Approved control evaluation, and allow for custom messages to be added which are then displayed in the control details table.

This provides a middle ground to customers who need more flexibility to define their own control logic with custom messaging without requiring a Custom Mod or a Turbot mod to extend the feature.

New & Improved Turbot

• Performance & scale:

  • Made significant performance improvements on the database infrastructure through use of Graviton instance types and GP3 storage.

  • Optimized indexes for increased UI performance.

  • Targeted performance improvements for resource activity views, mod uploads, user grants deletion and the permissions detail views.

  • Added an events priority queue for actions initiated via the UI, that allows these actions to take precedence over the cloud event backlog.

  • Added log warnings when a query is using fuzzy matching, to alert users to poor performing queries.

• New improvements:

  • Improved the state reasons and details in control messages.

  • Added v5 support for IAM user mode.

  • Added v5 support for LDAP integrations.

• New capabilities - we are very proud to have the broadest coverage of any CSPM tool on the market.

  • Turbot now supports 734 resource types, 3,988 control types and 9,313 policy types (22% increase over 2021).

  • 79 updated and 10 new mods in 2022.

  • New compliance mods included: Azure CIS v1.2, AWS HIPAA, AWS PCI v3.2.1, AWS NIST 800-53.

Our customers continue to do amazing things

The thing that got our team most excited in 2022 was seeing our customers do super cool stuff with Turbot. Here are a few interesting automation use cases we saw in 2022:

• Using Turbot to discover and remove 600,000 unneeded EBS snapshots.

• Using a custom Service Now workflow to feed project cost center information into Turbot that was used to tag cloud resources.

• Using AWS > IAM > Role > Approved custom calculated policy to allow use of GitHub Actions only from specific GitHub repos owned by the organization.

• Pulling account tagging metadata from AWS organizations to tag all resources in an account with the same cost center metadata.

• Restricting cross-account bucket access to specific OU’s in an organization by using Turbot to automatically set bucket policy restrictions using `aws:PrincipalOrgPaths` condition keys.

• Using Turbot Firehose to get a real-time streaming view of multi-cloud resource change over time.

• Using Turbot’s RBAC to manage time-based access to raw data at the AWS level for 600+ researchers.

Off to the races in 2023

We have some exciting new features that are just around the corner and look forward to showing them off soon. Thanks to all who used and supported Turbot & Steampipe in 2022. Your engagement fuels our passion to keep innovating, and will continue to inspire us in 2023!

Turbot includes thousands of prescriptive cloud controls to ensure cloud environments are secure and cost-optimized. These controls quickly detect issues as they occur and instantly correct misconfigurations. Customers often map Turbot controls to their own Governance Risk Compliance (GRC) tools, and use the resulting evidence to prove continuous adherence to internal controls or external standards such as Center for Internet Security (CIS), HIPAA, PCI, SOC2, etc. 

Over the last year Turbot has open-sourced Steampipe.io, a tool that enables cloud engineers to easily query & report across their cloud, code, logs, and more, using the standard language of data: SQL. Steampipe includes thousands of ready-to-use controls and dashboards that deliver insights into your cloud data. These controls and dashboards leverage a suite of plugins that translate cloud APIs into Postgres tables. One of those plugins, by the way, translates the Turbot v5 API into SQL-queryable tables!

Cloud teams have asked Turbot to also include industry standards within the Turbot platform. We started by open-sourcing thousands of compliance controls, and enlisting the Steampipe community to provide feedback and contributions. Based on community feedback, we are now starting to port these control frameworks into Turbot.  Our first addition in Turbot was the Payment Card Industry Data Security Standard (PCI DSS) v3.2.1 standard for AWS, then the Health Insurance Portability and Accountability (HIPAA) standard for AWS. Note: When mapping to external standards, Turbot leverages either the published industry standard mapping or the mapping provided by each cloud provider. We are excited to add the NIST 800-53 controls to the platform to complement our existing HIPAA, PCI & CIS Controls, along with AWS, Azure & GCP CIS benchmarks.

This week’s [Turbot On] will look at how to enable NIST 800-53 controls across all your AWS accounts in Turbot.

Traditional Workflow

There are various cloud-native and 3rd-party tools to evaluate cloud infrastructure NIST 800-53 compliance. However cloud-native tools generally work with only one cloud provider, and only do periodic scans.  3rd-party tools may support multiple cloud providers, but fall short on benchmark coverage and, again, scan and report only periodically, missing real-time changes in your environment.  These tools often work on a per-account basis, without delivering resource-level granularity. And they are limited in their ability to manage the time-based exceptions that enable you to handle the nuances in your organization.

Get it done with Turbot

In Turbot, NIST 800-53 guardrails are readily available to control your cloud resource configurations.  These guardrails work similar to others, continuously evaluating adherence as changes to your cloud resources occur.  First, make sure you have the `@turbot/aws-nist-800-53` mod installed and any dependent mods installed in your workspace.  Then you can enable NIST 800-53 through the following Turbot policy in just a few clicks: `AWS > NIST 800-53`:

Setting the configuration via the Turbot Terraform Provider is just as simple:

After enabling this policy, Turbot will immediately evaluate all applicable resources compliance with NIST 800-53.  You can view your controls across AWS services:

Drill further into subsections:

Analyze which specific resources are impacted per control:

Make it happen

See for yourself on how easy it is to view your NIST 800-53 requirements across your cloud resources.  If you need any assistance please reach out to Turbot Support, and keep an eye on your inbox for another [Turbot On] post in the near future!

Cheers,

Bob

For the last eight years, Turbot has focused on remediation of operational and compliance issues at enterprise scale. Our customers love using Turbot automation to find and fix problems in real-time across hundreds of cloud service accounts. However, there are many situations where cloud professionals want to quickly take a specific one-time action on a resource while remaining in the context of their multi-cloud compliance dashboard.

Introducing Quick Actions

Now generally available in Turbot v5.39.0, Quick Actions enable DevOps engineers to instantly remediate cloud configuration issues (e.g. enable encryption on a resource), snooze compliance alarms, or take operational actions (e.g. tag a resource, start/stop an instance) from the Turbot Compliance Dashboard.

Action types are specific to the service and the resource, meaning that S3 Buckets support different actions than EC2 instances. After enabling Quick Actions in your workspace (see below), you can browse a list of available actions for a given resource by clicking on the orange “Actions” button (located in the top right of each resource detail page):

Traditional Workflow

The Turbot console is a powerful tool for exploring cloud resources across large multi-account environments. In the past, when cloud teams discovered an issue within their environment using Turbot alarms, they would have to make a choice between leaving the compliance dashboard and addressing the issue in another tool, or configuring Turbot policies to remediate the problem via automation.

The choice to fix the issue manually forces the DevOps engineer to switch context from the CMDB and dashboard into other tools to apply the change. By removing that cognitive load, we make that same engineer more productive and effective.

Make it green™ with Turbot

Now with Quick Actions, the most common functions required by cloud operations teams can be executed immediately, without context switching. The result is increased operator productivity leading to a clean (all green) compliance dashboard with full audit trail. The list of available quick actions is growing weekly, and will continue to do so based on customer feedback., Here is a short list of the currently supported resource types across AWS, GCP & Azure:

Amazon Web Services

• EC2 Instances

• EC2 Volumes & Snapshots

• Auto-Scaling & Target Groups

• Load Balancers (All types)

• Load Balancer Listeners

• Key Pairs

• Launch Configurations

• Launch Template & Versions

• IAM Users, Roles, Groups

• Access Keys

• IAM Policies, Inline Policies

• Server Certificates

• KMS Keys

• Lambda Functions

• Lambda Alias & Versions

• RDS DB Clusters & Instances

• DB Snapshots

• DB Parameter Groups

• DB Option & Subnet Groups

• S3 Buckets

• SNS Topics & Subscriptions

• SQS Queues

Google Cloud Platform

• Projects

• Compute Instances

• Compute Image

• Instance Templates

• Node Groups & Templates

• Compute Health Check

• HTTPS Health Check

• Region Health Check

• Compute Disk

• Regional Disks

• Compute Snapshots

Azure Cloud

• Virtual Machines

• Images

• Snapshots

• Disk Encryption Sets

• Compute Availability Sets

• Compute Disks

• Virtual Networks

• Application Security Groups

• Network Security Groups

• Network Interface

• Public IP Addresses

• Route Tables

• Subnets

Typical use cases

1. Start/Stop Instances and Databases – Forgot to turn off that m5.16xlarge you were testing on last week? Oops! Shut it down right now.

2. Delete Resources – Found 90TB of three year old EC2 snapshots? Clean ‘em up as you identify them.

3. Snooze Alarms – Give that critical app team 90 days of runway to clean up their environment by snoozing their alarms.

4. Tag resources – Instantly apply your custom tagging template to the untagged resource you just found.

5. Enable Encryption – Found a rogue bucket without default encryption? Turn it back on without breaking a sweat.

Get started with Quick Actions

The Quick Action feature is available to all Turbot SaaS and Enterprise customers on version 5.39.0 or higher. To enable quick actions in your Turbot workspace, simply set the policy: Turbot > Quick Actions > Enabled == "Enabled" for a single account or the entire environment.

For details on how to limit access to quick actions and create custom permission sets please see this guide in the Turbot docs. If you need any assistance please reach out to Turbot Support, and keep an eye on your inbox for another [Turbot On] post in the near future!

Turbot includes thousands of prescriptive cloud controls to ensure cloud environments are secure and cost-optimized. These controls quickly detect issues as they occur and instantly correct misconfigurations. Customers often map Turbot controls to their own Governance Risk Compliance (GRC) tools, and use the resulting evidence to prove continuous adherence to internal controls or external standards such as Center for Internet Security (CIS), NIST 800-53, PCI, GDPR, SOC2, etc. 

Over the last year Turbot has open-sourced Steampipe.io, a tool that enables cloud engineers to easily query & report across their cloud, code, logs, and more, using the standard language of data: SQL. Steampipe includes thousands of ready-to-use controls and dashboards that deliver insights into your cloud data. These controls and dashboards leverage a suite of plugins that translate cloud APIs into Postgres tables. One of those plugins, by the way, translates the Turbot v5 API into SQL-queryable tables!

Cloud teams have asked Turbot to also include industry standards within the Turbot platform. We started by open-sourcing thousands of compliance controls, and enlisting the Steampipe community to provide feedback and contributions. Based on community feedback, we are now starting to port these control frameworks into Turbot.  Our first addition in Turbot was the Payment Card Industry Data Security Standard (PCI DSS) v3.2.1 standard for AWS, followed by the Health Insurance Portability and Accountability (HIPAA) standard for AWS. Note: When mapping to external standards, Turbot leverages either the published industry standard mapping or the mapping provided by each cloud provider. We are excited to add the HIPAA controls to the platform to complement our existing PCI & CIS Controls, along with AWS, Azure & GCP CIS benchmarks.

This week’s [Turbot On] will look at how to enable HIPAA controls across all your AWS accounts in Turbot.

Traditional Workflow

There are various cloud-native and 3rd-party tools to evaluate cloud infrastructure HIPAA compliance. However cloud-native tools generally work with only one cloud provider, and only do periodic scans.  3rd-party tools may support multiple cloud providers, but fall short on benchmark coverage and, again, scan and report only periodically, missing real-time changes in your environment.  These tools often work on a per-account basis, without delivering resource-level granularity. And they are limited in their ability to manage the time-based exceptions that enable you to handle the nuances in your organization.

Get it done with Turbot

In Turbot, HIPAA guardrails are readily available to control your cloud resource configurations.  These guardrails work similar to others, continuously evaluating adherence as changes to your cloud resources occur.  First, make sure you have the `@turbot/aws-hipaa` mod installed and any dependent mods installed in your workspace.  Then you can enable HIPAA through the following Turbot policy in just a few clicks: `AWS > HIPAA`:

Setting the configuration via the Turbot Terraform Provider is just as simple:

After enabling this policy, Turbot will immediately evaluate all applicable resources compliance with HIPAA.  You can view your controls across major sections of the HIPAA standard:

Drill further into subsections:

Drill further to expand the subsections to view particular per cloud service controls:

Analyze which specific resources are impacted per control:

Inverse the view to visualize all primary controls on a particular resource:

Make it happen

See for yourself on how easy it is to view your HIPAA requirements across your cloud resources.  If you need any assistance please reach out to Turbot Support, and keep an eye on your inbox for another [Turbot On] post in the near future!

Cheers,

Bob

Customers value Turbot to provide cloud governance solutions to increase their security and compliance posture across their cloud environments. Customers use Turbot to enable policy based automation for AWS, Azure and GCP to discover & correct misconfigurations instantly. Turbot ensures environments are secure & cost optimized at all times elevating the cloud team to do more with less manual effort. To support Turbot's automation, the product features an event driven, real-time CMDB to capture audit trail & asset inventory, 9500+ point-and-click ready policies to set with inheritance & time-based exceptions, multi-cloud timed-based RBAC management, and managed IaC stack deployments.

Turbot is a powerful governance tool in our customer's cloud environments, earning and maintaining our customers' trust is of the utmost importance to us at Turbot. Our operations, product and engineering teams, are dedicated to ensuring that our products are designed, architected and developed with both the security of our products, and of our customers' data in mind. Because of this we have invested heavily in security, and we are excited to share that Turbot has achieved our annual SOC 2 Type II compliance for Turbot Cloud (SaaS).

With the adoption of industry best practices for controls and processes throughout our environments and software development lifecycle, we strive for best-in-class security. This includes security awareness training for all employees, achieving our CIS Benchmark Certification, undergoing multiple Well-Architected audits as part of our AWS Advanced Tier Security & Cloud Management status, support for our privacy policy for GDPR, CCPA, and Privacy Shield, enterprise supplier audits, and continuous penetration testing.

Ensuring we meet the data security, privacy and compliance needs of our customers is core to our business. This achievement validates our commitment as we strive to earn and maintain our customers' trust, and, as we progress in our compliance journey to help us mature our security posture.

Our SOC audits are conducted annually each April. The auditors prepare their audit report which is then released each May. Our complete SOC 2 Type II audit report is available to customers and prospects under NDA upon request. Visit our Turbot Security page for more information about our security practices.

If you have any questions please reach out to Turbot Support, and keep an eye on your inbox for another Turbot post in the near future!

Turbot includes thousands of prescriptive cloud controls to ensure cloud environments are secure and cost-optimized. These controls quickly detect issues as they occur and instantly correct misconfigurations. Customers often map Turbot controls to their own Governance Risk Compliance (GRC) tools, and use the resulting evidence to prove continuous adherence to internal controls or external standards such as Center for Internet Security (CIS), NIST 800-53, HIPAA, GDPR, SOC2, etc. 

Over the last year Turbot has open-sourced Steampipe.io, a tool that enables cloud engineers to easily query & report across their cloud, code, logs, and more, using the standard language of data: SQL. Steampipe includes thousands of ready-to-use controls and dashboards that deliver insights into your cloud data. These controls and dashboards leverage a suite of plugins that translate cloud APIs into Postgres tables. One of those plugins, by the way, translates the Turbot v5 API into SQL-queryable tables!

Cloud teams have asked Turbot to also include industry standards within the Turbot platform. We started by open-sourcing thousands of compliance controls, and enlisting the Steampipe community to provide feedback and contributions. Based on community feedback, we are now starting to port these control frameworks into Turbot.  Our first addition is the Payment Card Industry Data Security Standard (PCI DSS) v3.2.1 standard for AWS. Note: When mapping to external standards, Turbot leverages either the published industry standard mapping or the mapping provided by each cloud provider. We are excited to add the PCI controls to the platform to complement our existing CIS Controls and AWS, Azure & GCP CIS benchmarks.

This week’s [Turbot On] will look at how to enable PCI controls across all your AWS accounts in Turbot.

Traditional Workflow

There are various cloud-native and 3rd-party tools to evaluate cloud infrastructure PCI compliance. However cloud-native tools generally work with only one cloud provider, and only do periodic scans.  3rd-party tools may support multiple cloud providers, but fall short on benchmark coverage and, again, scan and report only periodically, missing real-time changes in your environment.  These tools often work on a per-account basis, without delivering resource-level granularity. And they are limited in their ability to manage the time-based exceptions that enable you to handle the nuances in your organization.

Get it done with Turbot

In Turbot, PCI guardrails are readily available to control your cloud resource configurations.  These guardrails work similar to others, continuously evaluating adherence as changes to your cloud resources occur.  First, make sure you have the `@turbot/aws-pciv3-2-1` mod installed and any dependent mods installed in your workspace.  Then you can enable PCI through the following Turbot policy in just a few clicks: `AWS > PCI v3.2.1`:

Setting the configuration via our Terraform Provider is just as simple:

After enabling this policy, Turbot will immediately evaluate all applicable resources compliance with PCI.  You can view your controls across services:

Drill further into sub controls of a specific service:

Analyze which specific resources are impacted:

Visualize all the resources primary controls including PCI:

Make it happen

See for yourself on how easy it is to view your PCI requirements across your cloud resources.  If you need any assistance please reach out to Turbot Support, and keep an eye on your inbox for another Turbot post in the near future!

Cheers,

Bob

A common headache for cloud operations is dealing with infrastructure that wasn’t optimally deployed from the start. Application teams may not be experts in all areas of infrastructure as code deployment and as such may assume that default settings for their AWS infrastructure implement best practices (like encryption) without the need for them to take an active role in defining all aspects of their infrastructure design.

It is easy for an inexperienced developer or data scientist to use the AWS console to deploy new instances and add storage to them, however the default settings do not enforce many best practices, including encryption at rest for EBS volumes and snapshots.

The good news is that AWS is listening and continually rolling out new options to help address these issues. One such feature is the ability to specify default encryption for EBS resources in each region. This means that any new EBS volumes created in that region will automatically get encryption without any action needed by the developer.

This week’s [Turbot On] will look at how to automate EBS default encryption across all your accounts and regions. 

Traditional Model

Enabling the default encryption setting must be done for every individual region and account, using this setting in the AWS console:

Performing this action across a couple regions or accounts is trivial, but ensuring that the setting is consistently applied across 20+ regions and hundreds of accounts is 100% a job for automation!

Get it Done with Turbot

Turbot’s automation can ensure that the correct KMS key is setup for EBS default encryption, in every region and across hundreds of accounts simultaneously. Every time a new account is on-boarded, Turbot will identify that the setting is not enabled and set it appropriately without need for any manual intervention.

One policy setting is all it takes:

Setting the configuration via our Terraform Provider is just as simple:

Once this policy is set, Turbot will automatically discover all regions not configured to use default EBS volume encryption and enforce the policy setting. If you are not ready to enforce the change, you can still audit all accounts/regions that do not have the setting enabled, simply change the Turbot policy setting from “Enforce: …” to “Check: …” and Turbot will create alarms for any regions without the correct configuration.

It is important to note that the default setting is just that. It doesn’t apply to already provisioned instances and it can be changed/ignored/bypassed by administrators. If EBS default encryption does not meet your organizations use case, or you want to take a more in-depth preventative approach, you can use Turbot’s EC2 Encryption at Rest controls such as:

• ‘AWS > EC2 > Instance > Approved > Root Volume Encryption at Rest’

• ‘AWS > EC2 > Volume > Approved > Encryption at Rest’

• ‘AWS > EC2 > Snapshot > Approved > Encryption at Rest’

To implement more stringent resource level policy and exceptions around encryption.

Make it happen

See for yourself on how easy it is to manage your encryption at rest requirements across your cloud resources. A ready-to-run Terraform template is available to enable this configuration from the Turbot Development Kit (TDK). If you need any assistance please reach out to Turbot Support, and keep an eye on your inbox for another Turbot tip next week!

Cheers,

Bob

The AWS Well-Architected Framework was initially developed as a whitepaper in 2012 to help architects moving applications to, or building natively on, AWS. The framework asks introspective questions that lead you to best practices and ensure that application teams are getting the most out of cloud native architectures.

In 2018, AWS released the first version of the Well-Architected Tool. This service from AWS helps walk application teams through assessment of their workloads against well-architected principals. The tool systematizes the way assessments are performed, and allows the application team building on AWS to gain insight from an expert system. The use of the tool is free; teams just define their workload and answer a set of questions regarding operational excellence, security, reliability, performance efficiency, and cost optimization.

This week’s Turbot On looks at how to leverage the AWS Well-Architected tool for large organizations with hundreds (or thousands) of AWS accounts.

Turbot has long promoted the benefits of multi-account isolation when it comes to enterprise use of AWS. In this model each application workload is deployed into separate AWS accounts with some common centrally managed infrastructure. This allows the application team to focus on building their application, and the centralized cloud operations and security teams to standardize critical security, compliance and data protection setup that should be consistent across the organization.

However, this poses two challenges for organizations looking to leverage the AWS Well-Architected Tool. First, the application teams have less knowledge of some key infrastructure for their accounts; if the cloud operations team is deploying their VPC configuration or their CloudTrail logging is centralized, it can make it difficult for the application team to answer many of the questions in the Well-Architected review. Secondly, very large enterprises have a scale issue. Namely, how do you conduct thorough reviews across hundreds of AWS accounts and ensure that the application teams are asking and answering the right questions.

Get it Done with Turbot

Turbot’s automation can ensure that Well-Architected Framework questions are automatically answered, covering aspects of the application and infrastructure architecture that are centrally managed (e.g. Networking, Identity, etc). Automatically answering common questions helps reduce the amount of work required for a Well-Architected review, which removes excuses and decreases the time & effort application teams spend on their assessments.

Questions can be answered statically and dynamically.

1. Static answers can be used when a centralized deployment, tool, process, etc. covers the intent of the question on behalf of the application team. They can also cover the case of ‘Not Applicable’ when questions are not relevant to the workload.

2. Dynamic answers are based on conditional logic. Using Turbot’s Calculated Policies, data from the Turbot CMDB can be used to evaluate the appropriate answer to specific review questions based on resource or Turbot configuration.

Turbot has two Mods that work together for automating the Well-Architected Tool. The ‘aws-wellarchitected` Mod automates configuration of the Well-Architected Tool and associated Workloads, and the ‘aws-wellarchitected-framework’ Mod which has 340+ policies that allow your team to automate answers to the default ‘AWS Well-Architected Tool Framework Lens’. 

Static Example:

In this example, our network is centrally managed, so the cloud operations team will configure Turbot to automatically answer questions related to the organization’s network configuration. In the UI we can browse the framework questions and see potential answers:

To automate the answer to this question across all 300 of our AWS accounts, we only need to set two policies.  First, let’s choose the ‘Enforce non-overlapping private IP address ranges…’ answer:

Now that the appropriate answer is chosen for our workloads, we tell Turbot to apply the answer via the parent policy:

These same controls can also be configured via Turbot’s Terraform provider:

Dynamic example 1 - using CMDB resource metadata:

In this example, our KMS keys are classified with specific tags which identify they are centrally managed. If the keys are present in the account, you can use Turbot’s Calculated Policies to lookup information from the CMDB to decide on the answer to specific questions. As an example in “SEC 08. How do you protect your data at rest?” One of the available answers is “Implement secure key management”.

Using a calculated policy Turbot checks to see if a specifically tagged KMS key is present in the current account. If Turbot finds the key it sets the answer to the Framework question to be ‘True’ if not found, the value is set to false.

Dynamic example 2 - using control states in Turbot:

In this example, our organization has encryption at rest requirements for many AWS services. Instead of enumerating checks on all the required services, we will use the Encryption at Rest control category to evaluate if encryption at rest controls are applied effectively. Control categories aggregate related policies across services and across cloud platforms. You can use Turbot’s Calculated Policies to look for evidence that Encryption at rest controls are in use to answer the question “SEC 08. How do you protect your data at rest?”.

This calculated policy will count the controls in ‘ok’ and ‘alarm’ state for a given scope. If Turbot finds evidence that 1 or more controls are in place it sets the answer to the question to ‘True’ and if it finds no active controls it will set the value to ‘False’.

Once any static and or calculated policies are set, Turbot will automatically discover all workloads configured in the Well-Architected Tool and apply the defined answers to the specified questions. These policies can be futher grouped into Turbot Smart Folders to apply to specific classes of AWS accounts (e.g. ‘Production’, ‘Development’, etc.).

Make it happen

See for yourself how Turbot can help you manage Workload Assessments at scale. As you are working towards setting these policies, an initial baseline to get started is available from the Turbot Development Kit (TDK). Additional calculated policy examples for dynamic answers (resource data and control states) are also available in the TDK in the calculated policies section. If you need any assistance please reach out to Turbot Support, and keep an eye on your inbox for another Turbot tip next week!

Cheers,

Bob

Amazon Elastic Container Registry (Amazon ECR) can be used to host private container image repositories in AWS. This allows teams great flexibility in choice of centralizing or federating repositories for your application teams.

One of the key benefits of moving to a container architecture for your application is to microsegment application functionality with immutable containers for greater security. To maintain that secure configuration, the software libraries used in each container should be scanned for vulnerabilities.

The open source Clair project maintains a list of known vulnerabilities and tools for static analysis of your application containers. Amazon ECR uses the Common Vulnerabilities and Exposures (CVEs) database from the Clair project and provides a list of scan findings. You can review the scan findings for information about the security of the container images that are being deployed.

You can manually scan container images stored in Amazon ECR. Or, alternatively, you can configure your repositories to scan images when you push them to a repository. 

In today’s Turbot On we look at how you can easily enable Turbot to enforce automated scanning of every container when they are pushed to your repositories.

Traditional Workflow

The popularity of ECR with application development teams means that you may have dozens (or hundreds) of ECR Repositories across your environment and new ones will be created at any time. Relying on application teams to manually enable scan on push configuration in their development environments can lead to last minute discovery of issues as those teams release software. Automation is needed to support discovery of existing and future ECR repositories, and automated configuration of the scan on push setting.

Get it done with Turbot

In Turbot, Amazon ECR Repository guardrails are readily available to control your cloud resource configurations. We can set the Turbot automation `AWS > ECR > Repository > Scan on Push` policy in just a few clicks:

Setting the configuration via our Terraform Provider is just as simple:

After setting these policies, Turbot will identify all ECR repositories without Scan on Push enabled and then handle remediation (i.e. enable Scan on Push). 

If you are not yet ready to enforce remediation, you can still assess your environment’s ECR Repository Scan on Push compliance by setting the value of the policy to ‘Check: Enabled’ at the Turbot level. In ‘Check’ mode Turbot will alarm on Amazon ECR Repositories that do not have Scan on Push in place. After review of the alarms, selectively apply the enforcement settings or create exceptions as desired. 

Given that Scan on Push may not be applicable for all repositories, make use of Turbot’s policy exceptions and time-based expiration settings features to mark exceptions to the rule or automatically reset a configuration when the exception expires.

Make it happen

See for yourself how easy it is to manage your image scanning requirements across your cloud resources. A ready-to-run Terraform template is available to enable this configuration from the Turbot Development Kit (TDK). If you need any assistance please reach out to Turbot Support, and keep an eye on your inbox for another Turbot tip next week!

Cheers,

Bob

Turbot Mod Changes

• AWS SageMaker controls for Model, Training Job, Endpoint, Service

• Improvements on AWS Event Rule sorting in CMDB data to remain consistent

• EC2 Instance CMDB supporting EC2:BidEvictedEvent for spot instances

• EC2 Snapshot discovery and CMDB handling improvements

• Turbot Firehose updated templates in support of StreamAlert integration

Additional updates can be found in the full Release Notes.

Turbot UI Changes

• AWS External ID while in protected mode now accepts a custom suffix input copied into the UI.

• Aging calculations in all reports now correctly calculate the duration.

Additional updates can be found in the full TE Release Notes.

Turbot Enterprise Changes

The current recommended deployment versions for Turbot Enterprise are updated here: https://turbot.com/v5/docs/releases

Apollo becoming new default UI

In the upcoming v5.37.0 release, the default UI for all Turbot users will become the Turbot Console Apollo UI. For users already using the Apollo UI, no change will occur, and for users who still prefer the original UI, you can switch back with a link in the header of the console. For Turbot Cloud (SaaS) customers this change will occur automatically.  For Turbot Enterprise customers this change will occur when you upgrade to the v5.37.0 release or higher.

The existing (non-Apollo) console will be considered deprecated in the v5.37.0 release, and in a few months, the v5.40.0 release will fully remove the non-Apollo UI. This will not impact APIs, but will impact saved URLs pointing to specific screens in the old UI.

Since its release in Nov 2020, the Apollo UI is the preferred UI among Turbot users. You can learn more about Apollo in our highlights video.

Key Performance Improvements:

• Moving resources to new locations in the hierarchy is more responsive in the UI.

• Cleanup of unused tables (action_history) and unused indexes (controls_history, resources_history, and policy_values_history) to reduce DB disk space.

• Critical database indexes are now re-created weekly to improve performance.

• Improved handling for long running control to avoid infinite execution.

• DB parameter group support for 11.10, 11.11, 12.6 and 13.2.

• Postgres version 13.2 is now the default selection; Turbot continues to support Postgres 11 and 12.  New installations will default to Postgres 13.x.  When appropriate, we will recommend an update path from 11.x & 12.x to 13.x

  Text within this block will maintain its original spacing when published

Full Release Notes:

• Turbot Enterprise

• Turbot Enterprise Foundation (TEF)

• Turbot Enterprise Database (TED)

Turbot Developer Tools

Terraform - https://turbot.com/v5/docs/releases/terraform

• FAQ Guide - How do I create a resource with multiple AKAs?

Turbot CLI - https://turbot.com/v5/docs/releases/cli

• FAQ guide - Can I use cli/graphql command to get the list of current users and their associated grants?

Turbot Plugin for Steampipe: https://hub.steampipe.io/plugins/turbot/turbot

• Initial release of the plugin provides an open source CLI to instantly query Turbot’s API using SQL!  The initial tables supported are controls, policies, resources, smart folders and tags.

You can contribute to the project and keep us posted on any feedback and suggestions.

Turbot On Posts

• [Turbot On] Managing default routes ‘0.0.0.0/0’ - how to manage security group rules for public facing accounts. 

• [Turbot On] Custom CMDB Data - how to import your org’s custom reference data into the Turbot CMDB.

• [Turbot On] S3 Bucket Logging - how to automate server access logging on your S3 buckets.

• [Turbot On] Encryption at Rest for SNS - how to enforce encryption at rest to all your Amazon SNS topics.

• [Turbot On] Turbot Product Updates - April highlights of Turbot product changes.

The default route ‘0.0.0.0/0’ is treated as evil by many organizations; most security scanning tools will flag any security group containing a default route as insecure. The default route is often improperly used when troubleshooting connectivity issues and frequently used by lazy developers as a shortcut. However, there are many legitimate uses of the default route.

Given that the deployment of public facing websites, web apps and open APIs are increasing in number (even for large enterprises), it makes sense to find more nuanced ways of providing governance for these accounts and the security groups within them.

This week [Turbot On] focuses on how to write governance controls for public facing security groups.  We will use AWS as an example, but the same principals and process will work for Azure and GCP as well.

Traditional Workflow

Many large organizations have historically outsourced public facing websites and web applications to 3rd parties to reduce the liability of having public endpoints into their data centers.  In the world of cloud-based systems, this makes little sense, but the phobia of the default route continues to persist. We see organizations struggle with how to manage exceptions for these types of applications. Groups that leave the false positive alarms in place suffer from alert fatigue and teams that turn them off expose themselves to real vulnerabilities.

Get it done with Turbot

Turbot’s Object Constraint Language (OCL) gives you great flexibility and control over security group rules.  OCL works using simple REJECT and APPROVE filters. It executes from top to bottom and the first matching filter wins. Cleverly constructing the order of the filter statements allows for a wide variety of policy implementations.  We can implement these rules individually for ingress and egress rules using policies like AWS > VPC > Security Group > Ingress Rules > Approved > Rules:

In the simple rule above, we reject any ingress rule that uses the IPV4 default route (0.0.0.0/0) or the IPV6 default route (::/0).  For our public facing web application let’s create a rule that allows http/https traffic (ports 80 and 443) with a default route source, but denies other usage of it:

In this example the first check looks for a security group rule matching ports 80 or 443 with ingress from the default route. If it matches, the APPROVE rule is applied and processing stops on that rule (each rule in a security group is evaluated independently). If the first rule does not match, then the next check would reject any other port with default route ingress. 

Like all policies, these can also be set via Turbot’s Terraform provider.  Here is an example of creating a Smart Folder named “public_web_sg_check” with the same rules. Using a smart folder allows you to attach the same policies to multiple accounts.

After setting this policy, Turbot will identify all security group rules that match REJECT statements as alarms, allowing you to request remediation of the improper security groups from the app team.

If you are ready to enforce automated remediation, you can change the policy setting from ‘Check: Approved’ to ‘Enforce: Delete unapproved’ and Turbot’s automation will delete all security group rules that match a REJECT statement.

Make it happen

See for yourself how easy it is to manage your network ingress and egress rules across all your cloud services. A ready-to-run Terraform template is available to enable this configuration from the Turbot Development Kit (TDK). If you need any assistance, please reach out to Turbot Support, and keep an eye on your inbox for another Turbot tip next week!

Cheers,

Bob

Having correct metadata for a resource is crucial for automated resolution of operations, security and compliance incidents. Teams often rely on owner-assigned resource tags to add external context to resources; however, additional deep-context can be added to resources via the Turbot CMDB.

This week, [Turbot On] will look at how to import and use custom CMDB data using the Turbot Files feature.

If you’re intrigued the idea of building custom automated governance controls, please consider registering for our talk at AWS Fest 2021 on June 22nd.

Our CTO will be discussing the pros and cons of industry benchmarks vs. custom controls, and doing a live coding demo to show how easy it is to get started.

Traditional Workflow

Databases outside of the CMDB often contain reference data (and master data) that is important to cloud operations and security: ‘cost centers’, ‘approved project IDs’, ‘distribution lists’ and ‘data classification’ just to name a few. This type of data changes over time but its correctness can be very important when automating governance controls.

Requiring application teams to create and update tagging metadata is notoriously difficult to enforce. Even when the teams do maintain data, simple data entry mistakes, alternate spellings and capitalization mismatches are very common errors.

Turbot Files

A Turbot ‘File’ is a text-based data object that typically contains a JSON formatted string. Once the object is imported into the Turbot CMDB, the data in it can then be referenced at runtime in Calculated Policies and Stacks.

• A File resource can contain any arbitrary data. Customers will often utilize a JSON schema to make it easier to reference data inside each file.

• A File resource can be a child of the root Turbot resource or a Turbot Folder.

• The name (internally called `aka`) of the File resource is user-definable.

Get it done with Turbot

Turbot Files can be managed using standard GraphQL API or the Turbot Terraform Provider. Management can be automated in a variety of ways depending on your organization's requirements. Some examples:

• Use a trigger to update the Turbot File whenever asset data in an inventory management tool changes.

• Write a shell script to pull data from a third party API and update the File with the Turbot CLI.

• Manually update the File via Terraform.

• Write a Lambda to update the File via the GraphQL API.

In this example we will use a Terraform template to create a Turbot File in order to add application metadata to our CMDB. Once the data is imported we will use it to enhance our tagging controls.

This Terraform template contains a File resource (note the name of the resource is “dmiapps”). Applying the template adds the JSON object specified in `content` to our Turbot workspace.

Now that we have our Turbot File created we can use the metadata inside of it as a data-source for our calculated policies. For example, we can use Turbot’s tagging controls to tag resources using metadata from the “dmiapps” file. To demonstrate the approach we will use a calculated policy to tag our S3 Buckets with the correct values based on the enclosing AWS Account.

The GraphQL query (see “Step 1” below) retrieves both the bucket resource’s metadata object, and the data from the “dmiapps” file.

As seen in Step 2, above, we can then extract the AWS account ID from the resource and use it to lookup keyed values stored in the “dmiapps” file.

The provided example can easily be adjusted for any resource that can be tagged (across all supported cloud services). The best part is that whenever the Turbot File is updated (e.g. a new app is added, change in App owner, etc.), any affected tags will automatically be updated as well.

Now that our tagging template above is set via a calculated policy, we can begin enforcing the tagging control by setting the Tags policy value to `Enforce: Set tags`:

After setting this policy, Turbot will identify all resources that do not have the tags applied correctly, and then handle their remediation (i.e. set the tags).

If you are not yet ready to enforce remediation, you can still assess (and get alerts for) what resources do not have matching tags by changing the policy setting from `Enforce: Set tags` to `Check: Tags are correct`.

Make it happen

See for yourself how easy it is to manage your custom CMDB metadata configurations across all your cloud resources. A ready-to-run Terraform template is available to enable this configuration from the Turbot Development Kit (TDK). If you need any assistance please reach out to Turbot Support, and keep an eye on your inbox for another Turbot tip next week!

Cheers,

Bob

One aspect of cloud compliance I often see overlooked is audit readiness. If you work in any regulated industry or host data for your customers, you will eventually have your cloud environments audited.

When addressing an audit finding it is critical that you can answer the key questions of who, what, and when data was accessed or changed. Fail to answer these questions and your processes will be considered “out of control”. Availability of logs to answer these questions can be the key difference between a minor audit finding (e.g. improved training) and having to revamp your whole change control process.

This week [Turbot On] will look at how to automate server access logging for Amazon S3 buckets.

Server access logging provides detailed records for requests that are made to an Amazon S3 bucket. With logging enabled, the access log record contains the request type, the resources that are specified in the request, and the time and date that the request was processed. Access logging allows security teams and auditors to understand how authenticated users are interacting with your S3 buckets. 

Traditional Workflow

The AWS console or APIs can be used to enable and disable access logging. When you enable this feature, Amazon periodically collects access log records, consolidates the records in log files, and then uploads log files to a target bucket as S3 objects. 

Configuration considerations can increase complexity:

• The target bucket for logging must be in the same region as the source bucket.

• The target bucket must be owned by the same account as the source bucket.

• Multiple source buckets logging into the same target bucket may need different prefixes.

• Setting appropriate permissions on the logging bucket for separation of duties.

• Setting lifecycle policies for the logs based on the organization’s approved log retention period.

This means that we need to create a logging destination bucket in each region we operate in, setup lifecycle policies on it, and configure it so the application team can read, but not delete from the bucket. One it is created we need to update every bucket in the account to point to that location with a designated prefix, and make sure they stay configured that way over time.

While it is possible to configure this manually for a few buckets across a couple regions, automation will be needed to support larger environments.

Get it done with Turbot

In Turbot, access logging guardrails control your cloud resource configurations. The key policy setting for access logging can be found directly under the resource in the policy hierarchy: AWS > S3 > Bucket > Access Logging. Policy sub-settings determine the target bucket and how to prefix the logs inside the bucket:

• AWS > S3 > Bucket > Access Logging > Bucket

• AWS > S3 > Bucket > Access Logging > Key Prefix

Using calculated policies, you can implement dynamic naming conventions to handle unique buckets per account, region and service as applicable. By using our prebuilt default logging settings for the destination and prefix, we can enable Turbot’s automation with just a few clicks:

After setting these policies, Turbot will identify all S3 buckets that do not have access logging enabled, and remediate them (i.e. enable the logging configuration).

If you are not yet ready to enforce remediation, you can still assess what buckets don’t have access logging enabled by setting the policy value to ‘Check: Enabled’. In ‘Check’ mode Turbot will alarm on buckets which do not have access logging in place. It’s also possible to check that a specific logging configuration is in place by using the alternate setting: ‘Check: Enabled to Access Logging > Bucket’. After review of the alarms, you can selectively apply the enforcement settings or create exceptions as desired. 

Given that access logging may not be appropriate for all buckets (e.g. development), make use of Turbot’s policy exceptions as necessary to achieve your desired compliance outcome across all environments.

Make it happen

See for yourself how easy it is to manage your access logging configurations across your cloud resources. A ready-to-run Terraform template is available to enable this configuration from the Turbot Development Kit (TDK). If you need any assistance please reach out to Turbot Support, and keep an eye on your inbox for another Turbot tip next week!

Cheers,

Bob

Amazon Simple Notification Service (Amazon SNS) is a messaging Platform as a Service (PaaS) that is frequently used as part of cloud-native, loosely-coupled application architectures (e.g. microservices, distributed systems, etc.). Turbot itself uses SNS in conjunction with CloudWatch events and SQS as a highly reliable and subscribable message queue. If your SNS topics are processing sensitive data, it makes sense to encrypt them in transit and at rest.

This week we will look at how Turbot can automate checking and enforcement of encryption at rest for your Amazon SNS topics.

Traditional Workflow

In a cloud native architecture, SNS topics are often created programmatically (i.e. a software factory that creates new lambda functions, might also create a new SNS topic to enable pub/sub messaging for the function. This means that your large scale applications may have hundreds or thousands of topics under management.

Furthermore, limitations in the past with cross service KMS key access prevented some applications from using encryption with SNS. If your applications were built a few years ago, those applications may not have updated their architecture and could still be generating new topics without encryption. Manually looking for them and remediating the issues is a tedious and time consuming task.

Get it done with Turbot

In Turbot, Amazon SNS Topics guardrails are readily available to control your cloud resource configurations. We can set the Turbot automation `AWS > SNS > Topic> Encryption at Rest` policy in just a few clicks:

Setting the configuration via Turbot’s Terraform provider is just as easy:

After setting these policies, Turbot will identify all SNS Topics without a configured AWS KMS managed key, and then handle remediation (i.e. set the correct encryption configuration).

If you are not yet ready to enforce remediation, you can still assess your environment’s SNS encryption compliance by setting the value of the policy to ‘Check: AWS managed key or higher’ at the Turbot level. In ‘Check’ mode Turbot will alarm on Amazon SNS Topics that do not have encryption at rest in place. After review of the alarms, selectively apply the enforcement settings or create exceptions as desired. 

Given that encryption may not be applicable for all topics, make use of Turbot’s policy exceptions and time-based expiration settings features to mark exceptions to the rule or automatically reset a configuration when the exception expires.

Make it happen

See for yourself how easy it is to manage your encryption configurations across your cloud resources. A ready-to-run Terraform template is available to enable this configuration from the Turbot Development Kit (TDK). If you need any assistance please reach out to Turbot Support, and keep an eye on your inbox for another Turbot tip next week!

Cheers,

Bob

Turbot Mod Changes

• AWS EC2 Mod improvements when tagging events occurred

• AWS ECR Image controls added

• AWS SageMaker controls for Code Repository, Endpoint Configuration, Lifecycle Configuration

• AWS Well-Architected Tool Tagging control

• Azure Network Security Group rules have an added condition for service tags approved

• GCP Firebase controls for Android App, Firebase Project, Web App, and iOS App

• New services and resources added for Turbot AWS Permissions; Connect, Cloud Directory, DataSync, MWAA, Cloud Map, Direct Connect, Translate, Rekognition, Cognito, AWS Tagging, Chatbot, Device Farm, Polly, Macie2, IAM Access Analyzer, AppFlow, Billing

• Turbot Event Handler custom rules -- custom options to reduce unused high volume AWS EC2 and AWS VPC events

• Additional updates can be found in the full Release Notes.

Turbot UI Changes

Turbot Best Practice Reports

Turbot’s best practice reports combine key controls for given resources into a single easy to read report. The image below shows a combined report for S3 buckets pulling to together results for nine separate controls into a single line item for each bucket. These reports are based on your policies settings and can be exported to CSV.

15 New Turbot Reports: 

• Turbot Best Practice - AWS S3 Buckets (See above)

• Oldest Azure Compute Disks

• Well-Architected Tool Workloads

• Azure Compute Disks Resource Details

• Unencrypted AWS CloudWatch Log Groups

• AWS EC2 Instance AMI usage

• AWS Default VPC

• AWS EC2 AMIs

• AWS Public Route 53 Hosted Zones

• Recent User Login

• Detached GCP Compute Engine Disks

• Unencrypted AWS CloudTrail Trails

• Aging AWS Access Keys

• Aging Turbot Access Keys

• Mods Admin List shows more information on the latest available version and last updated

Additional updates can be found in the full TE Release Notes.

Turbot Enterprise Changes

The current recommended deployment versions for Turbot Enterprise are updated here: https://turbot.com/v5/docs/releases

External ID Best Practices

The Turbot UI now auto-generates complex random external IDs to adhere to best practices and organizations can enforce use of unique external IDs using the `AWS > Account > Turbot IAM Role > External ID > Protection` policy. See the v5 FAQs for more info.

Apollo becoming new default UI

In the upcoming v5.37.0 release, the default UI for all Turbot users will become the Turbot Console Apollo UI. For users already using the Apollo UI, no change will occur, and for users who still prefer the original UI, you can switch back with a link in the header of the console. For Turbot Cloud (SaaS) customers this change will occur automatically.  For Turbot Enterprise customers this change will occur when you upgrade to the v5.37.0 release or higher.

The existing (non-Apollo) console will be considered deprecated in the v5.37.0 release, and in a few months, the v5.40.0 release will fully remove the non-Apollo UI. This will not impact APIs, but will impact saved URLs pointing to specific screens in the old UI.

Since its release in Nov 2020, the Apollo UI is the preferred UI among Turbot users. You can learn more about Apollo in our highlights video.

Postgres 13 support

Starting with TED v1.20.1 new installations will default to using Postgres 13. Existing Postgres 11 & 12 installs will not be impacted and no action needs to be taken now. When appropriate, we will recommend an update path.

Key Performance Improvements: 

• Moving resources to new locations in the hierarchy is more responsive in the UI.

• Process logs are saved to S3 as a single operation, reducing request costs.

• Cleanup of unused tables (action_history) and unused indexes (controls_history, resources_history, and policy_values_history) to reduce DB disk space.

• Critical database indexes are now re-created weekly to improve performance.

• Workspace will now pause on processing events during a TE upgrade.

  Text within this block will maintain its original spacing when published

Full Release Notes:

• Turbot Enterprise

• Turbot Enterprise Foundation (TEF)

• Turbot Enterprise Database (TED)

Turbot Developer Tools:

Terraform - https://turbot.com/v5/docs/releases/terraform

• Turbot’s Terraform Provider v1.8.2 has been tested compatible with Terraform version 14 and 15.

Turbot CLI - https://turbot.com/v5/docs/releases/cli

• FAQ guide - Can I generate AWS Access Keys programmatically?

  Text within this block will maintain its original spacing when published

Turbot On Posts:

• [Turbot On] Tagging with Context - how to automate the application of resource tags from CMDB metadata. 

• [Turbot On] S3 Public Access Blocks - how to automate AWS S3 account and bucket level public access blocks. 

• [Turbot On] GCP Firewall Rule Logging - how to automatically enable GCP Firewall Logging for one or more firewall rules.

• [Turbot On] Automated Snapshot Cleanup - how to save big by cleaning up older snapshots on a retention schedule.

With heightened emphasis on security and encryption of data in the cloud, an often overlooked aspect of data protection is backup and recovery of your organizations data. In the cloud, developers have programmatic access to delete resources and a simple slip of the CLI can sometimes lead to unrecoverable data loss. Ensuring that backups are created and available in the cloud is critical to being able to recover in these circumstances.

This week we will look at how Turbot can automate enabling continuous backups with Point-in-Time Recovery of your Amazon DynamoDB tables.

Traditional Workflow

When database service capabilities were managed by central teams, developers didn’t need to worry about backups. The owner of the ITSM service that managed their database ensured robust configuration and protection of enterprise data assets.  Cloud databases have similar capabilities, albeit with the condition that the development team must elect to enable and configure the backup services, a configuration step that can be forgotten, or in some cases enabled and then turned off at later points in time. Monitoring the current configuration of all your databases and ensuring that they meet the organization's data retention and backup requirements should be an automated governance control for all cloud databases.

Get it done with Turbot

In Turbot, Amazon DynamoDB Table guardrails are readily available to control your cloud resource configurations.  We can set the Turbot automation `AWS > DynamoDB > Table > Point-in-Time Recovery` policy in just a few clicks:

Setting the configuration via Turbot’s Terraform provider is just as easy:

Terraform template to set the AWS >DynamoDB > Table > Point-in-Time Recovery policy in Turbot.

After setting these policies, Turbot will identify all DynamoDB tables that are not enabled for point-in-time recovery, and then handle remediation (i.e. enable the configuration).

If you are not yet ready to enforce remediation, you can still assess the impact of this in your environment by setting the value to 'Check: Enabled` at the Turbot level.  In ‘Check’ mode Turbot will alarm on tables which do not have point-in-time recovery in place. After review of the alarms, selectively apply the enforcement settings or create exceptions as desired. 

Given that continuous backups may not be appropriate for all tables (e.g. development), make use of Turbot’s policy exceptions as necessary to achieve your desired compliance outcome across all environments.

Make it happen

See for yourself how easy it is to manage your backup configurations across your cloud resources. A ready-to-run Terraform template is available to enable this configuration from the Turbot Development Kit (TDK). If you need any assistance please reach out to Turbot Support, and keep an eye on your inbox for another Turbot tip next week!

Cheers,

Bob

Tagging is a crucial component for cloud operations, security and compliance. The most common tagging methodologies rely on owner-assigned resource tags to add external context to resources; however, additional deep context can be added to resources via automation.

This week we will look at how to “level up” your tagging game using automation and additional context from the Turbot CMDB

Traditional Workflow

Quick and transparent visibility to resource metadata can save precious minutes during an incident, but compliance from application teams to create and update tags is notoriously difficult to enforce. This leaves the cloud team in the unenviable position of nagging application teams to complete tagging of their resources.

For information that is dynamic, the problem is even more difficult. We don’t recommend trying to implement tagging standards for dynamic data unless you are using automation to implement it.

Get it done with Turbot

Turbot’s tagging controls are consistent across AWS, Azure and GCP resources. Furthermore, all resource metadata is stored in Turbot’s cloud scale CMDB and updated in real-time as configurations change.  Any detailed information in the CMDB can be leveraged for your resource tagging templates. For example, an AWS EC2 instance has over 100 fields that could be used in tag templates:

To demonstrate the approach we will use a Turbot calculated policy to tag our EC2 instances with their corresponding ImageId, Instance Type, SubnetId and VpcID:

First, set the calculated policy query:

{
  instance {
    ImageId
    InstanceType
    SubnetId
    VpcId
  }
}

Then, the calculated policy output template:

Image: "{{ $.instance.ImageId }}"
Type: "{{ $.instance.InstanceType }}"
Subnet: "{{ $.instance.SubnetId }}"
VPC: "{{ $.instance.VpcId }}"

Finally, set a standard policy to enforce the Tag control:

The example above can easily be adjusted for any resource that can be tagged:

• The tagging controls and templates are always found as a subset of the resource e.g.: `{Cloud Provider} > {Service} > {Resource} > Tags > Template`

• The naming and functions are consistent across all cloud providers & resources.

• In addition, whenever the underlying configuration changes, Turbot will update the tags with new correct values.

Setting the configuration via Turbot’s Terraform provider is just as easy:

After setting this policy, Turbot will identify all resources that do not have the tags applied correctly, and then handle their remediation (i.e. set the tags).

If you are not yet ready to enforce remediation, you can still assess (and get alerts for) what resources do not have matching tags by changing the policy setting from `Enforce: Tags are correct `to `Check: Tags are correct`.

Make it happen

See for yourself how easy it is to manage your tagging configurations across your cloud resources. A ready-to-run Terraform template is available to enable this configuration from the Turbot Development Kit (TDK). If you need any assistance please reach out to Turbot Support, and keep an eye on your inbox for another Turbot tip next week!

Cheers,

Bob

Newly created S3 buckets are secure and private by default, but AWS S3 provides features that allows administrators to share buckets with other authenticated and unauthenticated (public) entities.  This makes it possible to share a bucket between different applications running in separate AWS accounts, and to use the S3 service to host public information (e.g. a static website).

However, we often see developers relax permissions for a bucket or make bucket access public to try and resolve/troubleshoot a permissions issue. If not immediately remediated, this can lead to a data breach down the road. Given that these types of lapses occur all too frequently, AWS has implemented new features to give organizations more control over the ability to enable public access.

This week we will look at the two types of AWS S3 public access blocks and show you how to use Turbot to ensure they are automatically enabled across all of your accounts and buckets.

S3 Account Public Access Blocks

Account-level public access blocks have the ability to eliminate public access in both access control lists (ACLs) and in S3 bucket IAM resource policies.  You also have the option to block the creation of any new public access and/or to retroactively apply those settings to existing ACLs and bucket policies.  This work can be easily accomplished through the console UI for a single account, but retroactively enabling those settings across hundreds of accounts (and making sure they are not turned off) is a job for automation.

S3 Bucket Public Access Blocks

Bucket-level public access blocks apply protection in broad strokes, but won’t work in use cases where some bucket sharing is necessary.  For example, a customer might want to share a bucket between their data lake account and separate accounts that perform compute functions on the data.  In these circumstances you can use bucket level access blocks (instead of account level) to allow for some public or cross-account sharing on a bucket-by-bucket basis.

Enabling these public blocks across thousands of buckets, maintaining exception lists and ensuring that the configuration does not drift, is not something that can be done manually, it requires an automated continuous compliance solution.

Get it done with Turbot

In Turbot, AWS S3 public access block settings are readily available to control your cloud resource configurations.  Setting the correct Turbot policies can be accomplished with just a few clicks:

AWS S3 Account-Level Public Access Block:

AWS S3 Bucket Public Access Block:

Setting the configuration via Turbot’s Terraform provider is just as easy:

After setting these policies, Turbot will identify all AWS accounts and S3 buckets that are not enabled for public access block settings, and then handle remediation (i.e. set the configurations).

If you are not yet ready to enforce remediation, you can still assess the impact of this in your environment by setting the value to 'Check: Per `Public Access Block  > Settings` at the Turbot level.  In ‘Check’ mode Turbot will alarm on accounts and buckets that do not have public blocks in place. After review of the alarms, selectively apply the enforcement settings or create exceptions as desired. 

Given that denying public access may not be applicable for all accounts and buckets, make use of Turbot’s policy exceptions and time-based expiration settings features to mark exceptions to the rule or automatically reset a configuration when the exception expires.

Make it happen

See for yourself how easy it is to manage your public access block settings. A ready-to-run Terraform template is available to enable this configuration from the open source Turbot Development Kit (TDK). If you need any assistance getting with these policies please reach out to Turbot Support, and keep an eye on your inbox for another Turbot tip next week!

Cheers,

Bob

Firewall Rule Logging in Google Cloud (GCP) allows for audit, verification, and analysis of the configuration of firewall rules. With logging enabled it’s possible to determine if a firewall rule is functioning as intended, and how many connections are affected by any given rule.

Each log record record contains the source and destination IP addresses, the protocol and ports used, the date and time, and a reference to the firewall rule that applied to the traffic.  This information can assist in identifying potential operational and security risks in your environment.

This week we will look at how Turbot can automate the task of enabling GCP Firewall Rule logging on a single rule or across all rules in many GCP projects.

Traditional Workflow

The GCP console or APIs can be used to enable and disable firewall rule logging. When you enable this feature, the GCP Firewall service makes the logs available in Logs Explorer and in Firewall Insights. While it is simple to configure for a few firewall rules, ensuring that this logging is always enabled for all rules across dozens (or hundreds) of GCP projects would require development of automation scripts.

Get it done with Turbot

In Turbot, GCP Firewall Logging guardrails are readily available to control your cloud resource configurations.  We can enable this automation by setting the `GCP > Network > Firewall > Logging` policy with just a few clicks in the Turbot GUI:

Setting the configuration via Turbot’s Terraform provider is just as easy:

After setting this policy, Turbot will identify all firewall rules that are not enabled for firewall rule logging, and then handle their remediation (i.e. enable the logging configuration).

If you are not yet ready to enforce remediation, you can still assess what rules do not have logging enabled by setting the value to `Check: Enabled` at the Turbot level. 

Turning on firewall logging can generate a large number of logs which can increase GCP Stackdriver costs.  To prevent costs from running out of control, use Turbot’s time-based policy expiration feature to automatically reset the configuration after a given time period has elapsed, or when you no longer need logging enabled.

Make it happen

See for yourself how easy it is to manage your logging configuration across your GCP firewalls. A ready-to-run Terraform template is available to enable this configuration from the Turbot Development Kit (TDK). If you need any assistance please reach out to Turbot Support, and keep an eye on your inbox for another Turbot tip next week!

Cheers,

Bob