After I added the Google Cloud Storage backend to PrivateBin, I was
interested in its performance characteristics. To fake a real PrivateBin client, I needed to generate an encrypted message.
I searched for a client library and found PBinCLI. As it is written in Python,
I could write a simple load test in Locust. Nice and easy. How wrong was I.

create the test

Within a few minutes I had a test script to create, retrieve and delete a paste.
To validate the functional correctness, I ran it against https://privatebin.net:

$ locust -f locustfile.py \
   --users 1 \
   --spawn-rate 1 \
   --run-time 1m \
   --headless \
   --host https://privatebin.net

The following table shows measured response times:

NameAvg (ms)Min (ms)Max (ms)Median (ms)
create-paste237140884200
get-paste28273029
delete-paste27272927

Now, I had a working load test script and a baseline to compare the performance with.

baseline on Google Cloud Storage

After deploying the service to Google Cloud Run, I ran the single user test. And it was promising.

$ locust -f locustfile.py \
  --users 1 \
  --spawn-rate 1 \
  --run-time 1m \
  --headless \
  --host https://privatebin-deadbeef-ez.a.run.app

Sure, it is not lightning fast, but I did not expect that. The response times looked
acceptable to me. After all, privatebin is not used often nor heavily.

NameAvg (ms)Min (ms)Max (ms)Median (ms)
create-paste506410664500
get-paste394335514380
delete-paste587443974550

multi-user run on Google Cloud Storage

next I wanted to know the performance of PrivateBin with my Google Cloud Storage onder moderate load. So, I scaled the load test to 5 concurrent users.

$ locust -f locustfile.py \
  --users 5 \
  --spawn-rate 1 \
  --run-time 1m \
  --headless

The results were shocking!

NameAvg (ms)Min (ms)Max (ms)Median (ms)
create-paste4130662106664500
delete-paste344976862833800
get-paste290952055692400

How? Why? Was there a bottleneck at the storage level? I checked the logs and saw steady response times reported by Cloud Run:

POST 200 1.46 KB 142 ms python-requests/2.25.1  https://privatebin-37ckwey3cq-ez.a.run.app/
POST 200 1.25 KB 382 ms python-requests/2.25.1  https://privatebin-37ckwey3cq-ez.a.run.app/
GET  200 1.46 KB 348 ms python-requests/2.25.1  https://privatebin-37ckwey3cq-ez.a.run.app/?d7e4c494ce4f613f

It took me a while to discover that locust was trashing my little M1. It was running at 100% CPU without
even blowing a fan to create the encrypted messages! So I needed something more efficient.

k6 to the rescue!

When my brain thinks fast, it thinks golang. So I downloaded k6. The user scripts
are written in JavaScript, but the engine is pure go. Unfortunately, the interpreter
is custom built and has limited compatibility with nodejs and browser JavaScript engines.
This meant that I could not use any existing JavaScript libraries to create an encrypted message.

Fortunately with xk6 you can call a golang
function from your JavaScript code. This is what I needed! So I created the k6 privatebin extension and wrote
an equivalent load test script.

build a customized k6

To use this new extension, I build k6 locally with the following commands:

go install github.com/k6io/xk6/cmd/xk6@latest
xk6 build --with github.com/binxio/xk6-privatebin@v0.1.2

k6 baseline run on Google Cloud Storage

Now I was ready to run the baseline for the single user, using k6:

$ ./k6 --log-output stderr run -u 1 -i 100  test.js

The baseline result looked pretty much like the Locust run:

NameAvg (ms)Min (ms)Max (ms)Median (ms)
create-paste440396590429
get-paste355320407353
delete-paste382322599357

k6 multi-user run on Google Cloud Storage

What would happen, I i scaled up to 5 concurrent users?

$ ./k6 --log-output stderr run -u 5 -i 100  test.js

Woohoo! The response times stayed pretty flat.

NameAvg (ms)Min (ms)Max (ms)Median (ms)
create-paste5843502612555
get-paste484316808490
delete-paste460295843436

The chart below shows the sum of the medians for the single- and multi-user load test on Locust and k6:

It is clear that Locust was skewing the results way too much. With k6, I could even simulate 20 concurrent users, and still only use 25% of my M1.

NameAvg (ms)Min (ms)Max (ms)Median (ms)
create-paste7134141204671
get-paste562352894540
delete-paste515351818495

These are way more users than i expect on my privatebin installation and these response times are very acceptable for me. Mission accomplished!

conclusion

In the future, my goto tool for load and performance tests will be k6. It is a single binary which you can run anywhere. It is fast, and the golang extensions makes it easy to include any compute intensive tasks in a very user-friendly manner.

I have recently worked on a 100%-Terraform based project where I made extensive use of Workspaces and modules to easily manage the infrastructure for different environments on Google Cloud. This blog post explains the structure I have found to work best for the purpose.

What are Terraform workspaces?

Workspaces are separate instances of state data that can be used from the same working directory. You can use workspaces to manage multiple non-overlapping groups of resources with the same configuration.
To create and switch to a new workspace, after running terraform init, run:

terraform workspace create <name>

To switch to other workspaces, run instead:

terraform workspace select <name>

Use the selected workspace in the Terraform files

The selected workspace is made available in your .tf files via the terraform.workspace variable (it’s a string). I like to assign the value to a local variable called environment (since the name of the workspaces and that of the environments match).

locals {
    environment = terraform.workspace
}

Add the environment-based configuration to a new module

Now that you have a variable containing the environment (it’s just a string) that you are operating on, you can create a module containing the environment-based configuration. I created a vars module inside the modules directory of my repository, which contains at least the following files:

  • main.tf
    This file will never changes, as it’s only needed to aggregate the variables that will be exported.

    locals {
      environments = {
        "development" : local.development,
        "acceptance" : local.acceptance,
        "production" : local.production
      }
    }
  • outputs.tf
    This file too, will never change. Here I am defining the output of the vars module so that it can be used from anywhere else in the Terraform repository.
    The exported values are based on the selected workspace.

    output "env" {
    value = local.environments[var.environment]
    }
  • variables.tf
    This file defines the variables required to initialize the module. The outputs of the module are based on the selected workspace (environment), which it needs to be aware of.

    variable "environment" {
    description = "The environment which to fetch the configuration for."
    type = string
    }
  • development.tf & acceptance.tf & production.tf
    These files contain the actual values that differ by environment. For example, when setting up a GKE cluster, you might want to use cheap machines for your development node pool, and more performant ones in production. This can be done by defining a node_pool_machine_type value in each environment, like so:

    // in development.tf
    locals {
    development = {
        node_pool_machine_type = "n2-standard-2"
    }
    }
    // in acceptance.tf
    locals {
    acceptance = {
        node_pool_machine_type = "n2-standard-4"
    }
    }
    // in production.tf
    locals {
    production = {
        node_pool_machine_type = "n2-standard-8"
    }
    }

The vars module is now ready to be used from anywhere in the repository, for example in main.tf file. To access the configuration values, initialize the module like so:

#
# Fetch variables based on Environment
#
module "vars" {
    source      = "./modules/vars"
    environment = local.environment
}

The correct configuration will be returned based on the Terraform Workspace (
environment name) being passed to it, and values can be accessed via module.vars.env.<variable-name>. For example:

node_pools = [
    {
        ...
        machine_type = module.vars.env.node_pool_machine_type
        ...
    }
]

Summary

In this blog post I have shown you how you can use Terraform Workspaces to switch between different configurations based on the environment you are working on, while keeping the setup as clean and simple as possible. Are you interested in more articles about Terraform? Checkout How to Deploy ElasticSearch on GKE using Terraform and Helm!

Credits: Header image by Luca Cavallin on nylavak.com

We have read and selected a few articles and other resources about the latest developments around cloud technology so you don’t have to. Read further and keep yourself up-to-date in five minutes!

In a data-driven, global, always-on world, databases are the engines that let businesses innovate and transform. As databases get more sophisticated and more organizations look for managed database services to handle infrastructure needs, there are a few key trends we’re seeing.

Read more:
https://cloud.google.com/blog/products/databases/6-database-trends-to-watch

How does Anthos simplify hybrid & multicloud deployments?

Most enterprises have applications in disparate locations—in their own data centers, in multiple public clouds, and at the edge. These apps run on different proprietary technology stacks, which reduces developer velocity, wastes computing resources, and hinders scalability. How can you consistently secure and operate existing apps, while developing and deploying new apps across hybrid and multicloud environments? How can you get centralized visibility and management of the resources? Well, that is why Anthos exists!

Read more:
https://cloud.google.com/blog/topics/developers-practitioners/how-does-anthos-simplify-hybrid-multicloud-deployments

Automate your budgeting with the Billing Budgets API

Budgets are ideal for visibility into your costs but they can become tedious to manually update. Using the Billing Budgets API you can automate updates and changes with your custom business logic.

Read more:
https://cloud.google.com/blog/topics/developers-practitioners/automate-your-budgeting-billing-budgets-api

Amazon RDS for PostgreSQL Integrates with AWS Lambda

You can now invoke Lambda functions from stored procedures and user-defined functions inside your PostgreSQL database using the aws_lambda PostgreSQL extension provided with RDS for PostgreSQL. You can choose to execute the lambda synchronously or asynchronously. Also you get access to the execution log, the error and the returned result for further processing. This probably also means you can call Lambda function from triggers on tables to respond to data changes!

Read more:
https://aws.amazon.com/about-aws/whats-new/2021/04/amazon-rds-postgresql-integrates-aws-lambda/

AWS SAM CLI now supports AWS CDK applications (public preview)

Some features of SAM now become available for CDK developers. Local testing before synthesizing of Lambda functions and API endpoints. And building your Lambda resources into deployable zip files. All this while still being able to use the CDK CLI for creating, modifying, and deploying CDK applications.

Read more:
https://aws.amazon.com/about-aws/whats-new/2021/04/aws-sam-cli-supports-aws-sdk-applications-public-preview/

AWS CloudFront Functions

It is now possible to manipulate requests (viewer request and viewer response) with new CloudFront Functions. These are Javascript functions that execute within 1 millisecond on the outer edge locations (218 in stead of 13!). You can’t do really crazy stuff, most suitable for header manipulations, redirects, url rewrites and token validations. The cost of using functions is 1/6 of the Lambda@Edge price!

Read more:
https://aws.amazon.com/blogs/aws/introducing-cloudfront-functions-run-your-code-at-the-edge-with-low-latency-at-any-scale/

Amazon EC2 Auto Scaling introduces Warm Pools

You can now keep some pre-initialized stopped instances handy to quickly start them when autoscaling needs them. This is much cheaper then keeping live instances ready. Starting these pre-initialized is then a matter of 10s of seconds in stead of minutes.

Read more:
https://aws.amazon.com/about-aws/whats-new/2021/04/amazon-ec2-auto-scaling-introduces-warm-pools-accelerate-scale-out-while-saving-money/

Credits: Header image by Luca Cavallin on nylavak.com

Did you even wonder how to connect to CloudSQL with IAM authentication? Since this year, CloudSQL for PostgreSQL allows IAM users and IAM service accounts to login as a database user. In this short blog I will show you how to do this using Terraform.

(more…)

In this blog I introduce the Renovate bot, a great tool to automate the update application dependencies in your source code.

(more…)

I was recently tasked with deploying ElasticSearch on GKE using Terraform and Helm, and doing so in most readable way possible. I wasn’t very familiar with Helm before, so I did some research to find approach that would fulfill the requirements. In this post I will share with you the Terraform configuration I used to achieve a successful deployment.

What is Helm?

Helm is, at its most basic, a templating engine to help you define, install, and upgrade applications running on Kubernetes. Using Helm, you can leverage its Charts feature, which are simply Kubernetes YAML configuration files (that can be further configured and extended) combined into a single package that can be used to deploy applications on a Kubernetes cluster. To be able to use Helm via Terraform, we need to define the corresponding provider and pass the credentials needed to connect to the GKE cluster.

provider "helm" {
  kubernetes {
    token                  = data.google_client_config.client.access_token
    host                   = data.google_container_cluster.gke.endpoint
    cluster_ca_certificate = base64decode(data.google_container_cluster.gke.master_auth[0].cluster_ca_certificate)
  }
}

Terraform configuration

I am defining an helm_release resource with Terraform, which will deploy the ElasticSearch cluster when applied. Since I am using an Helm chart for the cluster, doing so is incredibly easy. All I had to do was tell Helm the name of the chart to use and where it is located (repository), along with the version of ElasticSearch that I would like to use.
With the set blocks instead, I can override the default values from the template: this makes it easy to select an appropriate storage class, amount of storage and in general any other piece of configuration that can be changed (you have to refer to the documentation of the chart itself to see which values can be overridden), directly from Terraform.

resource "helm_release" "elasticsearch" {
  name       = "elasticsearch"
  repository = "https://helm.elastic.co"
  chart      = "elasticsearch"
  version    = "6.8.14"
  timeout    = 900

  set {
    name  = "volumeClaimTemplate.storageClassName"
    value = "elasticsearch-ssd"
  }

  set {
    name  = "volumeClaimTemplate.resources.requests.storage"
    value = "5Gi"
  }

  set {
    name  = "imageTag"
    value = "6.8.14"
  }
}

Creating a new storage class

I then had to provision a new storage class, which will be used by the ElasticSearch cluster to store data. The configuration below sets up the SSD (SSD is recommended for such purpose, since it’s faster than a regular HDD) persistent disk that I referenced in the main configuration above.

resource "kubernetes_storage_class" "elasticsearch_ssd" {
  metadata {
    name = "elasticsearch-ssd"
  }
  storage_provisioner = "kubernetes.io/gce-pd"
  reclaim_policy      = "Retain"
  parameters = {
    type = "pd-ssd"
  }
  allow_volume_expansion = true
}

Summary

In this blog post I have shown you how to deploy ElasticSearch on GKE using Terraform and Helm. The required configuration is simple and very readable, lowering the barrier to handling all of your infrastructure via Terraform, rather than, for example, using Cloud Marketplace, managed services, or other custom solutions.

Credits: Header image by Luca Cavallin on nylavak.com

When I test new ideas and features, I often rack up accidental cloud cost on Google Cloud Platform. I forget to delete things I created, and end up paying for resources that I kept running as a result.

(more…)

When you want to create a Python command line utility for Google Cloud Platform, it would be awesome if you could use the active gcloud credentials in Python. Unfortunately, the Google Cloud Client libraries do not support using the gcloud credentials. In this blog, I will present a small Python library which you can use to do just that.

How use the gcloud credentials in Python

It is really simple. You install the package gcloud-config-helper and call the default function, as shown below:

import gcloud_config_helper
credentials, project = gcloud_config_helper.default()

You pass the credentials to a service client as follows:

c = compute_v1.InstancesClient(credentials=credentials)
for zone, instances in c.aggregated_list(request={"project": project}):
    for instance in instances.instances:
        print(f'found {instance.name} in zone {zone}')

That is all there is to it! Check out the complete example of using the gcloud configured credentials.

How does it work?

The library executes the command gcloud config config-helper. This commands provides authentication and configuration data to external tools. It returns an access token, an id token, the name of the active configuration and all associated configuration properties as show below:

    configuration:
      active_configuration: playground
      properties:
        core:
          account: markvanholsteijn@binx.io
          project: playground
        ...
    credential:
      access_token: ya12.YHYeGSG8flksArMeVRXsQB4HFQ8aodXiGdBgfEdznaVuAymcBGHS6pZSp7RqBMjSzHgET08BmH3TntQDOteVPIQWZNJmiXZDr1i99ELRqDxDAP8Jk1RFu1xew7XKeQTOTnm22AGDh28pUEHXVaXtRN8GZ4xHbOoxrTt7yBG3R7ff9ajGVYHYeGSG8flksArMeVRXsQB4HFQ8aodXiGdBgfEdznaVuAymcBGHS6pZSp7RqBMjSzHgET08BmH3TntQDOteVPIQWZNJmiXZDr1i99ELRqDxDAP8Jk1RFu1xew7XKeQTOTnm22AGDh28pUEHXVaXtRN8GZ4xHbOoxrTt7yBG3R7ff9ajGV
      id_token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2FjY291bnRzLmdvb2dsZS5jb20iLCJhenAiOiI5OTk5OTk5OTk5OS5hcHBzLmdvb2dsZXVzZXJjb250ZW50LmNvbSIsImF1ZCI6Ijk5OTk5OTk5OTk5LmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tIiwic3ViIjoiMTExMTExMTExMTEyMjIyMjIyMjIyMjIiLCJoZCI6ImJpbnguaW8iLCJlbWFpbCI6Im1hcmt2YW5ob2xzdGVpam5AYmlueC5pbyIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJhdF9oYXNoIjoic2Rmc2prZGZqc2tkZnNrZGZqcyIsImlhdCI6MTU4ODcwMTY4MywiZXhwIjoxNTg4NzA1MjgzfQ.-iRKFf69ImE93bvUGBxn3Fa5aPBjhyzeWfLzuaNdIGI
      token_expiry: '2020-05-05T19:01:22Z'

When the token expires, the library will call the helper again to refresh it. Note that Google is unsure whether the config-helper is a good thing. If you read gcloud config config-helper --help, you will notice the following sentence:

This command is an internal implementation detail and may change or disappear without notice.

For the development of command line utilities which integrate into the Google Cloud SDK ecosystem, it would be really handy if Google would provide an official way to obtain the active gcloud configuration and credentials.

conclusion

With the help of this library, it is possible to create a command line utility in Python for the Google Cloud Platform using the gcloud credentials in Python. It is unfortunate that Google marks the config-helper as a volatile interface. Given the simplicity of the interface, I trust this library will be able to deal with any future changes. It would be even better, if Google would provide official support.

We also created a library for authenticating with gcloud credentials in Go.

Image by Kerstin Riemer from Pixabay

What is the next big thing? That’s Cloud. But what is the value of cloud training? And why should you train with an authorized training partner?

In this video and blog post, Max Driessen, chief of cloud training, and Martijn van de Grift, cloud consultant and authorized instructor for AWS and GCP, discuss the importance of cloud training and reasons to train with an ATP.

How important is cloud training?

“That depends on the phase your organization is in, and on the people within your organization”, Max starts.

“Generally speaking, proper cloud training is a great enabler to speed up development and go-to-market, and to implement best-practices such as cost optimization, performance efficiency, and security".

“Organizations usually start their cloud journey by implementing virtual machines or migrating small databases. That isn’t really different from what their engineers have been doing all along”, Martijn adds.

“What is essential in this phase, is that the management defines a vision on cloud and a roadmap. Managers need to become familiar with operating models in the cloud, different cost models and cost allocation".

“As organizations advance towards using the cloud-native building blocks, new challenges arise. Organizations in this phase can really benefit from fundamental training”, says Martijn.

“If a company is already really progressing well with the cloud, they move past the need for the fundamentals towards a potential to dive deep with specific technical cloud training to speed up their development".

What kind of training needs do you see?

Without a blink, Max sums up: “I see three different general training needs: Cloud Awareness, Cloud Transformation, Cloud Skills".

“Cloud Awareness training is for companies that are setting their first steps with the cloud. This training is ideal for managers, project leads, product owners, basically anyone within an organization who needs to get a better grip on the cloud. To develop an understanding of the capabilities, the capacity, the return on investment of cloud even," Max explains.

“Cloud Transformation is for companies that are already in a hybrid or a migration process, who need support and skills within that process of transformation". Max continues. “They will learn about cloud target operating models, defining a business case around cloud, but also, they will learn about the organizational impact of a cloud transformation".

Martijn says: ”Cloud Skills are off-the-shelf training from the curriculum of AWS, GCP, or Azure. These learning paths focus on roles like architects, developers, security or data specialists, and even help you to get prepared to take a certification exam. Each program in the curriculum of AWS, GCP, or Azure, is great preparation for an exam".

“If you want to take the exam, that is something you have to decide on your own, but the training will definitely help you to develop the right competencies", Max adds.

Martijn concludes: “Obviously, you also need to gain practical experience. That is why it’s so useful to have an instructor who has a couple of years of practical experience under his belt".

Why train with Binx?

“Three reasons stand out: First of all, we make sure that each individual receives the best training experience possible by meeting you where you are. Secondly, we combine our training activities with high-end consultancy, so we have first-hand experience with cloud projects. Thirdly, but certainly not the least: we are an authorized training partner for AWS, and GCP," Max says with a smile.

But how do you define the training need and the right training? “If you identify a training need, but you are not really sure on which topic you need training, we have an assessment for you. We can help to identify the knowledge gaps, on which domains. We can help you to fill in the blanks. An assessment could also be a questionnaire, where we have a conversation with a company, to check where they are and to assess where we can be of any help," Max explains.

Martijn has been working as a cloud engineer for some years now. Does that provide added value when delivering training?

“Being not only a consultancy firm but also a training partner, is definitely a big plus. We are working with clients four or five days a week, helping the customers to solve their problems. And on top of this, we are also delivering training".

"We know first-hand what the challenges are that customers are typically facing”, explains Martijn. “And we are also up to date in all the courses and materials. During the training courses, you work together with the instructor. There is plenty of time to ask the instructor. The instructor will tap into his experience and knowledge to explain all the topics. This is definitely a big benefit over an online course".

“Being an authorized training partner with AWS and GCP is very special. Especially if you are authorized in for multiple cloud service providers. That is something that rarely happens. The bar is set high. For example, scoring a 9 out of 10 in feedback for a training", Max says.

Martijn adds: “To become an authorized trainer, you also have to meet a high standard. You have to know everything about the cloud providers and their services. About Amazon, Google, and Azure. Instructors are tested every year by the cloud service provider itself".

Can you share some of your personal training experiences?

Max starts to smile as he recalls his first cloud training a while ago: “In the meantime, I joined three courses, and it has been quite a journey. It was quite challenging at the start, but having a good trainer really made the difference. In my case, I was lucky to have Martijn as the trainer."

Martijn says “My favorite training is the Cloud Architect Professional training. This training really gives a global overview of everything a cloud provider has to offer you. It’s actually more than the cloud from a developer point-of-view, or only from a security point-of-view. It really gives you an overview of the services and how you can use them properly."

Interested to learn more about the benefits of training for you or your organization? Get in touch with Max Driessen to discuss the possibilities.

We have read and selected a few articles and other resources about the latest developments around cloud technology so you don’t have to. Read further and keep yourself up-to-date in five minutes!

Inventory management with BigQuery and Cloud Run

Cloud Run is often used just as a way of hosting websites, but there’s so much more you can do with it. In this blog, Aja Hammerly (Developer Advocate) will show us how to use Cloud Run and BigQuery together to create an inventory management system. The example uses a subset of the Iowa Liquor Control Board data set to create a smaller inventory file for a fictional store. 

Read more:
https://cloud.google.com/blog/topics/developers-practitioners/inventory-management-bigquery-and-cloud-run

3 common serverless patterns to build with Workflows

The Workflows orchestration and automation service is a serverless product designed to orchestrate work across Google Cloud APIs as well as any HTTP-based API available on the internet, and it has recently reached General Availability. At the same time, Workflows has been updated with a preview of Connectors, which provide seamless integration with other Google Cloud products to design common architecture patterns that can help you build advanced serverless applications. 

Read more:
https://cloud.google.com/blog/products/application-development/building-serverless-apps-with-workflows-and-connectors

Google Cloud offers lots of products to support a wide variety of use cases. To make it easier to orient yourself, Google has created a set of resources that makes it easy to familiarize yourself with the Google Cloud ecosystem. You can use these resources to quickly get up to speed on different products and choose those that you’re most interested in for a deeper dive into documentation and other available resources. 

Read more:
https://cloud.google.com/blog/topics/developers-practitioners/back-popular-demand-google-cloud-products-4-words-or-less-2021-edition

Node.js 14.x runtime now available in AWS Lambdar

While it is still OK to use node.js 12.x until the end of 2021, version 14.x brings a few things that make the life of a developer a lot nicer. Most of the changes come from the newly upgraded Javascript engine: v8.1 on which nodejs runs! New features include nullish coalescing operator, nullish assignment and optional chaining.

Read more:
https://aws.amazon.com/blogs/compute/node-js-14-x-runtime-now-available-in-aws-lambda/

Cost savings on AWS

AWS recently introduced the “Compute Savings Plan” for Lambda. It allows for 12% to 17% discount on the expenses. So if you spend say $730 per month ($1 per hour) on lambda you can commit for $640 per month ($0.88 per hour). Leading to yearly savings of $1080. ($1 per hour is a LOT of Lambda usage though so probably your actual situation will not benefit this much).

Also it is now possible to save costs on CloudFront outbound traffic. In as use case where the traffic amounts to $50 per day the savings are around $800 per month!

Read more:
https://aws.amazon.com/blogs/aws/savings-plan-update-save-up-to-17-on-your-lambda-workloads/ and https://aws.amazon.com/about-aws/whats-new/2021/02/introducing-amazon-cloudfront-security-savings-bundle/

AWS DynamoDB audit logging and monitoring with CloudTrail

DynamoDB change logging was usually done through streams. Now it is possible to do logging with CloudTrail data events. This also allows for access logging including the queries used to access the data. These Cloudwatch data events also support access logging for S3, Lambda (invokes) and Managed Blockchain.

Read more:
https://aws.amazon.com/blogs/database/amazon-dynamodb-now-supports-audit-logging-and-monitoring-using-aws-cloudtrail/

S3 Object Lambda

A new type of Lambda can be attached to an S3 bucket that allows you to change the response for get requests. The Lambda is attached to an S3 Control Access Point which will lead the requests to the Lambda. Inside the Lambda you you can change header, body and status code based on your requirements. Use cases could be: resizing images, partially hiding some data based on requester identity or adding headers from another datasource.

https://aws.amazon.com/blogs/aws/introducing-amazon-s3-object-lambda-use-your-code-to-process-data-as-it-is-being-retrieved-from-s3/

Credits: Header image by Luca Cavallin on nylavak.com

When building custom images for Google Cloud Platform using Hashicorp Packer, you either specify a fixed, or latest for the Google source image version. When specifying a fixed version, you run the risk of building with a deprecated image. If you use latest, your build may introduce changes unknowingly. As we adhere to the principle to keep everything under version control, we created a utility which ensures that your packer template is always referring to the latest, explicit version of an image.

(more…)

Since version 0.13 terraform supports the notion of a provider registry. You can build and publish your own provider in the public registry. At the moment, there is no private registry implementation available. In this blog, I will show you how to create a private terraform provider registry on Google Cloud Storage. A utility will generate all the required documents to create a static registry.

(more…)

At a current customer, we’re using GCP’s IAP tunnel feature to connect to the VMs.
When deploying fluentbit packages on existing servers for this customer, I decided it would save some time if I would make an Ansible playbook for this job.
As I only have access to them through IAP, I ran into a problem; Ansible does not have an option to use gcloud compute ssh as a connection type.

It turns out, you do have the option to override the actual ssh executable used by Ansible. With some help from this post, and some custom changes, I was able to run my playbook on the GCP servers.

Below you will find the configurations used.

ansible.cfg

contents:

[inventory]
enable_plugins = gcp_compute

[defaults]
inventory = misc/inventory.gcp.yml
interpreter_python = /usr/bin/python

[ssh_connection]
# Enabling pipelining reduces the number of SSH operations required
# to execute a module on the remote server.
# This can result in a significant performance improvement 
# when enabled.
pipelining = True
scp_if_ssh = False
ssh_executable = misc/gcp-ssh-wrapper.sh
ssh_args = None
# Tell ansible to use SCP for file transfers when connection is set to SSH
scp_if_ssh = True
scp_executable = misc/gcp-scp-wrapper.sh

First we tell Ansible that we want to use the gcp_compute plugin for our inventory.
Then we will point Ansible to our inventory configuration file from which the contents can be found below.

The ssh_connection configuration allows us to use gcloud compute ssh/scp commands for our remote connections.

misc/inventory.gcp.yml

contents:

plugin: gcp_compute
projects:
  - my-project
auth_kind: application
keyed_groups:
  - key: labels
    prefix: label
  - key: zone
    prefix: zone
  - key: (tags.items|list)
    prefix: tag
groups:
  gke          : "'gke' in name"
compose:
  # set the ansible_host variable to connect with the private IP address without changing the hostname
  ansible_host: name

This will enable automatic inventory of all the compute instances running in the my-project GCP project.
Groups will be automatically generated based on the given keyed_groups configuration and in addition I’ve added a gke group based on the VM’s name.
Setting the Ansible_host to the name will make sure our gcloud ssh command will work. Otherwise Ansible will pass you the instance IP address.

misc/gcp-ssh-wrapper.sh

contents:

#!/bin/bash
# This is a wrapper script allowing to use GCP's IAP SSH option to connect
# to our servers.

# Ansible passes a large number of SSH parameters along with the hostname as the
# second to last argument and the command as the last. We will pop the last two
# arguments off of the list and then pass all of the other SSH flags through
# without modification:
host="${@: -2: 1}"
cmd="${@: -1: 1}"

# Unfortunately ansible has hardcoded ssh options, so we need to filter these out
# It's an ugly hack, but for now we'll only accept the options starting with '--'
declare -a opts
for ssh_arg in "${@: 1: $# -3}" ; do
        if [[ "${ssh_arg}" == --* ]] ; then
                opts+="${ssh_arg} "
        fi
done

exec gcloud compute ssh $opts "${host}" -- -C "${cmd}"

Ansible will call this script for all remote commands when the connection is set to ssh

misc/gcp-scp-wrapper.sh

contents:

#!/bin/bash
# This is a wrapper script allowing to use GCP's IAP option to connect
# to our servers.

# Ansible passes a large number of SSH parameters along with the hostname as the
# second to last argument and the command as the last. We will pop the last two
# arguments off of the list and then pass all of the other SSH flags through
# without modification:
host="${@: -2: 1}"
cmd="${@: -1: 1}"

# Unfortunately ansible has hardcoded scp options, so we need to filter these out
# It's an ugly hack, but for now we'll only accept the options starting with '--'
declare -a opts
for scp_arg in "${@: 1: $# -3}" ; do
        if [[ "${scp_arg}" == --* ]] ; then
                opts+="${scp_arg} "
        fi
done

# Remove [] around our host, as gcloud scp doesn't understand this syntax
cmd=`echo "${cmd}" | tr -d []`

exec gcloud compute scp $opts "${host}" "${cmd}"

This script will be called by Ansible for all the copy tasks when the connection is set to ssh

group_vars/all.yml

contents:

---
ansible_ssh_args: --tunnel-through-iap --zone={{ zone }} --no-user-output-enabled --quiet
ansible_scp_extra_args: --tunnel-through-iap --zone={{ zone }} --quiet

Passing the ssh and scp args through the group-vars makes it possible for us to set the zone to the VM’s zone already known through Ansible’s inventory.
Without specifying the zone, gcloud will throw the following error:

ERROR: (gcloud.compute.ssh) Underspecified resource [streaming-portal]. Specify the [--zone] flag.

Hopefully this post will help anyone that ran into the same issue!

When you create a Google service account key file for an external system, the private key has to
be transported. In addition, the generated key is valid until January 1st in
the year 10000. In this blog I will show you, how an external system can identity itself as the
service account without exposing the private key.

(more…)

I recently had to optimize the performance of a PHP-based API on Cloud Run. After a performance test, we discovered that the API became very slow when we put some serious load on it (with response times exceeding 10 seconds). In this post you’ll learn what changes I made to get that down to a stable 100ms.

The API uses PHP 7.4, Laravel 8.0 and MySQL on Cloud SQL (the managed database on Google Cloud). The API needs to handle at least 10,000 concurrent users. The container image we deploy to Cloud Run has nginx and PHP-FPM.

This is what I did to improve the response times. (Don’t worry if not everything makes sense here, I’ll explain everything).

  • Matching the number of PHP-FPM workers to the maximum concurrency setting on Cloud Run.
  • Configuring OPcache (it compiles and caches PHP scripts)
  • Improving composer auto-loading settings
  • Laravel-specific optimizations including caching routes, views, events and using API resources

Matching Concurrency and Workers

The application uses nginx and PHP-FPM, which is a process manager for PHP. PHP is single threaded, which means that one process can handle one (exactly one) request at the same time. PHP-FPM keeps a pool of PHP workers (a worker is a process) ready to serve requests and adds more if the demand increases.
It’s a good practice to limit the maximum size of the PHP-FPM worker pool, to make sure your resource usage (CPU and memory) is predictable.

To configure PHP-FPM for maximum performance, I first set the process manager type for PHP-FPM to static, so that the specified number of workers are running at all times and waiting to handle requests. I did this by copying a custom configuration file to the application’s container and configuring the environment so that these options will be picked up by PHP-FPM (you must copy the configuration where it is expected, in my case, into /usr/local/etc/php-fpm.d/). The settings I needed are:

pm = static
pm.max_children = 10

However, if you set a limit, and more requests come to a server than the pool can handle, requests start to queue, which increases the response time of those requests:

Nginx and php-fpm request model

Limiting Concurrent Requests on Cloud Run

To avoid request queuing in nginx, you’ll need to limit the number of requests Cloud Run sends to your container at the same time.

Cloud Run uses request-based autoscaling. This means it limits the amount of concurrent requests it sends to a container, and adds more containers if all containers are at their limit. You can change that limit with the concurrency setting. I set it to 10, which I determined is the maximum number of concurrent requests a container with 1GB of memory and 1vCPU can take for with this application.

Cloud Run concurrent requests with Nginx and php-fpm

You really want to make sure Cloud Run’s concurrency setting matches the maximum number of PHP-FPM workers! For example, if Cloud Run sends 100 concurrent requests to a container before adding more containers, and you configured your PHP-FPM to start only 10 workers, you will see a lot of requests queuing.

If these tweaks aren’t enough to reach the desired performance, check the Cloud Run metrics to see what the actual utilization percentages are. You might have to change the amount of memory and vCPUs available to the container. The downside of this optimization is that more containers will be running due to lower concurrency, resulting in higher costs. I also noticed temporary delays when new instances are starting up, but this normalizes over time.

Configuring OPCache

OPCache is a default PHP extension that caches the compiled scripts in memory, improving response times dramatically. I enabled and tweaked OPCache settings by adding the extension’s options to a custom php.ini file (in my case, I put it in the /usr/local/etc/php/conf.d/ directory). The following is a generic configuration that you can easily reuse, and you can refer to the documentation for the details about every option.

opcache.enable=1
opcache.memory_consumption=128
opcache.interned_strings_buffer=64
opcache.max_accelerated_files=32531
opcache.validate_timestamps=0
opcache.save_comments=1
opcache.fast_shutdown=0

Optimizing Composer

Composer is a dependency manager for PHP. It lets you specify the libraries your app needs, and downloads them for you to a directory. It also generates an autoload configuration file, which maps import paths to files.
If you pass the --optimize-autoloader flag to composer, it will generate this file only once, and doesn’t dynamically update it if you add new code. While that is convenient in development (your changes show up immediately), in production it can make your code really slow.

You can optimize Composer’s autoloader passing the --optimize-autoloader flag like this:

composer install --optimize-autoloader --no-dev

Laravel-specific optimizations

The application I optimized is built with Laravel, which provides a number of tools that can help improving the performance of the API. Here’s what I did on top of the other tweaks to get the response times below 100ms.

  • I have leveraged Laravel’s built-in caching features during builds to reduce start-up times. There are no downsides to these tweaks, except that you won’t be able to use closure-defined routes (they can’t be cached). You can cache views, events and routes with these commands:

    php artisan view:cache
    php artisan event:cache
    php artisan route:cache

    Avoid running php artisan config:cache since Laravel ignores environment variables if you cache the configuration.

  • Using Laravel API Resources further improves the response times of your application. This has proven to be much faster than having the framework automatically convert single objects and collections to JSON.

Summary

In this blog, I shared with you what I learned from optimizing the performance of a PHP-based API on Cloud Run. All of the tweaks together helped cut response times to one tenth of the original result, and I think the most impact was made by matching concurrency and PHP-FPM workers (if you’re in a hurry, do only this). Watching the application metrics has been fundamental throughout the performance testing phase, just as inspecting Cloud Run logs after each change.

If your application still shows poor performance after these changes, there are other tweaks you can make to improve response times, which I haven’t discussed here.

  • Increase PHP memory limits if needed
  • Check MySQL for slow queries (often due to missing indexes)
  • Cache the responses with a CDN
  • Migrate to PHP 8.0 (up to 3x faster)

Would you like to learn more about Google Cloud Run? Check out this book by our own Wietse Venema.

Credits: Header image by Serey Kim on Unsplash

We have read and selected a few articles and other resources about the latest developments around cloud technology so you don’t have to. Read further and keep yourself up-to-date in five minutes!

Introducing GKE Autopilot: a revolution in managed Kubernetes

GKE Autopilot manages the infrastructure, letting you focus on your software. Autopilot mode automatically applies industry best practices and can eliminate all node management operations, maximizing your cluster efficiency (you will not be charged for idle nodes), helping to provide a stronger security posture and ultimately simplifying Kubernetes.

Read more:
https://cloud.google.com/blog/products/containers-kubernetes/introducing-gke-autopilot

What is Amazon Detective?

Amazon Detective makes it easy to analyze, investigate, and identify the root cause of potential security issues or suspicious activities. It automatically collects log data from your AWS resources and uses machine learning to build a linked set of data that enables you to easily conduct faster and more efficient security investigations.

Watch video:
https://www.youtube.com/watch?v=84BhFGIxIqg

Introduction to Amazon Timestream –Time Series Database

Amazon Timestream is a serverless time series database service for IoT and operational applications that makes it easy to store and analyze trillions of events per day. Amazon Timestream saves you time and cost in by keeping recent data in memory and moving historical data to a cost optimized storage tier.

Watch video:
https://www.youtube.com/watch?v=IsmhOkimHyI

What are my hybrid and multicloud deployment options with Anthos?

Anthos is a managed application platform that extends Google Cloud services and engineering practices to your environments so you can modernize apps faster. With Anthos, you can build enterprise-grade containerized applications with managed Kubernetes on Google Cloud, on-premises, and other cloud providers.

Read more:
https://cloud.google.com/blog/topics/developers-practitioners/what-are-my-hybrid-and-multicloud-deployment-options-anthos

Accelerate data science workflows with Looker

Looker is a modern business intelligence (BI) and analytics platform that is now a part of Google Cloud. It’s a full-fledged data application and visualization platform and it allows users to curate and publish data, integrating with a wide range of endpoints in many different formats ranging from CSV, JSON, Excel, files to SaaS and in-house built custom applications.

Read more:
https://cloud.google.com/blog/topics/training-certifications/how-to-use-looker-on-google-cloud-for-data-governance

While doing Cloud Native migrations, we depend on container images a lot. In this mini-blog I will present you with the
simplest Google Cloud Build config file that will build both snapshot and release versions at the same time.

(more…)

When you work with AWS ECS and you apply the least-privileged security principle, you create an IAM role for each task. While developing the software, you may want to test the role too. However, this is not easily done. In this blog we will show you how to can be done in the official way, and in a slightly more alternative way with the iam-sudo utility for development purposes.

(more…)

In the Xebia First Candidates restaurant, Xebia business unit managers share what excites them about working for their respective units.

Love is like a cloud migration: Thrilling at the start, but once you get going, you never want to go back.

Smooth Operations

Besides being an avid gamer, Elaine Versloot is also the chief of Operations at cloud boutique Binx.io. Here, she focuses on turning the dynamics of cloud migration and modernization into a smooth operation.

We are looking for people with a passion for cloud. Developing cloud-native applications should be in the blood. We find it just as important that individuals are fun to be around, the social aspect is also very important.

When someone found his or her expertise and can talk about this with passion. I love that! Nothing gets me going more than that excitement in someone’s eye.

Go On a Blind Date With Elaine

Interested to learn more about the way Binx approaches cloud?

Go on a blind business date with Elaine

I recently passed the Google Cloud Associate Cloud Engineer certification exam. This post describes what the exam is about, how I prepared (with links to useful resources), and the strategies I used to answer the questions in under two minutes per question (skip down to Answering Strategies if that’s just what you want to know).

If you’re reading this before February 9, you can still join my webinar! I’ll help you put together an actionable roadmap to pass the certification exam and share more insights.

What’s an Associate Cloud Engineer?

First, some background if you don’t know what this is about, and are trying to figure out if you should take the certification. The Associate Cloud Engineer certification is one of the most technical Google Cloud certifications: I think it is even harder than the Professional Cloud Architect exam I’m currently preparing for. Nevertheless, this is the first stepping stone into Google Cloud, making it an obvious choice to get started if you are a software engineer, like I am.

Google Cloud defines no prerequisites for this exam, but recommends a minimum of six months hands-on experience on the platform. The topics covered in the certification are:

  • Projects, Billing, IAM
  • Compute Engine, App Engine and Kubernetes Engine
  • VPC and Networking
  • Cloud Storage
  • Databases (included are for example Cloud SQL, Memorystore, and Firestore)
  • Operations Suite (Cloud Logging and other services). You might know this as Stackdriver.

The next step in the learning journey after the Associate Cloud Engineer exam is the Professional Cloud Architect certification. From there, you can choose to specialize in Data and Machine Learning Engineering, Cloud Development and DevOps, or Security and Networking. (Check out the certification paths here).

Why Did I Choose Google Cloud?

Google Cloud strikes me as the most developer-friendly cloud provider. At the same time, it is also the smallest of the three main players but it is growing fast, meaning lots of room for new experts. I like the certification program because it is role-based (instead of product-centered), preparing not just for this provider, but the position itself.

About Me

I want to tell you a little bit about me, so you know how to interpret my advice. I have been a Software Engineer for the past 8+ years, doing mostly full-stack development. The ability to work on what “powers” the internet and an interest for what’s under the hood of software systems brought me to the cloud. Furthermore, the managed services provided by cloud providers make it easy to add all sorts of features into applications, from sending emails to advanced machine learning – these days nothing is stopping you from starting your next company from the comfort of your bedroom.

I grew up in Italy and moved to the Netherlands a few years ago, where I now live with my wife and cat. When I am not working on clouds and other atmospheric phenomena, I enjoy photography and cycling.

Feel free to connect with me on LinkedIn or send me an email at lucacavallin@binx.io!

What Does the Exam Look Like?

This is how a certification exam works: you have two hours to answer 50 multiple choice and multiple selection questions. That means you’ll have roughly two minutes per question, with 20 minutes left at the end to review your answers.

The exam can be taken remotely or in person, costs 125 USD (roughly 100 EUR), and can be retaken after a cool down period if you fail. The certification is valid for two years (you’ll have to take the full exam again to recertify.

Exam Guide

Google publishes an exam guide for every certification. It details the topics of the certification, providing details of what you need to know for each of the areas.

I used the exam guide to track my progress while studying. You can make a copy of it and use color coding to highlight the topic you feel confident about and those that need more of your attention.

Learning resources

There are many books and online courses you can take to prepare for the exam, some more effective than others. For most people, a combination of written and audio-visual material works best. These are the resources that helped me best (and why):

  • Official Google Cloud Certified Associate Cloud Engineer Study Guide (book)
    • Most comprehensive resource
    • Questions after each chapter
    • Includes practice tests and flashcards
  • ACloudGuru (online course)
    • This is a good introduction (start with this course if you have no Google Cloud experience)
    • You also need the “Kubernetes Deep Dive” course
    • Hands-on labs
    • This course does not cover everything you need to know

Practice Tests

Practice tests are a key part of your preparation because they let you test your knowledge in a setting similar to the actual exam. If you finish the test, they provide you with a detailed explanation for each question, documenting the correct and wrong answers.

Take note of the topics that require attention and review the documentation accordingly. Once you consistently score at least 90% on practice tests, you are ready for the exam.

The Official Study Guide book provides review questions at the end of each chapter and an online portal with two practice tests. The ACloudGuru course also has a practice exam you can take, and you can find similar resources on Udemy.

Answering Strategies

If you do the math, you’ll see that you only have two minutes to answer every question, which is not much, given that each of them is quite lengthy. Here are the strategies I used when answering the questions.

  • Identify the core question
  • Review it carefully since a single word can make the difference
  • Eliminate answers that are wrong or in conflict with the question
  • Choose the cheapest and most secure option
  • Read the question again, keeping in mind the answer you chose

You can also mark answers during the test to come back to them later. While you don’t have much time left for review at the end, this practice will save you from over thinking and losing valuable time.

Taking the Remote Proctored Exam

If you decide to take the exam remotely, once you have arranged it, you will have to install a Sentinel tool provided by the testing authority, verify your identity and pass a number of checks.

TIP: The operator taking you through the process doesn’t talk, so you need to scroll the chat for their questions. For me, the chat didn’t auto-scroll, so it was a bit awkward at first.

Join My Webinar

If you’re reading this before February 9, you can still register for my webinar! I’ll share all my learnings. I would love to meet you and answer any questions! Make sure to register here.

Summary

In this post, I shared my experience preparing and taking the Google Cloud Associate Engineer Certification exam. Here are the four most important things to take away from this:

  • You can do this! If you focus your attention and take as many practice tests as possible, and carefully review your correct and incorrect answers, you’ll pass the exam.
  • I think the Official Study Guide is hands down the best resource to use for preparation.
  • It is very useful to have real-world experience and practice with gcloud CLI and Kubernetes.
  • During the exam, attention to detail is important. Read every question carefully and read the question again after choosing your answer.

Credits: Header image by KAL VISUALS on Unsplash

We have read and selected a few articles and other resources about the latest and most significant developments around cloud technology so you don’t have to. Read further and keep yourself up-to-date in five minutes!

Introducing WebSockets, HTTP/2 and gRPC bidirectional streams for Cloud Run

Support for streaming is an important part of building responsive, high-performance applications. With these capabilities, you can deploy new kinds of applications to Cloud Run that were not previously supported, while taking advantage of serverless infrastructure.

Read more:
https://cloud.google.com/blog/products/serverless/cloud-run-gets-websockets-http-2-and-grpc-bidirectional-streams

Amazon Location – Add Maps and Location Awareness to Your Applications

Amazon Location Service gives you access to maps and location-based services from multiple providers on an economical, pay-as-you-go basis. You can use it to display maps, validate addresses, turn an address into a location, track the movement of packages and devices, and more.

Read more:
https://aws.amazon.com/blogs/aws/amazon-location-add-maps-and-location-awareness-to-your-applications/

Eventarc brings eventing to Cloud Run and is now GA

Eventarc is a new eventing functionality that lets developers route events to Cloud Run services. Developers can focus on writing code to handle events, while Eventarc takes care of the details of event ingestion, delivery, security, observability, and error handling.

Read more:
https://cloud.google.com/blog/products/serverless/eventarc-is-ga

Lifecycle of a container on Cloud Run

Serverless platform Cloud Run runs and autoscales your container-based application. You can make the most of this platform when you understand the full container lifecycle and the possible state transitions within it. In this article on the Google Cloud blog, our very own Wietse Venema takes you through the states, from starting to stopped.

Read more:
https://cloud.google.com/blog/topics/developers-practitioners/lifecycle-container-cloud-run

Amazon Aurora supports PostgreSQL 12

Amazon Aurora with PostgreSQL compatibility now supports major version 12. PostgreSQL 12 includes better index management, improved partitioning capabilities, and the ability to execute JSON path queries.

Read more:
https://aws.amazon.com/about-aws/whats-new/2021/01/amazon-aurora-supports-postgresql-12/

Keeping your Docker container image references up-to-date

Binx.io released an open source utility which allows you to list, detect and repair outdated references to container images, so you always build the most up-to-date images.

Read more:
https://binx.io/blog/2021/01/30/how-to-keep-your-dockerfile-container-image-references-up-to-date/

Over the past few months, our Cloud Engineers have proven to be so much more than cloud technology specialists. Out of necessity, they have also taken on side jobs as teacher, entertainer and nanny, all from the comfort of their own homes.

Helping Binx Parents Through The Lockdown

We are lucky to have a very creative department that looks after the wellbeing of all people within the company. Overcoming the impossible, they have made it a habit to look at what is possible in every given circumstance. The Nanny-on-Tour initiative is the latest rabbit to come from their hats. The objective is to help all the parents at Binx to get through the lockdown as comfortable as possible by installing a dedicated pool of qualified nannies!

Turning A Pool of Nannies Into Reality

We sat down with HR Advisors Roosmarijn van Zessen and Véronique Reijinga to discuss this wonderful initiative.

"We received several phone calls from colleagues who are really struggling with working from home and taking care of their children at the same time. The power of Binx, as part of Xebia, is that we help each other out", Roosmarijn van Zessen shared.

Véronique added: "In the past, we had a big pool of students, often children of colleagues, working for our hospitality crew or assisting in other departments. We’ve asked them if they would be open to work for Xebia as a nanny, and many of them responded enthusiastically! A couple of hospitality colleagues have also indicated that they would like to participate in this project. Our Xebia Nanny pool was born!"

It’s a Match!

Almost immediately after announcing this initiative, the HR team received many positive responses. The project was launched last week, and already they’ve been able to match four colleagues with a nanny from the nanny network!

It is great to see how happily surprised they were we already found a nanny for them. We will get through the homeschooling and lockdown together!

Whenever you build a container image, chances are that you are using public images as a base. But how do you keep your image up-to-date with the latest releases? In this blog I will introduce a utility which will allow you to keep your Dockerfile container image references up-to-date.

(more…)

Did you know that, once you have authenticated using the Google Cloud Platform SDK, the credential is valid for all eternity? With the Google Cloud session control tool you can limit the validity to as little as an hour.

(more…)

Hi, my name is Max Driessen, since November 1st, I joined the Binx team as Chief Cloud Training. In this role, it is my ambition to empower professionals with the right cloud skills by offering the best training programs. I have a background in information technology but lack hands-on programming experience.

In my second week at Binx, my colleague Martijn van de Grift— cloud consultant and authorized trainer for both Google Cloud Platform as well as Amazon Web Services— organized a Google Cloud Fundamentals training.

“This is really something for you,” Martijn told me. “This way you not only learn more about Google Cloud, but you also experience what it is like to attend a training.”

I decided to jump right in to experience what it’s like to attend a Binx training first hand.

Before I realized it, there I was; attending an online GCP Fundamentals training, with Martijn as the trainer. Obviously, this is a fundamentals training, most of our training courses go way deeper into specific topics, but for me this felt as the gateway to a new world. Although I roughly knew what to expect, this did not make me feel less nervous. But, I had one goal: to learn more about GCP.

Stretching the Comfort Zone

At the start of the training, I felt a bit tense about what was to come. That feeling vanished quickly due to the informal start where every attendee introduced him or herself briefly and shared personal learning goals with the rest of the group.

Then it was time to get going. The training was laid out in a logical order. After an initial introduction to the GCP interface, we went under the hood with various functionalities of the platform. Martijn alternated between theory and Q & A. Besides, a substantial part of the training was reserved for breakout sessions where every participant, including myself, could try out the various services with Qwiklabs exercises. These Qwiklabs provide exercises where you follow a step-by-step plan to learn how to use a specific service. What resonated with me was the way the trainer went out of his way to support every individual attendee with the successful completion of the labs.

It was obvious from the way that Martijn shared additional information on top of the materials, that he knows his way around Google Cloud like no other. From his experience, Martijn also handed out dozens of practical tips about the right way to implement the services. This resulted in a positive vibe, which also encouraged me to veer outside of my comfort zone further and further.

What Ground Was Covered?

This GCP Fundamentals training introduced me to a broad range of cloud concepts and GCP services. Besides the general introduction, I also learned about Virtual Machines and different types of Storage. Think about topics like cloud storage, SQL, Datastore, Bigtable, Spanner. Of course, containers and Kubernetes were not overlooked. We also took a look at App Engine, Stackdriver and Infrastructure as Code. Finally, there was even some time left to cover the basics of BigQuery and Managed Machine Learning APIs.

At the end of the day, it felt like I had just made it through one full week of training. 😉

Applicability

As mentioned, I had no previous experience of working with cloud platforms. For the first topics this was no problem, but the hands-on labs will be easier to apply in practice for someone who has some software engineering experience. For me personally, that was not the objective of attending this course, I wanted to obtain a global understanding of GCP. That content and experience exceeded my initial goals.
The Most Effective Format
For me it was very valuable that all attendees introduced themselves at the beginning of the training. This made it easier later on to work together and to ask questions. By sharing their individual learning goals beforehand, the trainer was able to personalize the learning experience for everyone. To me, this was a big plus compared to individual online training without a live instructor.

I must admit that a full day of online training is very intense. Especially at the end of day, it becomes more difficult to digest all the information. That is why we have chose to offer our online courses in half days by default. This way, it is easier to stay attentive, while it also makes it easier to combine training with other obligations.

Max Recommends

I am looking forward to navigating the world of cloud together with you. One of my goals is to regularly recommend specific training topics. As this Google Cloud Fundamental training was my first, let’s start by sharing who I would recommend this training for. I recommend this Google Cloud Platform Fundamentals training to everyone who would like to make a start with GCP. The training offers a broad introduction to the various GCP services and lays the groundwork for official Google Cloud certifications.

If you have any questions, feel free to get in touch with me directly by sending me an email via maxdriessen@binx.io

Two years ago, I created a utility to copy AWS SSM parameters from one account to another. I published the utility to pypi.org, without writing a blog about it. As I found out that quite a number of people are using the utility, I decided to unveil it in this blog.

(more…)

Building Serverless Applications with Google Cloud Run

Wietse Venema (software engineer and trainer at Binx.io) published an O’Reilly book about Google Cloud Run: the fastest growing compute platform on Google Cloud. The book is ranked as the #1 new release in Software Development on Amazon.com! We’re super proud. Here’s what you need to know about the book:

Praise

I’ve been fortunate enough to be a part of the Google team that helped create Knative and bring Cloud Run to market. I’ve watched Cloud Run mature as a product over the years. I’ve onboarded thousands of customers and I wrote a framework to help Go developers build Cloud Run applications faster–and even I learned a thing or two from this book. What took me three years to learn, Wietse delivers in less than a dozen chapters.
— Kelsey Hightower, Principal Engineer at Google Cloud

About the Book

If you have experience building web applications on traditional infrastructure, this hands-on guide shows you how to get started with Cloud Run, a container-based serverless product on Google Cloud. Through the course of this book, you’ll learn how to deploy several example applications that highlight different parts of the serverless stack on Google Cloud. Combining practical examples with fundamentals, this book will appeal to developers who are early in their learning journey as well as experienced practitioners (learn what others say about the book).

Who this Book is For

If you build, maintain or deploy web applications, this book is for you. You might go by the title of a software engineer, a developer, system administrator, solution architect, or a cloud engineer. Wietse carefully balances hands-on demonstrations with deep dives into the fundamentals so that you’ll get value out of it whether you’re an aspiring, junior, or experienced developer.

What’s Next

You can do either one or of all of the following things:

How to Automate the Kritis Signer on Google Cloud

Google Binary Authorization allows you to control the images allowed on your Kubernetes cluster. You name the images or allow only images required to be signed off by trusted parties. Google documents a manual process of creating Kritis signer attestions. In this blog, I show you how to automate the Kritis signer, using Terraform and Google Cloud Run.

Cloud-based IT infrastructures and software solutions are not only disrupting IT, they disrupt the way organizations, and even industries, operate. The full extent of the cloud’s scope of influence stretches far beyond the traditional IT landscape. To name a few, cloud impacts IT management and the ways teams work together. Last but not least, it requires a new skill set for individuals to operate in this sphere. These are just a few areas where the cloud has had significant impact. The cloud can be seen as a framework. In this article, we explain how cloud technology—if it’s applied with the right mind set—helps teams accelerate development.

The Cloud Framework

The cloud computing model has destroyed the boundary between the world of hardware and the world of software: the data center has become software too. The cloud provider offers you a programming framework as a foundation on which you can build anything. You program a datacenter with all of the components: computers, networks, disks, load balancers. When you start it up, a virtual instance of your data center is started. With this model it becomes very easy to change your data center. Just change the code and restart the program!

What Is Cloud-Native

So, what does it mean to be cloud-native? Do you have to use all of the cloud provider’s specific tools and resources? No, not at all. Cloud-native means you build and run applications that exploit the cloud benefits—such as scalability, flexibility, high availability and ease of maintenance. You focus on how to create these applications and how to run them: not on where they run.

Five Advantages of Cloud-Native

Organizations that adopt the Cloud-native style of software development and delivery have significant advantages over the traditional way of software delivery. If you’re responsible for IT projects, these five cloud-native advantages will sound like music to your ears:

1. High speed of Delivery

The cloud offers loads of off-the-shelf features so engineers can focus on developing unique ones—in other words, innovation. Leveraging these features, coupled with an ability to test every change with ease, allows development teams to deliver functional software faster.

2. Predictable Processes

As everything is programmed and automated, the software delivery process becomes very predictable.

3. Strong Reliability

Cloud providers offer robust services with consistent performance and multi-regional availability, which reduces the number of potential errors

4. Pay for what you use

Instead of investing in hardware for your own data center, you only pay for what you use in the cloud. However, don’t forget to shut off the service when you don’t need it!

5. Disaster Recovery

In the cloud, everything is code. So, in the unlikely event of a disaster, it is super simple to reboot the infrastructure and applications. Just add the data, and you’re operational again.

The Impact of Cloud-Native on Teams

A cloud-native infrastructure offers developers a stable, flexible environment with high availability — a place where engineers can release new features faster and easier than ever before. Cloud platforms provide automated services to manage infrastructure and increase its availability and reliability. All of this requires different skills from the people who are responsible for the infrastructure and applications. So, how will cloud technology impact your organizational culture?
Say Goodbye to Ops As You Know It
System administrators have traditionally been responsible for installing, supporting, and maintaining servers and systems, as well as developing management scripts and troubleshooting environments. In the cloud, this is replaced by managed services, immutable infrastructure and self-healing architectures. In the cloud everything is automated and coded: from the applications, monitoring, the infrastructure and delivery processes.

Bring True Developer Skills

Cloud-native requires people who can design systems, write code, automated deployment processes and automate the monitoring of systems.

They also need to know how to adopt the best way of working. In the cloud, DevOps has become the standard framework for the software delivery life cycle. This framework oversees the entire cycle of planning, developing, using, and managing applications. Since the cloud removes most of the “Ops” work, it’s essential for organizations to amp up their internal development skills.

Embrace Changes to Your Environment

In the classical IT world, changes to an existing system was to be avoided as much as possible, as it might possibly break things. In the cloud-native world, you embraces changes to your environment, as they may expose errors in your design or application that you need to fix, so that the error will not occur when the change is released to production.

Follow the Cloud-Native Roadmap

To exploit its advantages, it’s helpful for developers to follow the cloud-native roadmap. This roadmap consists of six steps to build and run cloud-native environments.

An Example of a Cloud-Native Set-Up

Vereniging Coin - Binx Customer
COIN is an association founded by telecom providers taking care of the transition of customers between telecom providers. To reduce management costs and increase development speed, COIN turned to Binx. As a cloud-partner, Binx could help them take full control of their software development, by migrating their infrastructure and applications to the cloud.

Initially, the development team, led by Binx, containerized all applications. This made it possible to release and deploy a new version of features as soon as there was a commit—a dream come true for the configuration manager. Everything deployed was visible, including all the resources associated with it.

The team decided to deploy Amazon RDS for Oracle, a high-available Oracle service, since they could not easily migrate the existing Oracle database to an open-source relational database management system.

The organization was provided with the self-service reporting tool (Amazon QuickSight) to access the data. This allowed end-users to create their own reports, and the development team to stay focused on developing features instead.

Because all applications were containerized, the application deployment process was automated and standardized which improved both reliability and speed of deployment.

COIN adopted a service called CloudFormation to code the infrastructure, to make the environment 100% reproducible. Binx developed a large number of custom resources for features like automated password generation and infrastructure setup. Managed services automatically deploy via a SAAS-based Git service, so there’s no in-house installation at COIN.

Last but not least, Binx implemented business service level checks and monitoring to ensure that the team is alerted of any disturbance in the service offered to their end users. These Service-level indicators (SLI) measure how well the services are performing and objectives determine what acceptable levels of performance are. These SLIs are also used on a daily basis to improve the system. Event for small aberrations, the team executes root cause analysis to see if the problem can be designed out of the system.

The service level objectives are continuously monitored and every breach automatically alerts the team and the on-call engineer.

Becoming Cloud-Native: Create Your Starting Point

Now that you can see the cloud as a framework, its benefits over more traditional approaches should be clear. The next step is to adopt cloud-native ways of working. Can your organization view infrastructure as code and migrate workloads to the cloud? What skills are lacking? Can your workforce adopt a cloud-native mindset? By answering these questions, you create a starting point for your journey towards becoming cloud-native. Safe travels, we look forward to seeing you out there!

Urgent Future: Digital Culture

This article was featured in Urgent Future: Digital Culture. A trend report featuring various view points on the culture that makes digital organizations thrive in today’s economy. Download the trend report here

Before I can use a particular service, I have to enable the API in my Google project. Sometimes when I do this, more services are enabled than the one I specified. In this blog I will show you how to find service dependencies like this.

To see this in effect, I am going to enable the Cloud Functions service. First I will show you that enabling Cloud Functions, will actually enable six services in total. Then I will show you how you can list the dependencies. Finally, I will present you with a small utility and graph with all dependencies for you to play around with.

enabling Cloud Functions

So I am going to show you that enabling Cloud Functions is actually enabling multiple services. First, I am going to check the enabled services in my project:

gcloud services list

The output looks roughly like this:

NAME                              TITLE
bigquery.googleapis.com           BigQuery API
...
servicemanagement.googleapis.com  Service Management API
serviceusage.googleapis.com       Service Usage API

In this case, the Cloud Functions service is not listed.

enabling the serivce

To enable Cloud Functions in my project, I type:

$ gcloud services enable cloudfunctions
Operation "operations/acf.3170fc7d-dc07-476f-851b-ff0cc2b9d79f" finished successfully.

Now, when I check the list of enabled services again, the number of active services has increased with the following six services!

NAME                              TITLE
cloudfunctions.googleapis.com     Cloud Functions API
logging.googleapis.com            Cloud Logging API
pubsub.googleapis.com             Cloud Pub/Sub API
source.googleapis.com             Legacy Cloud Source Repositories API
storage-api.googleapis.com        Google Cloud Storage JSON API
storage-component.googleapis.com  Cloud Storage
...

Could I have predicted this before i enabled it? Yes, I could have..

listing all available services

To list all the available services, I use the following command:

gcloud services list --available --format json

The result is a list of objects with meta information about the service:

{ 
  "config": {
    "name": "cloudfunctions.googleapis.com",
    "title": "Cloud Functions API",
    "documentation": {},
    "features": [],
    "monitoredResources": [
    "monitoring": {},
    "quota": {},
    "authentication": {},
    "usage": {}
  },
  "dependencyConfig": {
    "dependsOn": [],
    "directlyDependsOn": []
    "directlyRequiredBy": [],
    "requiredBy": []
  },
  "serviceAccounts": [],
  "state": "DISABLED"
}

I see four main attributes for each service: config, dependencyConfig, serviceAccounts and state. The fields dependencyConfig lists the service dependencies, while serviceAccounts lists the service accounts which are created in the project for this service. Note that these fields are not part of the documented service usage API.

listing specific dependencies

So, this service usage API provides all the dependencies of a specific service. To list all dependent services of Cloud Functions, I use the following command:

gcloud services list \
   --available --format json | \
jq --arg service cloudfunctions.googleapis.com \
    'map(select(.config.name == $service)| 
        { 
          name:      .config.name, 
          dependsOn: .dependencyConfig.dependsOn
        }
    )'

and the result is:

{
  "name": "cloudfunctions.googleapis.com",
  "dependsOn": [
    "cloudfunctions.googleapis.com",
    "logging.googleapis.com",
    "pubsub.googleapis.com",
    "source.googleapis.com",
    "storage-api.googleapis.com",
    "storage-component.googleapis.com"
  ]
}

These are precisely the six services that were previously enabled \o/.

If you want to explore dependencies yourself, you can use this bash script. If you do not want to type, you can browse through the entire graph
google cloud platform services dependencies.

I know it is a bit tiny, but luckily it is a scalable vector graphic. So open it in a separate window and you can pan and zoom.

conclusion

Thanks to some undocumented properties of by the Google service usage API, I can find all dependencies between Google Cloud Platform services.

For a number of Google Cloud platform services I need to perform a Google site verification in order to proof that I actually own a domain. Unfortunately, the Google Terraform provider does not provide support for this. In this blog I will show you how to automate this using a custom terraform provider.

(more…)

The AWS Cloud Development Kit (AWS CDK) makes it easy to build cloud infrastructure. And CDK Pipelines makes it "painless" to deploy cloud infrastructure. So let’s create an AWS CDK CI/CD pipeline, and build and run our application on AWS.

(more…)

Recently I worked on a PHP application that used cron for background processing. Since it took some time to get it right, I’m sharing the solution.

(more…)

Many applications log to files. In a container environment this doesn’t work well. The file system is not persisted and logs are lost. In GKE you can persist log files to Cloud Logging with the Cloud Logging agent. The agent, however, doesn’t specify the required resource type. Causing the logs to appear as non-Kubernetes Container logs. This blog shows how to resolve that.

Cloud Logging Agent in GKE

The Cloud Logging agent is a Fluentd service configured with the Google Cloud output plugin. Events sent to the output plugin must include a Cloud Operations for GKE resource types. Since without any resource type, the event will be registered as a VM instance log record in Cloud Logging.

Set the Kubernetes Container resource type

Kubernetes Container events use resource type k8s_container.[namespace].[pod].[container] and are specified by the attribute logging.googleapis.com/local_resource_id using a Fluentd filter.

<filter **>
  @type record_transformer
  enable_ruby true
  <record>
    "logging.googleapis.com/local_resource_id" ${"k8s_container.#{ENV['K8S_NAMESPACE']}.#{ENV['K8S_POD']}.#{ENV['K8S_CONTAINER']}"}
  </record>
</filter>

The entire configuration file is available on GitHub.

The filter uses Kubernetes’ Downward API to read resource type data from environment variables.

    spec:
      containers:
        - name: application
        - name: logging-agent
          env:
          - name: K8S_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: K8S_POD
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: K8S_CONTAINER
            value: application

Deploy the GKE logging agent

The logging agent uses a sidecar container deployment to access application log files.

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: app
          volumeMounts:
            - name: logs
              mountPath: /app/log
        - name: gke-fluentd
          env:
          - name: K8S_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: K8S_POD
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: K8S_CONTAINER
            value: app # Use the application container name
          volumeMounts:
            - name: logs
              mountPath: /app/log
      volumes:
        - name: logs
          emptyDir: {}

Try it yourself with the example application provided at GitHub.

Discussion

The logging agent allows us to persist logs. However, we don’t actually solve the problem of using log files in a container-environment..

In container-environments you should use cloud-native loggers, or stream all logs to the console. Changing the logger should be a dependency configuration-file update; and changing the logger behavior should be a logger configuration-file update.

Conclusion

The Cloud Logging agent allows you to keep using log files in container environments. Since the default configuration doesn’t specify the appropriate GKE resource types, you will have to maintain the agent. Therefore I recommend to fix your application logging. Allow configuration of a cloud-native logger and use structured logging to opt-in to all major log analysis services.

Image by Free-Photos from Pixabay

When you want to configure a SAML identity provider to enable SSO for AWS, you will find that CloudFormation does not provide support for this. In this blog we will present you with a custom provider which will allow you to configure the SAML identity provider in just a few lines!

How to use

To add a SAML identity provider using your AWS CloudFormation template, use a Custom::SAMLProvider resource with reference
to the metadata URL:

  SAMLProvider:
    Type: Custom::SAMLProvider
    Properties:
      Name: auth0
      URL: https://auth0.com/mytenant/providerurl
      ServiceToken: !Sub 'arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:cfn-saml-provider'

When the provider is created, it will load the metadata of the identity provider from the specified URL. If you want a static approach, you can also specify the metadata itself:

  SAMLProvider:
    Type: Custom::SAMLProvider
    Properties:
      Name: auth0
      Metadata: |
        <EntityDescriptor entityID="urn:binxio.auth0.com" xmlns="urn:oasis:names:tc:SAML:2.0:metadata">
                ....
        </EntityDescriptor>
      ServiceToken: !Sub 'arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:cfn-saml-provider'

On completion, it will return the ARN of the SAML Provider. That is all there is to it. From there on, you can configure IAM roles based upon the established identities in this account.

Deploy the custom provider

To deploy the provider, type:

aws cloudformation deploy  \
        --capabilities CAPABILITY_IAM \
        --stack-name cfn-saml-provider \
        --template-file ./cloudformation/cfn-saml-provider.json

This CloudFormation template will use our pre-packaged provider from s3://binxio-public-${AWS_REGION}/lambdas/cfn-saml-provider-latest.zip.

Demo

To install the simple sample of the SAML provider, type:

aws cloudformation deploy --stack-name cfn-saml-provider-demo \
        --template-file ./cloudformation/demo-stack.json

to validate the result, type:

aws iam list-saml-providers

conclusion

With just a few lines of code you can configure the SAML provider required to implement SSO for your AWS accounts, infrastructure as code style. And that is the only way you want it, right?

You may also like How to get AWS credentials and access keys using the Auth0 SAML identity provider and How to limit access to AWS Resources based on SAML Attributes using CloudFormation.

Image by jacqueline macou from Pixabay

Restricting access to IAM resources based on SAML Subject

Many larger organizations manage their own Active Directory servers. For AWS access, they typically create an identity provider to provide a single sign on (SSO) experience for logging onto AWS. The user is then often granted access to a particular role that grants particular rights. This approach, however, lacks practical fine-grained control. Managing these rights effectively means creating and maintaining many IAM roles and AD groups.

An alternative approach, outlined in this blog post, is to utilize the SAML assertions in your IAM policies. If you enable ABAC (which requires the IAM::TagSession privilege on the role) you will be able to differentiate based on custom SAML attributes. Without ABAC you can still use a couple of attributes such as the SAML subject. In this post we will first focus on the latter: How to use the SAML subject to restrict access to IAM Roles.

The CloudFormation template below creates an IAM Role. Its only permissions are listing buckets, listing a particular S3 bucket and full S3 access on a particular subfolder. Finally, we limit the access to the subfolder based on the list of SAML subjects provided as a parameter to this stack.

In the AssumeRolePolicyDocument (Trust policy) we define the principal as our SAML provider. That means that anyone that has federated access to AWS can assume this role:

        Statement:
          - Action:
              - sts:AssumeRoleWithSAML
              - sts:TagSession
            Condition:
              StringEquals:
                SAML:aud: https://signin.aws.amazon.com/saml
            Effect: Allow
            Principal:
              Federated: !Ref 'SamlProviderArn'

To limit access to a supplied comma-separated list of SAML Subjects we use the following Condition in the S3 IAM Policy:

              Condition:
                ForAllValues:StringLike:
                  saml:sub: !Split
                    - ','
                    - !Ref 'SamlSubjects'

This condition basically says: Only allow this if your SAML subject is one of those in the list provided. Note that you could also limit access to the role by moving the condition section to the Trust Policy.

The template below takes 3 parameters: The bucket name you wish to secure, the ARN of the SAML provider, and finally the comma-separated list of SAML subjects.

Complete template:

Description: Bucket Restriction Stack
Parameters:
  BucketName:
    Description: Bucket name to grant access to
    Type: String
  SamlProviderArn:
    Description: ARN of SAML provider
    Type: String
  SamlSubjects:
    Description: List of Strings
    Type: String
Resources:
  SamlBucketRole:
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action:
              - sts:AssumeRoleWithSAML
              - sts:TagSession
            Condition:
              StringEquals:
                SAML:aud: https://signin.aws.amazon.com/saml
            Effect: Allow
            Principal:
              Federated: !Ref 'SamlProviderArn'
        Version: '2012-10-17'
      Path: /
      Policies:
        - Statement:
            - Action:
                - s3:ListAllMyBuckets
                - s3:GetBucketLocation
              Effect: Allow
              Resource: '*'
            - Action:
                - s3:ListBucket
              Effect: Allow
              Resource:
                - !Sub 'arn:aws:s3:::${BucketName}'
            - Action:
                - s3:*
              Condition:
                ForAllValues:StringLike:
                  saml:sub: !Split
                    - ','
                    - !Ref 'SamlSubjects'
              Effect: Allow
              Resource:
                - !Sub 'arn:aws:s3:::${BucketName}'
          Version: '2012-10-17'
    Type: AWS::IAM::Role

Now you may not want to maintain a list of SAML subjects. Imagine, for example, you have an S3 Bucket you want to secure and it has a list of subfolders for each user, with the folder name being identical to the SAML subject. A policy for this access policy could look something like this:

        {
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::some-s3-bucket"
            ],
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "",
                        "${saml:sub}/*"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::some-s3-bucket/${saml:sub}/*"
            ]
        }

These parameterized way of creating IAM policies for SAML subjects scales really well, but you’ll lose some flexibility in terms of what you can provide access too.

Lastly, you may want to be able to restrict access based on custom SAML attributes. AWS’ default SAML attributes that can be used is very limited. By enabling Session Tags you can utilize custom SAML attributes and differentiate on them in your IAM policies.

Here is an example:

        {
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::my-s3-bucket"
            ],
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "",
                        "${aws:PrincipalTag/my-custom-attribute}/*"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::my-s3-bucket/${aws:PrincipalTag/my-custom-attribute}/*"
            ]
        }

Similarly to the first example, we can also maintain a list of explicit conditions using the custom SAML attribute if we use ABAC.

Using Infrastructure as Code to maintain and deploy these roles and policies reduces operational overhead and prevents human errors, however: there are hard limits on the size of policies and the amount of conditions one can use. It does not scale very well out of the box.

Conclusion

Federation requires you to think hard about your data access patterns. Using SAML attributes in your IAM policies is a powerful method of automating access. In order for this to scale for large organizations you must take great care of your deployment strategy of these policies. One approach could be to use several (nested) stacks and individual roles to work around the policy size limitations.

You may also like How to get AWS credentials and access keys using the Auth0 SAML identity provider and How to configure a SAML identity provider to enable SSO in AWS CloudFormation.

Organizations have an increasing need for total control of their cloud consumption. In many cases, multiple cloud accounts combined with unspecified invoices have led to the nearly impossible challenge to get complete insight in the organization-wide cloud consumption. While new features are released, organizations often don’t take the time to optimize their cloud workloads. With Cloud Control, full-service cloud consultancy boutique Binx offers organizations a complete solution to take control of their cloud consumption.

With Cloud Control, full-service cloud consultancy boutique Binx offers organizations a complete solution to take control of their cloud consumption.

“It is our ambition to make organizations cloud native. With Cloud Control we provide organizations with cloud environments that perform as optimal as the entire administrative process around organisation-wide cloud consumption”, said Bart Verlaat, CEO at Binx.io.

First of all, Binx Cloud Control accumulates all cloud spend in one clear invoice. Combined with real-time dashboards, organizations get access to unprecedented insights in their cloud consumption, even across multiple cloud platforms. Cloud Control also regularly reviews the cloud environment on potential optimization and provides support to prevent outages and underperformance.

With this complete service offering organization have immediate access to:

  • single view of organization-wide cloud consumption;
  • one aggregated invoice with a clear overview of all cloud – consumption, across cloud platforms;
  • regular reviews of cloud workloads;
  • 24/7 monitoring and support.

Cloud Billing

For many organizations, keeping track of cloud cost is a tedious task. With multiple departments and teams who use their own credit cards to consume cloud services, it is easy for the IT-department to lose control of the cloud consumption.

With Cloud Billing, organizations have the complete picture of the cloud consumption across the entire organization, even when this involves multiple cloud platforms. Clients receive specified invoices of all the cloud services used within the organization. On top of this, clients receive access to an interactive dashboard with real-time usages stats.

The Cloud Control team is an extension of your organization. This highly-specialized team is only a phone call or message away and has only one objective: to create the most effective cloud environment for you.

But the biggest bonus for clients is that all of these benefits come without any additional costs compared to a direct invoicing relationship with a cloud provider.

“Besides insights in the cloud consumption and the associated costs, we also actively report on potential points of improvement. Without spending an extra euro, organizations gain total control on their cloud investment”, explains Verlaat.

Cloud Control also offers solutions for organizations that are used to more traditional IT procurement (CapEx).

“Based on the expected cloud consumption, organizations can purchase bundles of cloud credits upfront instead of receiving invoices after every period”, said Verlaat.

The interactive dashboards of Cloud Billing provide real-time insight in the cloud consumption. The Finance department will appreciate the monthly aggregated invoice they can process seamlessly.

Cloud Reviews

As most cloud environments are not isolated, it is extremely difficult for cloud architects and cloud engineers to create and maintain environments that are completely optimized. In practice, added workloads and processes can lead to a more inefficient performance of an environment.

Binx Cloud Control offers regular reviews of workloads and infrastructure to evaluate architectures, and implement designs that will scale over time.

“A cloud environment review gives organizations an immediate insight in points of optimization”, emphasized Dennis Vink, CTO of Binx Cloud Control.

Cloud Reviews lead to cloud environments that are optimized to deliver minimum cost, maximum security, the most efficient performance, the highest reliability and operational excellence.

Cloud Support

For business-critical processes, downtime or underperformance are unacceptable. Cloud-native practices like extreme automation, standardization and event-driven architectures contribute to the eradication of performance issues.

Binx Cloud Control supports organizations to automate the operations of their cloud environment. Clients have direct access to a dedicated senior cloud consultant. This specialized engineer is familiar with the client’s specific situation and offers pragmatic support per phone, e-mail, Slack, or on-premises.

“Our support is both reactive and proactive”, Vink explains. "As soon as performance hits a predefined threshold, Binx Cloud Control receives notification. The senior cloud consultant immediately takes appropriate action to take away the root cause.”

With Cloud Support, organizations have the peace of mind of cloud environments with optimal performance.

For more information, please get in touch with Bart Verlaat

As of June 2020, you can enable the Firestore key-value database with Terraform. Two things are somewhat confusing:

  • You need to create an App Engine app to enable the Firestore database
  • You need to use Datastore IAM roles.

This is what you need to do:

Use the Google Beta Provider

Start with using the google-beta provider (it might be in main by the time you read this).

provider google-beta {
  project = var.project_id
  version = "~> 3.0"
}

Create an App Engine App

In order to use Firestore, you first need to create an App Engine app. As I understand, there is work underway to remove this limitation, but this is how it is right now. Here’s what you need to know:

  • You can only enable App Engine once per project.
  • The region (location_id) choice is permanent per project – and can not be undone.
  • You will not be charged for enabling App Engine if you don’t use it.
variable "location_id" {
  type        = string
  description = "The default App Engine region. For instance 'europe-west'"
}

# Use firestore
resource google_app_engine_application "app" {
  provider      = google-beta
  location_id   = var.location_id
  database_type = "CLOUD_FIRESTORE"
}

Using Firestore From Your Application

Enable the firestore API, to make sure your applications can connect using the Firestore client libraries.

resource google_project_service "firestore" {
  service = "firestore.googleapis.com"
  disable_dependent_services = true
}

If you are not using default service accounts (or disable the default grants) – you will need to provide the Datastore User role. Yes, that’s datastore, not firestore.

resource google_project_iam_member "firestore_user" {
  role   = "roles/datastore.user"
  member = "serviceAccount:[YOUR SERVICE ACCOUNT]"
}

On Google Cloud Platform, we use the Google Secret Manager to keep our secrets safe. But accessing the secrets from an existing application is intrusive. You either have to call the API in the application or use the secrets cli in the entry point script of the container. In this blog, I introduce you to the utility gcp-get-secret.This utility changes references to secrets into environment variable values.

(more…)

Creating dynamic infrastructures with Terraform used to be a challenge. Start using the for_each-meta-argument to safely and predictably create your infrastructure while limiting code duplication.

This post gives you a real-world example of how to effectively use the for_each meta-argument of Terraform 0.12.

(more…)

Google Cloud Platform (GCP) uses service accounts to authorize applications or services. Azure Pipelines typically stores these credentials as Service Connection. However, a GCP Service connection is unavailable. Therefore we use Secure Files.

(more…)

When you want to create a command line utility for Google Cloud Platform, it would be awesome if you could authenticate using the active gcloud configuration. Unfortunately, none of the Google Cloud Client libraries support using the gcloud credentials. In this blog, I will present a small go library which you can use to do just that.

How to use it?

It is really simple. You import the package github.com/binxio/gcloudconfig and call the GetCredentials function, as shown below:

package main

import "github.com/binxio/gcloudconfig"

func main() {
    name := ""
    credentials, err := gcloudconfig.GetCredentials(name)
    ...
}

The name specifies the configuration you want to use, or the current active one if unspecified. The credentials can be passed in when you create a service client, as shown below:

    computeService, err := compute.NewService(ctx,
                                 option.WithCredentials(credentials))

If the core/project property has been set, it is available in the credential too:

    project := credentials.ProjectId

That is all there is to it! Check out the complete example of using the gcloud configured credentials. If you want to access other settings in the configuration use GetConfig.

How does it work?

The function will executes the command gcloud config config-helper, which is a gcloud helper for providing authentication and configuration data to external tools. It returns an access token, an id token, the name of the active configuration and all of the associated configuration properties:

configuration:
  active_configuration: playground
  properties:
    core:
      account: markvanholsteijn@binx.io
      project: playground
    ...
credential:
  access_token: ya12.YHYeGSG8flksArMeVRXsQB4HFQ8aodXiGdBgfEdznaVuAymcBGHS6pZSp7RqBMjSzHgET08BmH3TntQDOteVPIQWZNJmiXZDr1i99ELRqDxDAP8Jk1RFu1xew7XKeQTOTnm22AGDh28pUEHXVaXtRN8GZ4xHbOoxrTt7yBG3R7ff9ajGVYHYeGSG8flksArMeVRXsQB4HFQ8aodXiGdBgfEdznaVuAymcBGHS6pZSp7RqBMjSzHgET08BmH3TntQDOteVPIQWZNJmiXZDr1i99ELRqDxDAP8Jk1RFu1xew7XKeQTOTnm22AGDh28pUEHXVaXtRN8GZ4xHbOoxrTt7yBG3R7ff9ajGV
  id_token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2FjY291bnRzLmdvb2dsZS5jb20iLCJhenAiOiI5OTk5OTk5OTk5OS5hcHBzLmdvb2dsZXVzZXJjb250ZW50LmNvbSIsImF1ZCI6Ijk5OTk5OTk5OTk5LmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tIiwic3ViIjoiMTExMTExMTExMTEyMjIyMjIyMjIyMjIiLCJoZCI6InhlYmlhLmNvbSIsImVtYWlsIjoibWFya3ZhbmhvbHN0ZWlqbkBiaW54LmlvIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsImF0X2hhc2giOiJScnhBVHRSaTE2TFNOSG1JdnZEWVdnIiwiaWF0IjoxNTg4NzAxNjgzLCJleHAiOjE1ODg3MDUyODN9.DWtAHSvWgXaW0zzoLf2MkiROS_DSS2Wf-k_HQj53g3I
  token_expiry: '2020-05-05T19:01:22Z'

When the token is expired, the library will call the helper again to refresh it. Note that Google is unsure whether the config-helper is a good thing. If you read gcloud config config-helper --help, you will notice the following sentence:

This command is an internal implementation detail and may change or disappear without notice.

Although, for the development of command line utilities which integrate into the Google Cloud SDK ecosystem, it would be really handy if Google would provide an official way to obtain the active gcloud configuration and credentials.

conclusion

With the help of this library, it is possible to create a command line utility in Go for the Google Cloud Platform, that integrates into the gcloud ecosystem. It is unfortunate that the config-helper is documented to be a volatile interface. Given the simplicity of the interface, I trust this library will be able to deal with any future changes. It would be even better, if Google would provide official support.

We also created a library for authenticating with gcloud credentials in Python.

Image by Kerstin Riemer from Pixabay

For the third consecutive year, Binx, Xebia DevOps and Computable conduct “The Cloud Survey”. The objective of this survey is to establish insight in how organization use cloud technology and to point out the major trends.

Download the report

Which Platforms Are Leading?

The past surveys indicated that only a few organizations are not using any cloud technology at all. Microsoft Azure, Amazon Web Services and Google Cloud Platform are the most-adopted cloud platforms. Furthermore, The Cloud Survey showed that the development of skills is essential for organizations to realize actual cloud transformations.

What Are The Benefits of Cloud Technology?

Organizations stand united in their belief of the competitive advantage cloud technology brings them. The most important reasons for this conviction are the possibilities to operate more flexibly, to reduce cost, and to increase the innovation power.

What Are The Challenges of Cloud Technology?

Large organizations with more than 500 employees indicated that they find it difficult to properly apply the principles of cloud technology. Cloud transformations often fail because of resistance to change from within the organization and/or a lack of experienced IT-staff to set the right example.

What’s Your Cloud Experience?

Did organizations manage to realize cloud transformations in the past year? What are the success factors to get the most out of cloud technology? Has machine learning made the leap to mainstream adoption? These are just a few of the questions that Computable, Xebia DevOps and Binx.io would like to see answered with the Cloud Survey.

Download the report
Binx - The Cloud Survey 2020 Prizes

Win Great Prizes

Participants are eligble to win some great prizes, including a box of Lego “International Space Station”, or a copy of the books “DevOps voor Managers” or “Building Serverless Applications on Cloud Run”.

At times, working from home can be challenging. At the same time, it can also be very rewarding to have the opportunity to spend more time around the house. In this booklet, the Binx consultants share 13 tips (and 1 bonus tip) on how to stay sane while working from home.

Look out for some hidden gems in the pictures:

  • Funny mugs
  • Hidden hand creams
  • Photobombing’kids
  • Collectable stones
    and more.

We hope you enjoy this booklet.

The Cloud Survey 2020 – Share your experiences – receive the report

Thirteen Tips For Working at Home

Tip 1: Turn commute time into gym time

Turn commute time into home gym time. Suddenly, this makes traffic jams feel not so bad at all.

Tip 2: Put the music on

Put the music on. Loud. And maybe, just maybe, pull off a little
dance.

Tip 3: Adjust the seat and settings

Make sure your chair provides the right support and your monitor is at the right height.

Tip 4: Work from the couch

It is no crime to work from the couch every now and then.

Tip 5:Be rigorous with your schedule

Let go of the 9-to-5 but be rigorous with your schedule; your monkey brain will thank you for it.

Tip 6: Be less efficient with the coffee run

Be less efficient. For most, home is smaller than the office. Grab only one drink on your coffee run.

Tip 7: Smoke a cigar behind your desk

Coffee is so pre-WFH. Smoke a cigar. Behind your desk. Go on, indulge yourself.

Tip 8: Get into the state of flow

Use flow to increase your effectiveness. Lose all distractions and go deep on your backlog!

Tip 9: Go out and play

Cherish the extra family time. Once the labor is out of the way, go all out and play.

Tip 10: Make lunch work for you

Make lunch work for you! Boil 2 eggs, add Udon noodles and cooked veggies. Top off with some nori sushi. Yum!

Tip 11: Eat lunch outside

Keep your house tidy. Eat lunch outside.

Tip 12: Reward yourself

Work hard, eat healthy. On top of this, take a moment each day to reward yourself for accomplishments.

Tip 13: Reset the brain

Take regular breaks away from the screen. A little car ride can help to reset the brain.

Bonus tip: Laugh more often

Humor heals. Instead of using SnapCam, dress up for real in a meeting.

Google Cloud CI/CD pipelines require Google Cloud SDK. Azure Pipelines however doesn’t provide a Tool Installer-task for it. Therefore we created it.

(more…)

With Google’s Cloud Run it has become very easy to deploy your container to the cloud and get back a public HTTPS endpoint. But have you ever wondered about how the Cloud Run environment looks from the inside?
In this blog post we will show you how to login to a running Google Cloud Run container.

(more…)

In automation environments such as Azure DevOps you can’t use Terraforms interactive approval. Therefore you create a deployment plan, wait for a plan approval and apply the deployment plan. This blog implements a plan approval in Azure Pipelines using the Manual Intervention-task.

(more…)

HashiCorp Packer is a great tool for building virtual machine images for a variety of platforms including Google Cloud. Normally Packer starts an GCE instance, builds the machine image on it and terminates the instance on completion. However sometimes the process is aborted and the instance is left running, racking up useless cloud spend. In this blog I present a utility to get delete lingering Packer instances.

(more…)

This event will be hosted by Amazon Web Services and Binx.io and takes place on April 24th. We’ll cover the second pillar of the Well-Architected Framework. This pillar is Performance Efficiency.

During this event, you can expect to hear from a senior solution architect with Amazon Web Services and a senior solution architect with Binx.io and they will go over performance efficiency in your cloud environment. Again, this event takes place on April 24th.We hope to see you there.

Click here to register

How do you update an EC2 instance with volume attachments using CloudFormation? When you have a stateful server with one more volumes attached to it in your infrastructure, the AWS::EC2::VolumeAttachment resource makes it impossible to update the instance. In this blog I will show you how to configure volume attachments that allow the instance to be updated using an auto scaling group and the EC2 Volume manager.

When you try to update an AWS::EC2::VolumeAttachment, CloudFormation will give the following error:

ERROR: Update to resource type AWS::EC2::VolumeAttachment is not supported.

This prevents you from updating the AMI, or any other property that requires a replacement of the instance.

I solved this problem by changing the resource definition of the stateful machine to a single instance auto scaling group and use the EC2 volume manager utility to dynamically attache volumes upon the start of instances.

To implement this, you have to:

  1. deploy the ec2 volume manager
  2. change from an instance to an auto scaling group
  3. enable rolling updates on the auto scaling group
  4. signal successful startup to CloudFormation
  5. attach ec2 volume manager tags to volumes and instances

Deploy the EC2 volume manager

Deploy the ec2-volume-manager using the following commands:

git clone https://github.com/binxio/ec2-volume-manager.git
cd ec2-ec2-volume-manager
aws cloudformation deploy \
        --capabilities CAPABILITY_IAM \
        --stack-name ec2-volume-manager \
        --template ./cloudformation/ec2-volume-manager.yaml

Change instance to auto scaling group

Change the definition of your persistence instance, from a AWS::EC2::Instance to an single instance auto scaling group. From:

StatefulServer:
  Type: AWS::EC2::Instance
  Properties:
    SubnetId: !Select [0, !Ref 'Subnets']
    LaunchTemplate:
      LaunchTemplateId: !Ref 'LaunchTemplate'
      Version: !GetAtt 'LaunchTemplate.LatestVersionNumber'

to:

AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      AutoScalingGroupName: !Ref AWS::StackName
      VPCZoneIdentifier:
        - !Select [0, !Ref 'Subnets']
      LaunchTemplate:
        LaunchTemplateId: !Ref 'LaunchTemplate'
        Version: !GetAtt 'LaunchTemplate.LatestVersionNumber'
      MinSize: '0'
      MaxSize: '1'
      DesiredCapacity: '1'

Enable rolling update

Instruct CloudFormation to perform a rolling update to replace the instances:

AutoScalingGroup:
  Type: AWS::AutoScaling::AutoScalingGroup
  Properties:
    ...
    UpdatePolicy:
      AutoScalingRollingUpdate:
        MinInstancesInService: 0
        MaxBatchSize: 1
        WaitOnResourceSignals: true

When the instance needs to be replaced, CloudFormation will update the auto scaling group by destroying the old instance first, followed by the creation of a new instance.

Signal successful startup

CloudFormation will wait until the instance reports it has succesfully started. This is done by the cfn-signal at the end of the boot commands in the launch template.

LaunchTemplate:
  Type: AWS::EC2::LaunchTemplate
  Properties:
    ...
    UserData: !Base64
      Fn::Sub: |
        bootcmd:
          ...
          -  /opt/aws/bin/cfn-signal --stack ${AWS::StackName} --resource AutoScalingGroup

Without this signal, the update is rolled back.

Attach tags on volumes and instance

The EC2 volume manager utility automatically attaches volumes to instances with the same tag values. When an instance with the tag ec2-volume-manager-attachments reaches the state running, it will attach all volumes with the same tag value. When the instance is stopped or terminated, all volumes with a tag ec2-volume-manager-attachments will be detached from it.

To get the volume manager to work, tag the volumes of the instance as follows:

  Disk1:
    Type: AWS::EC2::Volume
    Properties:
      AvailabilityZone: !Sub '${AWS::Region}a'
      Size: 8
      Tags:
        - Key: ec2-volume-manager-attachment
          Value: stateful-instance-1
        - Key: device-name
          Value: xvdf

Note that the volume manager also requires the tag device-name, referring to the device name of the volume for the operating system. Next, add the ec2-volume-manager-attachments to the auto scaling group:

AutoScalingGroup:
  Type: AWS::AutoScaling::AutoScalingGroup
  Properties:
    ...
    Tags:
      - Key: ec2-volume-manager-attachment
        Value: stateful-instance-1
        PropagateAtLaunch: true

That is all that is required to enable fully automated updates of a stateful server with attached volumes.

You can see all differences when you compare the CloudFormation template of the stateful server with the template for the ec2-volume-manager solution.

Deploy the demo

A demo is available. Deploy it with:

export VPC_ID=$(aws ec2  
                --output text \
                --query 'Vpcs[?IsDefault].VpcId' describe-vpcs)
export SUBNET_IDS=$(aws ec2 describe-subnets --output text \
                --filters Name=vpc-id,Values=$(VPC_ID) \
                          Name=default-for-az,Values=true \
                --query 'join(`,`,sort_by(Subnets[?MapPublicIpOnLaunch], &AvailabilityZone)[*].SubnetId)')

aws cloudformation deploy \
        --capabilities CAPABILITY_NAMED_IAM \
        --stack-name ec2-volume-manager-demo \
        --template ./cloudformation/demo-stack.yaml \
        --parameter-overrides VPC=$VPC_ID Subnets=$SUBNET_IDS

Conclusion

When you have a stateful server with one more volumes attached to it in your infrastructure, the AWS::EC2::VolumeAttachment resource makes it impossible to update the instance. But, when you use a single instance auto scaling group in combination with the EC2 Volume manager you can!

If you want to want you can also attach a static ip address to your stateful instance. Make sure to properly mount ebs volumes during boot.

Alternative solutions

Although, we always recommend to keep your EC2 instances stateless and use managed persistent services whenever possible, we have successfully used the EC2 volume manager for IBM MQ and Microsoft SQL Server instances.

If you do not like the magic of the EC2 volume manager, you can also attach the volumes in the boot script of the persistent instance.

HashiCorp Packer is a great tool for building virtual machine images for a variety of platforms including AWS. Normally Packer starts an EC2 instance, builds the AMI on it and terminates the instance on completion. However sometimes the process is aborted and the instance is left running, racking up useless cloud spend. In this blog I present a utility to get rid of old Packer instances.

(more…)

Last month I began on a new assignment. While waiting to get access to the customer’s Cloud environment, I started the work in my own AWS playground account. I made good progress and after a few days I could switch over. As I forgot to destroy the infrastructure, I ended up with a $1700 bill. To avoid this from happening every again, I automatically destroy all resources in my AWS account every day at 23:00 UTC.

(more…)

Most cloud environments are not optimized for cost. Despite the best intentions, it is extremely difficult for cloud architects and cloud engineers to create and maintain cloud environments that are completely optimized for the lowest possible spend and the least operational overhead.

Often, cost reductions can be accomplished by optimizing elements such as:

  1. Traditional Applications and Infrastructures
  2. Unknown Values Beforehand
  3. Unexpected Peak Loads
  4. Inefficient Architecture
  5. Operational Overhead

Five Ways to Minimize Your Cloud Spend

In this article, we discuss five actions cloud engineers can take to reduce the cost of their cloud environment.

Interested in Exploring Potential Cost Reduction?

To support organizations in the current challenging times, I am offering a free intake to explore potential cost savings. Get in touch for a free intake to discuss cost optimization >>

Cost Reduction 1. Refactoring Traditional Applications and Infrastructures

Traditional applications and infrastructures usually are not designed by the latest cloud principles and best practices. This makes them bulky, heavy, and inefficient.

Contemporary cloud infrastructures and applications scale when demand increases. This can be realized by designing processes that are horizontally scalable, are as compartmentalized as possible, and are preferably serverless so that you do not pay for idle but for actual use only. This means breaking up monoliths into a microservices architecture, statelessness of your application landscape, and minimizing the blast radius of any component to improve resiliency.

If not, a workload is likely to be inefficient, a tax on operations, and not cost-effective.

By refactoring these workloads, costs can be reduced significantly.

Cost Reduction 2. Filling in the Unknown Values

When architecting a cloud environment, it is usually difficult to calculate the actual operational cost of the workloads. Cloud services are invoiced on a pay per use basis. Despite calculation tools as the AWS Simple Monthly Calculator or the Google Cloud Platform Pricing Calculator, certain values are simply not known upfront.

For this reason, most companies start building their platform first. Based on the actual load or traffic, organizations experience the price associated with the process. Still, the return on investment (ROI) can differ per service for each use case of the platform.

As soon as a workload is live, it is wise to check the actual consumption and cost. A cloud expert usually is able to point out the room for optimization.

Cost Reduction 3. Engineering for the Right Load

In development, engineers need to make certain assumptions of the expected load on a service. In practice, the load on a workload or service can be much higher than originally anticipated. Services that are designed to be spun up only occasionally are not always well-designed to run continuously or to hit certain peak loads.

In addition, traditional architecture is not very well suited to deal with a sudden surge in traffic. In practice this means companies either provision resources for anticipated peak capacity, or run underutilized. This often leads to an exponential increase in cost, or a damage to reputation: downtime.

Cost reduction and being ready for peak loads do go hand in hand, however. For example, costs can be reduced by moving workloads to a container orchestration platform, and by migrating services to a pay per use model. Using the right data store for the right data can be a huge saving, and using a CDN to cushion your instances from sudden spikes of traffic, allowing for sufficient time to scale out.

Cost Reduction 4. Streamlining the Architecture

Most cloud environments are not isolated, but continuously evolve. In practice, added workloads and processes can lead to a more inefficient performance of an environment. It is smart to regularly review your workloads and infrastructure to get rid of inefficient use of services. It is even smarter to monitor these workloads and evolve the architecture through automation.

Cost Reduction 5. Operational Overhead

An often underestimated reduction in cost is operational overhead. Rarely, the human cost is taken into consideration: after all, it isn’t visible on your cloud vendor’s bill. Operational overhead can be reduced by opting for services offered by a cloud vendor that require no management whatsoever. Do not reinvent the wheel. The time saved by reducing operational overhead is time that can be spent on interesting and productive stuff. As such, you need less staff to do the same work while at the same time increasing the job satisfaction tremendously.

Example 1: Reducing Operational Overhead

Being in an operations engineering team of two is not fun, particularly when you have to maintain infrastructure that is built on legacy and good intentions. This was the case at Amsterdam-based Asellion, the developer of a global platform that makes the global chemical trade transparent and sustainable.

To reduce their cloud spend as well as the operational overhead, I set the operational overhead as THE metric for success. During weekly retrospectives we assessed if we were on the right path. By doing so I’ve been able to help Asellion by transitioning their architecture to a modern architecture based on best practices. While the team grew from 10 employees to over 50, employee satisfaction scores (eNPS) went through the roof. In addition, their developers are now deploying and maintaining their own applications and enjoy doing so. Asellion is definitely on my list as a cool cloud-based company now.

Example 2: Handle Peak Loads Gracefully

Some organizations experience the need for optimization as soon as they start scaling. At that moment, it becomes clear that processes are way too expensive. Stranded Flight Solutions (SFS) experienced this first hand.

SFS is a start-to-end global service recovery platform for airlines to improve the guest experience when these are challenged the most. I was asked to assess their infrastructure with a particular focus on the cost. I found that their applications were under-provisioned for peak capacity, while being over-provisioned for idleness. In other words: While most of the team the platform aw little traffic, as soon as passengers would find themselves stranded at an airport, the services could not handle the peak load. I helped Stranded Flight Solutions to transition to a mature architecture that is scalable by using serverless, CDNs and container orchestration. This has led to a substantial cost reduction

Free Review of Potential Cost Savings

To support organizations in the current challenging times, we offer a free intake where we explore potential cost savings. This call takes between 30-60 minutes. During the call, we will try to uncover potential savings on your monthly cloud bill. If potential savings are indicated, we can support you with an optional extensive review and remediation.

This cost optimization review helps you to identify the potential measurements you can take to minimize the cost of every individual use case.

After remediation, organizations save up to thousands of euros per month on their cloud bill.

Request your free cost optimization intake >>

Just like any other organization over the past week, Binx had to adjust to the new normal. What a week it has been! We hope that you and your beloved ones are in good health and spirit, and that you are in control of your business. Here’s an update of how we at Binx are coping with this abrupt change to our business.

It is my belief that the current situation will have a lasting impact on the way that business is done. By sharing our experiences, I’d like to give you an insight in what we are going through and the changes that we’ve made. If you have any other ideas or experiences to share, please let me know!

Decentralizing the Organization

In just one week, we have transformed into a decentralized company. For a business that is focused entirely on consultancy and training, working from home is quite the change. What didn’t change, is our commitment to help our clients make the most effective use of cloud technology.

Over the last week, we’ve been in close contact with many of our clients to look for creative and flexible solutions, so they can ensure their business continuity. Many of our clients have felt the immediate impact of the COVID-19 measures and it is great to see the solidarity and a ‘we can do this together’-mentality that has blossomed over the past few days.

Our IT Infrastructure

As a cloud-born consultancy and training boutique, we were set-up for a remote way of working from the start of our organization two years ago. Our cloud consultants work with the latest hardware and online tools. Luckily, most of them have already invested in a proper setup at their home office. If not, we support our consultants with this. Our business applications, mostly GSuite, are running in the cloud.
Working from home - Binx
We even allowed a few that got fed up with working from their kitchen chairs to borrow our office chairs!

Keeping the Team Aligned

For the team, we have regular Q&A sessions to keep everyone aligned and informed. We also do regular check-ins in the evening for the team to blow off some steam.

Our bi-weekly Xebia knowledge exchanges have been changed into virtual events. Normally, these XKEs take place from 4PM to 9PM every other Tuesday. The virtual XKE took place from 7PM to 9PM with four half-hour slots. There was even a 30-minute pre-recorded bootcamp to warm-up. 😉
Xebia Knowledge Exchange

Supporting our Clients

Our clients also see the impact of the Coronavirus on their business. Think of online travel agent Booking.com, flower auction Royal FloraHolland, cinema company Pathé, and all those other organizations that are hit by the semi-lockdown in the Netherlands.

Over the past week, we have focused primarily on keeping our clients cloud operations steady. Besides, we have increased our efforts to review and optimize the cost of workloads, brought our training curriculum online, and implemented managed cloud control for clients to support their mission-critical teams.

Keeping clients steady

We continue to help our clients to keep steady in these turbulent times by providing extra flexibility and creativity in the way that we are working with them.

Many of our consultants, much like the teams at our clients, have young families; the extra flexibility to work when they are able to focus, while also appreciating family commitments ensures that the work gets done and both our clients and employees are kept happy!

Cost Optimization

In these times, cost optimization is on the top of everyone’s agenda. We see that in nearly every cloud infrastructure there are things to improve, often leading to thousands of euros in monthly cost savings!

At Binx we are asked to review cloud workloads a lot. For these reviews, we look at the implementation of many aspects, like automation and security and advice on best-practices and remediation.

To support organizations in these trying times, we are conducting free consultation on potential cost reduction and optimization for cloud workloads on Azure, Amazon Web Services, Google Cloud, and Alibaba Cloud. If you are running workloads on the cloud, please feel free to ask us to do a free review on potential cost optimization.

Online Cloud Training

We understand that individuals and teams alike are still looking to develop their skills. For some, reskilling is more important today than before. With physical attendance of classroom training impossible, we have moved our entire curriculum online.

We are offering a complete curriculum of online cloud engineering training courses, offered through Xebia Academy.

Google Cloud Training

Wietse Venema, who is writing an O’Reilly book on Building Serverless applications with Google Cloud Run, is offering an online training for application developers to get going on Google Cloud. This training takes place on Friday, April 10th from 1PM CET until 5PM CET.

Amazon Web Services Training

Furthermore, we have complete and free hands-on learning journeys for AWS available on Instruqt.

Managed Cloud for Organizations

For some organizations, especially the ones with many external consultants, team size has been cut down to the bare minimum. We understand how stressful it can be if you’re the one responsible for mission-critical infrastructure. For those teams, being able to rely on managed cloud support can be more valuable than ever. We help organizations to implement cloud control, providing support and automating all the processes to the extreme.

We believe that especially in these times it is crucial to become as efficient and agile as possible. Adopting a cloud-native way of working contributes heavily to this. We are here to help you get there.

Let’s hope things take a turn for the better again soon. Take care.