In my previous posts I talked about data from a high level perspective and introduced Apache Avro, a data serialization system. This time we are going to get hands-on with Apache Avro and look at Schemas Introducing the domain We need to have a domain and because this is a blog, lets keep the domain simple. Lets create a domain that consists of a single person entity. The person entity will change over time, and we will have three versions. Read more...

In my previous posts I talked about data from a high level perspective. This time we are going to look at a data serialization system called Apache Avro. Introducting Apache Avro Apache Avro is created by Doug Cutting, the creator of Hadoop, Lucene and Nutch, specifically designed as an open data serialization system. Apache Avro is used by data processing systems like Hadoop, Spark and Kafka, and is the de-facto data serialization system for high volume, high performance, high throughput, data processing systems. Read more...

The Scala Build Tool or sbt for short, is a build tool for building source code. It is a very advanced tool that is based on a workflow engine. In contrast to other build tools, scala is very simple, well, if you know just a couple of concepts. Concepts Lets introduce the core concepts of our build. When we create a project for example with the following command: sbt new dnvriend/scala-seed. Read more...

Now, this time I want to talk about data, in an informal way so we get a high level understanding what data is. That way we get a feeling for data as a concept, and how we can shape and design data to become a valuable asset. Introducing Data Lets meet data. Data lives in our computers, and drives the modern industry. Data is everywhere and plays a big part in our daily lives. Read more...

When designing your data lake, there a lot of aspects that guide the design. In this blog we’ll look at how we can compose a data lake from a very small component called a ‘data segment’. Lets take a look! The Data Segment The data segment is a small component that is a building block for data lakes. The data segment provides streaming ingestion, streaming transformation, streaming aggregation and data persistence capabilities. Read more...

A lot of organisations turn to BigQuery as a datawarehouse solution. But as the amount of tables increases and the number of users grows, it is easy to lose track of what data is accessed by whom. Authorized Views can help organize and control access to your sensitive data. They don’t require you to change your logical dataset design and still enable your applications and analysts to use the full power of BigQuery. Read more...

The Swift Package Manager (SwiftPM) is a tool for building your Swift applications. Just like other build tools like Maven or Gradle, SwiftPM manages project dependencies, and can build, test and publish packages. SwiftPM provides a standard way to build Swift applications and is a replacement for build tools like CocoaPods and Carthage. SwiftPM is a cross platform build system that is available for MacOS and Linux. Apple published an introduction video Getting to Know Swift Package Manager - Apple WWDC 2018 and SwiftPM has been documented. Read more...

Kotlin is a statically typed programming language designed by Jetbrains, the creator of the IntelliJ IDE platform. The language was first released in 2016, and since then has received a lot of traction. Kotlin is used as a backend language, as a front-end language and on the mobile platform, specifically Android. Kotlin is evolving at an enormous pace, and will be available on all platforms, from JVM, JavaScript to native. Read more...

At re:Invent 2018, AWS announced Lambda Custom Runtimes where you can implement an AWS Lambda runtime in any programming language. In this blog we are going to create our own bootstrap in Go. Bootstrap Bootstrap is a file in the deployment archive, that is responsible for reading the ‘AWS_LAMBDA_RUNTIME_API’ environment variable, and interacting with the Lambda runtime API by getting the next event and sending a response. The Lambda runtime API is available as a webservice which makes it very easy to create a bootstrap application in Go. Read more...

Gradle is an open-source build automation tool focused on flexibility and performance. Gradle build scripts are written using a Groovy or Kotlin DSL. The focus of Gradle is to build your software. Grade needs Java to operate. Gradle was released in 2007, has a great track record, is battle proven and is a stable build tool. Gradle is widely used in the industry for automating the creation of software. Gradle comes with great documentation including several tutorials and a YouTube channel containing tutorials and talks. Read more...