Why DevOps?

DevOps is a cultural shift that combines development and operations into a single team. The automation framework is a key backbone to implementing a successful DevOps transformation. This article details the practical stages to implement DevOps on your stack.

Ask yourself:

• Do my developers test their features automatically or manually?
• Do my teams integrate their features automatically?
• Do we deliver artifacts to production manually?
• Do we prove code works automatically before promoting?
• Do we continually prove code works with the same tests?
• Do we release working features to production automatically?
• Can we recover production errors automatically?

All of these stages represent levels of DevOps maturity. This article provides a way for you to break down the DevOps automation components and plan for making 2021 the year when your team implements DevOps at a new level. …

To everything there a season, and a time to every purpose… Let’s keep track of those changes with a slowly changing dimension (SCD).

What is the slowly changing dimension? It is a table that changes gradually over time, like a group of elected officials. Suppose we have the following set of US Senators loaded into our database.

https://theunitedstates.io/congress-legislators/legislators-current.csv

Even though many have been in office for a long time, some are about to change. According to our constitution Article I, Section 3, clause 2, one-third of these senators will be subject to change.

Immediately after [the Senate of the United States] shall be assembled in Consequence of the first Election, they shall be divided as equally as may be into three classes. The Seats of the Senators of the first Class shall be vacated at the Expiration of the second Year, of the second Class at the Expiration of the fourth Year, and of the third Class at the Expiration of the sixth Year, so that one third may be chosen every second…

Most anti-patterns start with a trip to Stack Overflow, or better yet, Github. Say I need to import data into my analysis, and I want the quickest way that gets me data. The world of code snippets and spreadsheets has made it easy for me to import data and produce data products without much effort.

What are data anti-patterns? They’re solutions that look reasonable but perhaps don’t scale, aren’t secure, introduce quality issues, or some other affront to good data governance.

No one sets out to implement an anti-pattern. Say the election is coming up; it is! …

• The team starts building without clear business goals
• The team tries to solve problems it doesn’t understand fully using applications it understands even less
• The team chases Big Data “Shiny Object” technologies that are too complex and unnecessary

What can you do to build large scale data on AWS that will succeed?

• Separate execution and business flow using data pipelines
• Drive ingestion with proven event systems like SNS and SQS
• Only use “Big Data” systems for problems that need massive data throughput

Start with Clear Business Goals

Data initiatives often start with massive amounts of data that are scattered through silos and systems.

If your business goal is to make an aspect of that data useful for a decision-maker, the value of that data must be comprehended by the decision-maker — a human. A common data point is that short term memory can hold seven plus or minus two items. …

Every good adventure starts with a door. Perhaps the door is more than a door, but a gateway to some enchanted temple.

As a Dungeon Master, I want my software gateways to be as interesting as a D&D game. Sometimes a mysterious door leads to a hidden gem like Handling arbitrary HTTP requests in Amazon API Gateway.

It’s an excellent document because it cuts through all the complexity of AWS serverless functionality and exposes the crucial components. I did what any good dungeon master does, and took the source material and made it my own.

Mr. Eric Johnson, the author of the document distilled the key components of an API gateway. …

Ten years ago, we had a development database server we affectionately named “Terabytes Rex.” No irony. The mass extinction of on-premises database servers is well on its way. Like the extinction of the dinosaurs, the disintegration of the old order is blending with other catalysts of change and yielding a Cenozoic diversity of databases. A database now can be a massive in-memory store like Redis, a document like MongoDB, a column store databases such as Cassandra, or data warehouses like Redshift and Snowflake.

Is the old-school relational database like Postgres obsolete? Is it a waste of time to analyze primary and foreign keys? Should we spend time thinking about data types for columns or treat all data as strings? …