DevOps is a cultural shift that combines development and operations into a single team. The automation framework is a key backbone to implementing a successful DevOps transformation. This article details the practical stages to implement DevOps on your stack.
All of these stages represent levels of DevOps maturity. This article provides a way for you to break down the DevOps automation components and plan for making 2021 the year when your team implements DevOps at a new level. …
To everything there a season, and a time to every purpose… Let’s keep track of those changes with a slowly changing dimension (SCD).
What is the slowly changing dimension? It is a table that changes gradually over time, like a group of elected officials. Suppose we have the following set of US Senators loaded into our database.
Even though many have been in office for a long time, some are about to change. According to our constitution Article I, Section 3, clause 2, one-third of these senators will be subject to change.
Immediately after [the Senate of the United States] shall be assembled in Consequence of the first Election, they shall be divided as equally as may be into three classes. The Seats of the Senators of the first Class shall be vacated at the Expiration of the second Year, of the second Class at the Expiration of the fourth Year, and of the third Class at the Expiration of the sixth Year, so that one third may be chosen every second…
Most anti-patterns start with a trip to Stack Overflow, or better yet, Github. Say I need to import data into my analysis, and I want the quickest way that gets me data. The world of code snippets and spreadsheets has made it easy for me to import data and produce data products without much effort.
What are data anti-patterns? They’re solutions that look reasonable but perhaps don’t scale, aren’t secure, introduce quality issues, or some other affront to good data governance.
No one sets out to implement an anti-pattern. Say the election is coming up; it is! …
What can you do to build large scale data on AWS that will succeed?
Data initiatives often start with massive amounts of data that are scattered through silos and systems.
If your business goal is to make an aspect of that data useful for a decision-maker, the value of that data must be comprehended by the decision-maker — a human. A common data point is that short term memory can hold seven plus or minus two items. …
Every good adventure starts with a door. Perhaps the door is more than a door, but a gateway to some enchanted temple.
As a Dungeon Master, I want my software gateways to be as interesting as a D&D game. Sometimes a mysterious door leads to a hidden gem like Handling arbitrary HTTP requests in Amazon API Gateway.
It’s an excellent document because it cuts through all the complexity of AWS serverless functionality and exposes the crucial components. I did what any good dungeon master does, and took the source material and made it my own.
Mr. Eric Johnson, the author of the document distilled the key components of an API gateway. …
Ten years ago, we had a development database server we affectionately named “Terabytes Rex.” No irony. The mass extinction of on-premises database servers is well on its way. Like the extinction of the dinosaurs, the disintegration of the old order is blending with other catalysts of change and yielding a Cenozoic diversity of databases. A database now can be a massive in-memory store like Redis, a document like MongoDB, a column store databases such as Cassandra, or data warehouses like Redshift and Snowflake.
Is the old-school relational database like Postgres obsolete? Is it a waste of time to analyze primary and foreign keys? Should we spend time thinking about data types for columns or treat all data as strings? …
AWS API Gateway is one of the most common ways to trigger a Lambda function. The API gateway acts as a front end to the Lambda function, which is a wrapper for the functionality in your library.
What we want to do here is make a web service call that reproduces the pytest function we have already proven on our local system.
The original test is
The functional test I want to write will provide an end to end validation of the service as it is called through an HTTP get request as an API function.