I have a quick blog entry on the Jetbrains Security Breach.

https://www.owlmountain.net/post/jetbrains-security-breach


Let’s transform our deploy processes this year

Person setting off fireworks
Person setting off fireworks
Photo by Steve Halama on Unsplash

Why DevOps?

DevOps is a cultural shift that combines development and operations into a single team. The automation framework is a key backbone to implementing a successful DevOps transformation. This article details the practical stages to implement DevOps on your stack.

Ask yourself:

  • Do my developers test their features automatically or manually?
  • Do my teams integrate their features automatically?
  • Do we deliver artifacts to production manually?
  • Do we prove code works automatically before promoting?
  • Do we continually prove code works with the same tests?
  • Do we release working features to production automatically?
  • Can we recover production errors automatically?

All of these stages represent levels of DevOps maturity. This article provides a way for you to break down the DevOps automation components and plan for making 2021 the year when your team implements DevOps at a new level. …


Image for post
Image for post
Photo by Anthony Garand on Unsplash

To everything there a season, and a time to every purpose… Let’s keep track of those changes with a slowly changing dimension (SCD).

What is the slowly changing dimension? It is a table that changes gradually over time, like a group of elected officials. Suppose we have the following set of US Senators loaded into our database.

https://theunitedstates.io/congress-legislators/legislators-current.csv

Even though many have been in office for a long time, some are about to change. According to our constitution Article I, Section 3, clause 2, one-third of these senators will be subject to change.

Immediately after [the Senate of the United States] shall be assembled in Consequence of the first Election, they shall be divided as equally as may be into three classes. The Seats of the Senators of the first Class shall be vacated at the Expiration of the second Year, of the second Class at the Expiration of the fourth Year, and of the third Class at the Expiration of the sixth Year, so that one third may be chosen every second…


Image for post
Image for post
Photo by Author

Most anti-patterns start with a trip to Stack Overflow, or better yet, Github. Say I need to import data into my analysis, and I want the quickest way that gets me data. The world of code snippets and spreadsheets has made it easy for me to import data and produce data products without much effort.

What are data anti-patterns? They’re solutions that look reasonable but perhaps don’t scale, aren’t secure, introduce quality issues, or some other affront to good data governance.

No one sets out to implement an anti-pattern. Say the election is coming up; it is! …


Successful data initiatives use a simple architecture that will scale. Most big data projects fail. Why do data initiatives fail?

Image for post
Image for post
Photo by Serghei Trofimov on Unsplash
  • The team starts building without clear business goals
  • The team tries to solve problems it doesn’t understand fully using applications it understands even less
  • The team chases Big Data “Shiny Object” technologies that are too complex and unnecessary

What can you do to build large scale data on AWS that will succeed?

  • Separate execution and business flow using data pipelines
  • Drive ingestion with proven event systems like SNS and SQS
  • Only use “Big Data” systems for problems that need massive data throughput

Start with Clear Business Goals

Data initiatives often start with massive amounts of data that are scattered through silos and systems.

If your business goal is to make an aspect of that data useful for a decision-maker, the value of that data must be comprehended by the decision-maker — a human. A common data point is that short term memory can hold seven plus or minus two items. …


Every good adventure starts with a door. Perhaps the door is more than a door, but a gateway to some enchanted temple.

The Church of the Transfiguration — Athens, Greece
The Church of the Transfiguration — Athens, Greece
Temple of the Transfiguration. Athens, Greece.

As a Dungeon Master, I want my software gateways to be as interesting as a D&D game. Sometimes a mysterious door leads to a hidden gem like Handling arbitrary HTTP requests in Amazon API Gateway.

It’s an excellent document because it cuts through all the complexity of AWS serverless functionality and exposes the crucial components. I did what any good dungeon master does, and took the source material and made it my own.

Mr. Eric Johnson, the author of the document distilled the key components of an API gateway. …


Image for post
Image for post

Ten years ago, we had a development database server we affectionately named “Terabytes Rex.” No irony. The mass extinction of on-premises database servers is well on its way. Like the extinction of the dinosaurs, the disintegration of the old order is blending with other catalysts of change and yielding a Cenozoic diversity of databases. A database now can be a massive in-memory store like Redis, a document like MongoDB, a column store databases such as Cassandra, or data warehouses like Redshift and Snowflake.

Is the old-school relational database like Postgres obsolete? Is it a waste of time to analyze primary and foreign keys? Should we spend time thinking about data types for columns or treat all data as strings? …


AWS API Gateway is one of the most common ways to trigger a Lambda function. The API gateway acts as a front end to the Lambda function, which is a wrapper for the functionality in your library.

What we want to do here is make a web service call that reproduces the pytest function we have already proven on our local system.

The original test is

The functional test I want to write will provide an end to end validation of the service as it is called through an HTTP get request as an API function.

curl -H “X-API-KEY: ${X_API_KEY}” https://${API_ID}.execute-api.us-east-1.amazonaws.com/Prod/parse_filename?Path="api-gateway/tests/data/apple_health_tracking_201911231045steps.csv"

About

Tim Burns

I am a data architect and software engineer. See my blog https://www.owlmountain.net/. My opinions are my own.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store