A Case for Transience—Building Better Data Architectures by Using Throw-away Data
Transcience in Data Architecture
The modern data architecture spans a wide range of data storage. A few examples of the data engines the modern data architect encounters include big data engines like Databricks and Snowflake, caching systems like Redis, and sequential queues like SQS. When data needs persistence and transfer, a data architect must comprehend, design, integrate, govern, and manage it.
Not surprisingly, with such a diverse array of tools and paradigms, the best path for the data architect is often avoiding persistence. Transient data reduces cost and technical complexity because it disappears when you finish it.
A Case Study where Removing Persistence was Key to better Architecture
An Amazon team achieved significant cost savings and operational efficiency by transitioning from a storage system that relied on S3 to one that stores all intermediate data in memory. This successful move is detailed in the article ‘Scaling up the Prime Video audio/video monitoring service and reducing costs by 90%.’
We realized that distributed approach wasn’t bringing a lot of benefits in our specific use case, so we packed all of the…
