Data Stack

1. Data Sources

2. Capture

   • Batch Ingest
   • Stream Ingest
     • Fluentd
     • Apache Kafka

3. Store

   • Data Warehouse
     • Snowflake
     • Amazon Redshift
     • Azure Synapse
     • Google BigQuery
   • Data Lake
     • Databricks
     • Delta Lake

4. Transform

   • Orchestrate Tasks
     • Airflow
     • Fivetran
     • Azure Data Factory
     • AWS Data Pipeline
     • Google Dataflow
   • Batch Processing
     • Trino
     • Presto
     • dbt
   • Machine Learning
     • Spark
     • Amazon SageMaker
     • TensorFlow
     • PyTorch
     • Kubeflow
   • Stream Processing
     • Spark Streaming
     • Apache Flink

5. Publish

   • Feature Store
     • Tecton
   • Data Catalog
     • Data Hub
     • Amundsen
   • Real-Time Store
     • Apache Druid
     • MongoDB
     • Apache Pinot

6. Consume

   • Advanced Analytics
   • BI
   • Real-Time Apps

7. Data Products

References

• Dataops for the Modern Data Stack and how to get started, https://www.youtube.com/watch?v=T2FBl23rZt8