Adnan Rahic

Making Data Warehouses fast. A developer's story.

A developer went from 1.5-second BigQuery responses to under 500ms with 100 concurrent users. Here's the open-source tool they used to do it.

Making Data Warehouses fast. A developer's story.
#1about 3 minutes

The developer's struggle with data warehouse latency

High latency in applications built on data warehouses creates a poor user experience and presents a significant challenge for developers.

#2about 5 minutes

Differentiating between OLAP and OLTP database workloads

Data warehouses use OLAP for complex, low-volume queries on large datasets, contrasting with OLTP's high-volume, simple transactions.

#3about 3 minutes

Understanding the key factors of query latency

User-perceived performance is impacted by network delays and data scan times, making sub-second responses a critical goal.

#4about 7 minutes

Exploring BigQuery's caching and concurrency limitations

BigQuery's cache only works for identical queries and its concurrency is capped per project, impacting real-world application performance.

#5about 4 minutes

Benchmarking BigQuery's performance under concurrent load

Load testing reveals that BigQuery maintains a consistent query latency of around two seconds regardless of user concurrency up to its hard limit.

#6about 2 minutes

Introducing Cube as a semantic analytics API layer

Cube provides a semantic layer over data warehouses, enabling caching, pre-aggregations, and access control to build fast data apps.

#7about 3 minutes

Setting up a local Cube development environment

A local Cube instance can be configured using Docker Compose to connect to BigQuery and automatically generate data schemas.

#8about 5 minutes

How pre-aggregations dramatically improve query speed

Pre-aggregations act as materialized views that store condensed query results, reducing a query's response time from seconds to milliseconds.

#9about 3 minutes

Comparing benchmark results of Cube vs direct BigQuery

Benchmarks show that using Cube's pre-aggregation layer results in a nearly five-fold performance increase over querying BigQuery directly.

#10about 8 minutes

Answering questions on Cube's architecture and use cases

The discussion covers when to implement a caching layer, how Cube improves performance, and its utility for medium-sized databases.

Related jobs
Jobs that call for the skills explored in this talk.

Matching moments

Improving the developer feedback loop with specialized tools
03:16 MIN

Improving the developer feedback loop with specialized tools

Developer Time Is Valuable - Use the Right Tools - Kilian Valkhof

The origin story of the Polypane developer browser
05:28 MIN

The origin story of the Polypane developer browser

Developer Time Is Valuable - Use the Right Tools - Kilian Valkhof

The business case for sustainable high performance
03:34 MIN

The business case for sustainable high performance

Sustainable High Performance: Build It or Pay the Price

How a broken frontend culture impacts the web
04:02 MIN

How a broken frontend culture impacts the web

WeAreDevelopers LIVE – Frontend Inspirations, Web Standards and more

The growing importance of data and technology in HR
01:54 MIN

The growing importance of data and technology in HR

From Data Keeper to Culture Shaper: The Evolution of HR Across Growth Stages

Solving date and time issues with the Temporal API
06:47 MIN

Solving date and time issues with the Temporal API

WeAreDevelopers LIVE – You Don’t Need JavaScript, Modern CSS and More

Increasing the value of talk recordings post-event
04:57 MIN

Increasing the value of talk recordings post-event

Cat Herding with Lions and Tigers - Christian Heilmann

Why you might not need JavaScript for everything
02:33 MIN

Why you might not need JavaScript for everything

WeAreDevelopers LIVE – You Don’t Need JavaScript, Modern CSS and More

Featured Partners

Related Videos

See all videos
Uncharted Territories of Web Performance - Andrew Burnett-Thompson and David Burleigh49:57

Uncharted Territories of Web Performance - Andrew Burnett-Thompson and David Burleigh

Andrew Burnett-Thompson & David Burleigh

Database Magic behind 40 Million operations/s24:22

Database Magic behind 40 Million operations/s

Jürgen Pilz

Things I learned while writing high-performance JavaScript applications27:27

Things I learned while writing high-performance JavaScript applications

Michele Riva

Scaling: from 0 to 20 million users28:12

Scaling: from 0 to 20 million users

Josip Stuhli

Swapping Low Latency Data Storage Under High Load22:32

Swapping Low Latency Data Storage Under High Load

George Asafev

The Data Mesh as the end of the Datalake as we know it37:21

The Data Mesh as the end of the Datalake as we know it

Mario Meir-Huber

Lessons learned from building a thriving Vue.js SaaS application30:02

Lessons learned from building a thriving Vue.js SaaS application

Abdelrahman Awad

Interactive server side components46:13

Interactive server side components

Miško Hevery

Related Articles

View all articles
BB
Benedikt Bischof
Making Data Warehouses Fast: A Developer’s Story
Welcome to this issue of the WeAreDevelopers Live Talk series. This article recaps an interesting talk by Adnan Rahic who teaches the audience how to make data warehouses.About the Speaker: Adnan is senior developers advocate at Cube. His passion lie...
Making Data Warehouses Fast: A Developer’s Story

From learning to earning

Jobs that call for the skills explored in this talk.