Skip to content

Reporting without waiting on an engineer.

A busy application datastore was also carrying the reporting workload. We created a dedicated data pipeline and a way for the business team to query the results.

Project:An affiliate marketing business, in collaboration with Data DrivenA past project. The story describes the work delivered and the technology used at the time.

The problem

Elasticsearch was handling heavy application reads and writes while also serving reporting queries. Its custom query syntax meant business users depended on engineers to create or change reports.

An analyst using an organized reporting warehouse to answer questions without engineering help.

What we built

We worked with Data Driven to move reporting data into Amazon Redshift through an ETL pipeline, then connected Looker so the business team could work with it.

  • Moved data regularly from Elasticsearch into Redshift.
  • Separated reporting from the application’s busy datastore.
  • Connected and configured Looker for the business team.

What changed

The business team could answer its own questions.

The warehouse supported familiar SQL queries and an off-the-shelf reporting tool. Business users could create reports and run ad hoc queries without requesting engineering time for each change.

The technical decisions

Give the reporting workload its own place.

A dedicated analytical datastore avoided asking the application’s primary system to serve two competing workloads.

A pipeline alone was not enough.

The second part of the project was access: configuring a tool the business team could use. That turned the warehouse into a useful part of its day-to-day work.

Let’s talk

Have a similar problem?

A rough idea is enough. We’ll work out the next step together.

Tell us what you need