A WordPress.org Data Engineering Starter Project

WordCamp Asia 2026 — Mahangu Weerasinghe

SLIDES

Quick Start

Prerequisites: uv, git, make. Windows users: use WSL. See the GitHub repo for more details.

			
git clone https://github.com/mahangu/meltano-wordpress-org-data-starter-project.git
cd meltano-wordpress-org-data-starter-project
make quickstart

View on GitHub →

The Problem

WordPress.org API theme and plugin data is useful, but using it directly means writing a lot of code.

The SOLUTION: THe Open Source Data Stack

Four open-source tools, all running locally:

Meltano — Extract, Transform, and Load (ETL) data. Uses our custom extractor tap-wordpress-org and the target-duckdb loader.
DuckDB — Store and query large datasets fast, locally.
Jupyter — Run everything in a local notebook.
Python — Powers it all.

Available Commands

			
make help              Show available targets
make quickstart        Install, create sample data, extract events, launch notebook
make extract-plugins   Extract WordPress plugins data
make extract-events    Extract WordPress events data
make extract-themes    Extract WordPress themes data
make extract-all       Extract all available data streams
make sample-data       Create sample data from WordPress.org API
make notebook          Start Jupyter notebook
make check-data        Check what data is in the database

		

Extra Credit

LLMs can help with all of these:

Extract more data: make extract-all
Save to CSV or other formats
Create charts with matplotlib, seaborn, or plotly
Build interactive dashboards with Streamlit
Keep data fresh with Meltano schedules and Apache Airflow