Portfolio

Hands-on data engineering and data architecture projects, running on my own Open Source Kubernetes lakehouse.

Although this platform is built and managed with the help of AI, I believe people must stay in control and accountable. So my main principle is: I set the architecture and standards, and everything I demonstrate here I must also be able to do without AI. It just takes a little longer ;-)

More at tigor.nl

Sovereign, open source

Everything here runs on open-source software (Kafka, Spark, Delta Lake, ClickHouse, Airflow, dbt, Superset, Trino) on Kubernetes in a European cloud region in Amsterdam.

Data sits in open formats (Delta Lake, Parquet) on S3-compatible storage: no vendor lock-in, no licence fees, and you decide where your data lives, which matters for GDPR and strategic independence. The patterns match Databricks and Microsoft Fabric, so skills and designs carry over both ways.

Leveraging AI

I build with an AI coding assistant as a pair engineer: it writes code, deploys and debugs, while I set the direction, review and decide. One engineer can now build in days what used to take a team weeks.

The intention is to run and maintain the platform with AI as much as possible, in a responsible way: humans approve risky changes, secrets stay out of reach, and every change is reviewable in git.

Open source is what makes this work. AI can read the source code, documentation and logs of every component, and the whole platform is described as code (Helm, YAML, SQL). That helps enormously in managing the complexity of a stack this size, where a closed product would be a black box.

Projects

Live map of aircraft over the Netherlands LIVE

Spark Structured Streaming and the ClickHouse Kafka engine

Live aircraft and landings over the Netherlands: Kafka, streaming into ClickHouse and Delta, shown in a real-time report.

KafkaSpark Structured StreamingClickHouseSuperset
Dashboard with incidents per hour: fixed by the agent, fixed by hand, open LIVE

A loading process that maintains itself

Every 15 minutes an AI agent checks the streaming pipeline end to end, repairs known failures with tested code and reports to my phone. Live incident figures.

Agentic AIMCPSelf-healingClaude Code
Airflow DAG with bronze, silver and gold layers

Medallion lakehouse with Airflow, dbt and PySpark

Bronze, silver and gold Delta tables: PySpark loads, dbt transforms and tests, Airflow orchestrates. Includes a replay of a 990-million-row run.

AirflowdbtPySparkDelta LakeMedallion
Medallion layers landing, bronze, silver and gold next to a star schema

Sales data product: from CSV to star schema

Northwind sales through the medallion layers on Delta Lake up to a Kimball star schema, pre-joined consumption tables in ClickHouse, incremental loading in every layer, one Airflow DAG and a Superset dashboard.

MedallionAirflowdbtDelta LakeClickHouseKimball
opencode building an agent while Claude Code watches every tool call

One AI agent watching another

The open-source agent opencode, on a European model, built my visitor-notification agent from one prompt. Claude Code watched all 118 tool calls live, checked every alarm and reviewed the result: careful work, zero suspicious actions.

opencodeClaude CodeGuardrailsEU model
Airflow DAG with three data products in parallel and phone notifications of an automatic repair

Self-healing nightly pipeline: three data products, one Airflow DAG

One Airflow DAG runs Northwind, AdventureWorks and Dutch weather in parallel every night. When a task fails, an AI repair agent finds the cause, fixes it and reruns only what failed, and my phone gets “OK na herstel”.

AirflowSelf-healingClaude CodeMCPKubernetes
Superset dashboard AdventureWorks sales built by AI agents

A data product built by AI agents, end to end

Four agents, business analyst, architect-modeller, data engineer and BI developer, take AdventureWorks from requirements to a Superset dashboard, working from my standards and handing over reviewable documents.

Agentic AIClaude CodeStandardsdbtSuperset
Dictionary entry for the business term Revenue

Semantic layer for AI agents with OpenMetadata

Why agents need business definitions: SQL composed only from glossary terms, formulas and joins in OpenMetadata, with real examples.

OpenMetadataBusiness glossaryMCPText-to-SQL
An old Joomla page and a new static site, with an AI agent in between

A 2010 website rebuilt by an AI agent

An AI coding agent replaced the whole Joomla 1.5 backend (PHP, MySQL) with a static site, redirected every old link, backed everything up to S3 and made blog publishing an MCP tool.

AI agentClaude CodeMCPMigration
A plug and a socket: MCP connects agents to tools

MCP servers: one protocol, many agents

MCP servers that let agents deploy the platform, write to my blog and generate queries for self-service BI, with humans in the loop.

MCPClaude CodeAgentsTerraformHelm
Checklist of my AI principles

How I work with AI: my principles

Accountable, so in control: I understand how things are done and avoid lock-in to one model. Deterministic results on a robust data foundation, with guardrails and best practices.

Responsible AIMetadata-drivenHuman in the loop
Screenshot of the claims assistant app with a Databricks or open-source backend

Text-to-SQL agent: Databricks or open source

One chat app, two interchangeable backends: Databricks Model Serving or FastAPI, PostgreSQL and Llama on Kubernetes. Same model, prompt and API.

DatabricksMLflowFastAPILlama 3.3Text-to-SQL