We'd love to meet you in person. Drop by our office in Pondicherry, or reach out online.
Address
No 48/1, Plot no 73, 1st Floor, Aziz Nagar, 2nd Cross Street, Reddiarpalayam, Oulgaret Municipality, Pondicherry – 605010
Phone
+91 88387 24261ELT replaced ETL, warehouses replaced lakes for most workloads, and streaming stopped being exotic. A practitioner's look at what actually changed in modern data engineering, and what still trips teams up.
If you last touched a data warehouse architecture diagram five or six years ago, the field has moved further than you'd expect. Not just new tool names — the underlying philosophy of how data pipelines get built has shifted, and teams that haven't caught up are still fighting battles that better tooling has mostly solved.
The old model — transform data carefully before it ever touches the warehouse — made sense when compute was expensive and warehouses were slow. You couldn't afford to load messy data and clean it up later; you had to get the transformation right in a separate processing layer first, usually in a tool like Informatica or Talend, and usually maintained by a small team that became a bottleneck for every new data request.
Cloud warehouses changed the economics. Snowflake, BigQuery, and Redshift can chew through transformations on raw data cheaply enough that it no longer makes sense to do that work upstream. So the model flipped: load the raw data first, transform it where it lives, using SQL that analysts can actually read and modify themselves. Tools like dbt didn't invent this idea, but they made it accessible enough that transformation logic stopped being a black box owned by one team and became something version-controlled, tested, and reviewed like any other code.
The practical effect: a data request that used to take two weeks because it had to go through a pipeline engineer's backlog can now, in a lot of organizations, be handled by an analyst writing a dbt model and opening a pull request. That's not a small efficiency gain — it changes who can move the business forward with data.
A decade ago, "real-time" usually meant "we run the batch job every fifteen minutes instead of once a day," and true streaming architecture was reserved for a handful of companies with the engineering budget to justify it. That's no longer true. Kafka and its managed equivalents (Kinesis, Pub/Sub) have become common enough that a mid-sized company building a fraud detection feature or a live inventory dashboard reaches for streaming without it being a major architectural event.
That said, we'd push back on the instinct — and we see it a lot — to reach for streaming by default. Most reporting use cases genuinely don't need sub-second freshness, and a well-tuned batch pipeline running every few minutes is easier to build, easier to debug, and easier to hire for than a streaming system. Streaming earns its complexity when the business decision actually depends on the freshness — fraud detection, live pricing, operational dashboards someone is staring at in real time. Otherwise it's often complexity added for its own sake.
For a while, the industry pitched data lakes and data warehouses as two separate things you needed: a lake for cheap raw storage of everything, a warehouse for the structured, query-optimized layer analysts actually used. Maintaining both meant maintaining two copies of a lot of the same data, with all the drift and reconciliation headaches that implies.
The lakehouse pattern — cheap object storage (S3, GCS) with a transactional layer on top (Delta Lake, Iceberg) that gives you warehouse-like guarantees — has quietly closed that gap for a lot of teams. You get one source of truth instead of two, which sounds like a minor architectural detail until you've spent a weekend tracking down why a number in the warehouse doesn't match the same number in the lake.
Data quality checks used to be something a diligent engineer bolted onto a pipeline as an afterthought, usually a handful of SQL assertions nobody looked at until something broke. That's shifted toward treating data quality the way software engineering treats testing: expected, automated, and part of the definition of done for a pipeline.
Concretely, that looks like validating things at the point data lands, not weeks later when a report looks wrong:
select
count(*) as total_records,
count(distinct customer_id) as unique_customers,
min(order_date) as earliest_order,
max(order_date) as latest_order
from orders
where order_date::date = current_date;
A query like this, run automatically after every load and alerting when the numbers drift outside expected bounds, catches the kind of quiet data corruption that otherwise surfaces three months later as "why does finance's report not match sales'."
The single most common mistake we still see is skipping documentation on the assumption that the pipeline is self-explanatory to the person who built it. It is — for about six months. After that, the person who built it has moved teams, and whoever inherits the pipeline is reverse-engineering intent from SQL with no comments. Lightweight documentation of what a model does and why, kept alongside the code in something like dbt's built-in docs, pays for itself the first time someone other than the original author has to touch it.
The second recurring issue is treating orchestration as an afterthought until a pipeline fails silently over a long weekend. Airflow or Dagster aren't just schedulers — they're what makes a pipeline's failure visible and recoverable instead of a mystery discovered when someone finally opens a stale dashboard on Monday.
For a team building or rebuilding a data platform now, the practical starting point isn't picking the fanciest tool in each layer — it's being honest about actual freshness requirements and data volumes before reaching for streaming, a lakehouse, or a distributed processing framework that a much simpler stack would have handled fine. Start with the smallest architecture that solves the actual business question in front of you, instrument it well, and let real usage tell you where the genuine bottlenecks are before you add complexity to solve problems you don't have yet.
CosmoOps Team
CosmoOps Team