Reply to post: Re: So did I misread the article or..?

Fivetran slammed for dropping SQL support. CEO: 'Blame me for this'

Anonymous Coward
Anonymous Coward

Re: So did I misread the article or..?

Correct. DBT is SQL-ish. SQL if you squint. You're meant to template out the queries, which is, er, nice if you're into that kind of thing. This is honestly one of the biggest points of friction, because j2 templates are an arse way of managing the encapsulation and instantiation of queries. dbt is moving to fix this by enabling Python-based models. But this ultimately ends up feeling like you're working in a hobbled version of Spark (spoiler: you are) rather than a first-class transformation language.

The main difference versus pure SQL is that DBT has an opinionated view on how queries and execution should be structured. In particular DBT requires you to work in terms of materialised models, with each "model" being a select statement and some rules about how/whether to persist it and some constraints. The "magic" is that DBT enables you to perform optimisations across your entire DAG of jobs, and likewise to encapsulate and indepedently test parts of the DAG, in theory enabling a "software-like" workflow for data people. These are the bits 5tran are looking to inherit for minimum effort. For them otherwise building a SQL-centric solution would be much harder and less sticky.

Ultimately though I think both dbt and 5t and the whole "modern data stack" are just fads. There are serious limitations both to their "unbundled" architecture and to their technology implementations. A platform built on unbundled components is always going to have its lunch eaten by one that is able to introspect itself and self-optimise, which more mature/established players like Databricks (DLT), AWS (Glue) etc. are all doing.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon