We need to make it easier to build data-driven applications. Databases are great if all your application needs is storing and retrieving data. But if you want to build anything more interesting with data - like serving users recommendations based on the pages they are visiting, detecting fraudulent transactions on your site, or computing real-time features for your machine learning model - you end up building a ton of custom code and infrastructure around the database.

You need a queue like Kafka to hold your events, a stream processor like Flink to process data, a database like Postgres to store and query the result data, and an API layer to tie it all together.

And that’s just the price of admission. To get a functioning data layer, you need to make sure that all these components talk to each other and that data flows smoothly between them. Schema synchronization, data model tuning, index selection, query batching … all that fun stuff.

The point is, you need to do a ton of data plumbing if you want to build a data-driven application. All that data plumbing code is time-consuming to develop, hard to maintain, and expensive to operate.

We need to make building with data easier. That’s why we are sending out this call to action to uplevel our database game. Join us in figuring out how to simplify the data layer.

We have an idea to get us started: Meet DataSQRL.

After two long years of research, development, and teamwork, we're excited to announce DataSQRL 0.1! DataSQRL is a tool for building APIs from your data streams and datasets by defining your use case in an SQL.

This is our first “official” release of DataSQRL after many months of testing and bug-fixing.
Check out the release notes on GitHub for a rundown of all the features.

Introduction​

Every developer, whether you build applications or backend services, encounters two distinct types of data problems: transactional and reactive. The need to store and retrieve application state is a quintessential example of a transactional data problem. Conversely, when you're processing events or consuming data from external sources, you're confronted with a reactive data problem.

Knowing which problem you're up against is crucial to selecting the right tools from your developer's kit. It’s important to determine what type of data problem you are dealing with to choose the right tools and approaches for implementing a solution. After all, using a hammer for a screw job can leave you with more than a few cracks to mend.

In this post, we'll guide you on how to differentiate between transactional and reactive data problems and pick the right tools and strategies to solve each of them.

We are excited to launch DataSQRL with the mission to help developers and organizations build with data.

Collectively, we have spent decades building or helping others build data products. We have seen many struggles, failures, and piles of money being thrown out the window and figured that there must be a better way. We started DataSQRL to find it.

We believe that the technologies used to build data products are too complex and that the engineering processes used to build them are broken. Here is how we plan to fix these issues.