Getting started

Let's build a data pipeline with DataSQRL in just a few simple steps.

Installation

You can install DataSQRL on your Mac with HomeBrew or use Docker on any machine.

Mac

brew tap datasqrl/sqrl
brew install sqrl-cli

note

Check that you're on the current version of DataSQRL by running sqrl --version To update an existing installation:

brew upgrade sqrl-cli

Docker

Pull the latest Docker image to ensure you have the most recent version of DataSQRL:

docker pull datasqrl/cmd:latest

Implement SQL Script

We are going to build a data pipeline that aggregates sensor metrics. With DataSQRL, you can implement the entire data pipeline in a single SQL script.

1. Create a new folder for the data pipeline:

mkdir metrics; cd metrics

2. Then create a new file called metrics.sqrl and copy-paste the following SQL code:

metrics.sqrl

IMPORT datasqrl.example.sensors.SensorReading; -- Import metrics
IMPORT time.endOfSecond;  -- Import time function

/* Compute the average temperature per second for each sensor */
SecReading := SELECT sensorid, endOfSecond(time) as timeSec,
                     avg(temperature) as temp 
              FROM SensorReading GROUP BY sensorid, timeSec;

/* Get max temperature in last minute per sensor */
SensorMaxTemp := SELECT sensorid, max(temp) as maxTemp
                 FROM SecReading
                 WHERE timeSec >= now() - INTERVAL 1 MINUTE
                 GROUP BY sensorid;

3. Compile the SQL script to an integrated data pipeline:

Mac

sqrl compile metrics.sqrl

Docker

docker run -it -p 8888:8888 -v $PWD:/build datasqrl/cmd compile metrics.sqrl

4. By default, DataSQRL uses docker to run data pipelines locally. Start the pipeline with docker compose:

(cd build/deploy; docker compose up --build)

5. The pipeline exposes the data through a GraphQL API. You can query it here: http://localhost:8888/graphiql/

Run the query to see the results:

{
  SensorMaxTemp (sensorid: 1) {
    maxTemp
  }
}

Congrats! You've built a robust and scalable data pipeline with just a few queries! Check out more of what DataSQRL has to offer, you might be surprised how this simple abstraction can unlock advanced features and functionality.

For a continuation of the features of DataSQRL, check out the Sensor Data Tutorial.