Skip to main content

Principle #3: Stay Aligned

Integrate with existing software development processes, tools, and frameworks.

A data product is a piece of software. In order to bring it to production and maintain it in the long-run, it will have to be absorbed by the engineering or IT teams in your organization and integrate into existing development processes. If the process for developing data products is not aligned with the general software engineering process, that transition can be very painful, costly, and jeopardize the success of the data product.

Why Data Product Implementations Loose Alignment​

Building data products brings some unique challenges around data management, data integration, and data infrastructure design. At the beginning of a data product implementation it may seem difficult to impossible to address those challenges within the existing software development process, and teams choose to adopt other processes from data science or data engineering that better address those challenges.

However, those processes rely on different tools, structures, and frameworks which causes misalignment with the main development process. So, while there may be a short-term benefit of adopting a different process, the long-term costs of that decision are often higher and can fail the entire project.

Integrate with Existing Development Processes​

We find that organizations who invest into building their data product development process on top of their existing software engineering process are able to produce value-generating data products more reliably and with less heroic integration efforts. This means streamlining processes, tools, and development infrastructure.

While the specifics vary between organizations, we recommend:

  • Use the same development cycle.
  • Use the same planning and project management frameworks.
  • Code-as-truth: Keep all implementation and deployment artifacts in code.
  • Integrate into the CI/CD framework and other automation frameworks.
  • Manage data dependencies like you manage software dependencies.
  • Use the same software release and change-management process.