Key concepts

Data Package

A versioned code package that contains a live connection to a data source. It includes a library of both native functions and functions derived from your schema that enable you to easily execute a wide variety of queries. Typically, users install the library with a package manager and import it into their project as a dependency.

dpm CLI

An open source client (dpm) you can use to create and manage data packages.

dpm agent

When queries are submitted by a data package, they are routed through a server process called the dpm agent. The agent is responsible for enforcing access policies, translating queries into the appropriate source dialect, and translating the response into a standard JSON format. The agent also collects telemetry to power features like query logs, performance & usage monitoring, and auditing.

dpm web application

Use the dpm web application to create, manage, and discover data packages. You can also connect and manage sources, monitor and upgrade data package performance, and set access policies.

Source

The storage engine that will execute queries submitted from the data package, such as Snowflake, Databricks, S3, or Postgres. When packages are put in Accelerated mode, queries will be served by dpm's high performance engine.

Descriptor

The datapackage.json file that defines a data package. It specifies a schema, execution backend, and metadata such as the owner, version, name, and description of the data pacakge.