What are Data Pipelines?

A Data Pipeline is an automated process that transfers data from one system to another while also processing and transforming it along the way. It helps businesses collect data from multiple sources, prepare it for use, and deliver it to the required destination, such as analytics functionality, dashboards, databases, or internal systems.

Nowadays managing data manually would be inefficient and error-prone. Data Pipelines automate these operations and ensure that data remains consistent, structured, and accessible.

How it works?

Data delivery options

Database schemes

Payment Request scheme

The Payment Request scheme defines the data structure for a Payment Request used to initiate, process, and track payment transactions within the system. This entity contains information about the payment amount, currency, recipient, transaction status, as well as unique identifiers that allow tracking the execution of the Request.

The data in this Scheme is denormalised, which ensures fast access to key payment parameters without the need for additional queries to other tables. A Payment Request may include references to Payment Methods, accounting systems, or other financial sources for integration and transaction reconciliation.

Payment Commit scheme

The Payment Commit scheme defines the data structure used to record an attempt to process a Payment Request. It contains information about the transaction execution status, the Payment Provider’s response code, processing time, and any possible errors.

Each Payment Commit is linked to a corresponding Payment Request and represents an individual processing attempt. This allows the system to maintain a log of payment executions, handle retry attempts, and analyse the causes of potential failures.

Payment Transaction scheme

The Payment Transaction scheme contains detailed information about interactions with the Payment Provider during payment processing. It aggregates all stages of the payment operation lifecycle, including initiation, authorisation, charge, and Refund processing.

This entity records technical transaction details, statuses received from the payment system, unique operation identifiers, as well as metadata required for payment tracing and reconciliation. Payment Transaction is used to monitor payment execution, handle potential failures, and ensure the integrity of financial operations.

Refund scheme

The Refund scheme defines the data structure for a voluntary return of funds to a customer initiated by the Merchant. It contains information about the refund amount, currency, associated Payment Transaction, Refund reason, and processing status.

Data Pipelines benefits

1. Centralised and structured data

All payment-related data is automatically collected, processed, and delivered in a structured format. This simplifies data management and ensures consistency across reporting and analytics systems.

2. Faster access to business insights

With direct delivery of data into your infrastructure, teams can quickly access transaction information and generate reports without waiting for manual exports or synchronisation processes.

3. Full ownership and flexibility

Data Pipelines provide complete control over the data within your own environment. You can build custom BI reports, create internal monitoring tools, and adapt analytics processes according to specific business requirements.

4. Improved reliability and monitoring

Automated processing reduces the risk of possible errors and helps maintain data accuracy. Structured processing records also simplify troubleshooting, auditing, and reconciliation procedures.

5. Better transaction transparency

Detailed schemes such as Payment Request, Payment Commit, Payment Transaction, and Refund provide complete visibility into the payment lifecycle, making it easier to trace transactions, analyse failures, and monitor financial operations in real time.