Data Quality Engine

Data Quality Engine (Motor de Qualidade de Dados or MQD in Portuguese) is the open-source tool that evaluates the quality of data shared among participants of Open Finance, ensuring the integrity of the information.

Sensedia provides tools to facilitate integration between the Sensedia API Platform, Events Hub, and MQD, enabling efficient data processing.

The diagram illustrates this integration:

Integration flow between MQD

The diagram presents an Open Finance integration flow, where the Sensedia API Platform acts as the publisher of APIs eligible for MQD — such as accounts, cards, and loans.

Events generated by these APIs are asynchronously published to the Sensedia Events Hub, which consumes them, applies policies, classifies them, and forwards them via the MQD Subscriber API to the Financial Institution’s proxy.

This process ensures efficient, secure, and controlled data delivery.

Below is the step-by-step guide for integration:

Step 1: Configuring the Events Hub

Context

Create a context, as shown in the image below:

MQD context creation screen

For more information on configuring contexts in the Events Hub.

Handler

Create a handler, for example: MQD Handler.
In the Topics step:
1. Add client topics: accounts, consents, creditCardAccounts, etc. You must configure one topic for each API eligible for MQD.
2. Enable the context in each topic by clicking the magnifying glass.
  
  Event publication URL
  
  Each topic creates a publication URL, which must be configured in the Custom JS interceptor of the data APIs.
  
  See an example of a URL created by the Events Hub:

In the Policy step:

Create an IP Filtering Validation policy, specific to the outbound IPs of the Sensedia API Platform.

To check the IPs of the Sensedia API Platform, access the Infrastructure Manager:

Once logged into the platform, append /info directly to the end of the URL, as shown in the example below:

https://manager-example.sensedia.com/api-management/info/

Fill in the delivery settings fields:
- Retry count,
- Codes for automatic retries: 500, 502, 503, 504, 408, and
- Request timeout.

Once registered, the information should appear as follows:

IP Filtering Validation Policy

For more information on configuring handlers in the Events Hub.

Subscriber

Create a subscriber for MQD, which will be the MQD subscriber API.
In the Security step:
1. Validate the key and create a static token for the subscriber (optional).
In the Topics step:
1. Select the MQD Handler and then select the client topics. Enable the Publish option.
2. Fill in the Subscriber URL field with the endpoint of the MQD subscriber API.
3. Add the status code 200 in the corresponding field.

MQD subscriber configuration

If the API is inactive at the time of registration, the subscriber status will appear gray, but this does not mean it is inactive.

For more information on configuring subscribers in the Events Hub.

Step 2: Configuring the MQD subscriber API

Configure an MQD subscriber API that will act as a proxy with the financial institution’s environment. This API will be responsible for receiving data from the Events Hub and forwarding it to MQD.

When configuring this API:

In the Resources step:
1. Add a resource, for example: v1/report-data.

In the Flows step:

Insert the URL that will be the API’s destination, which varies depending on the API to be exposed.

Add the following interceptors:

IP Filtering: with the outbound IPs of the Events Hub (check the IPs with the support team).
Access Token Validation: the same inserted in the Events Hub.

Custom JS: will manipulate the header for MQD format. You can customize it for the client by removing some unnecessary headers.

Check the Custom JS interceptor in the API flow:

JS interceptor in MQD Subscriber API

You can configure the JS interceptor as follows:

const scriptName = String("MQD - Event Subscriber Interceptor ");
const scriptVersion = String("1.0.0");
try {
    $call.tracer.trace(">Sensedia debug => Custom javascript: Name [ "+scriptName+" ] version [ "+scriptVersion+" ]." );
    var json = JSON.parse( $call.request.getBody().getString( "utf-8" ) );
    $request.setHeader("endpointName", json.mqd.endpointName);
    $request.setHeader("serverOrgId", json.mqd.serverOrgId);
    $request.setHeader("x-fapi-interaction-id", json.mqd.fapiInteractionId);

    delete json['mqd'];
    $call.request.getBody().setString( $json.stringify(json), "utf-8" );

    $call.request.getAllHeaders().remove("x-clientname-webhooks-signature");
    $call.request.getAllHeaders().remove("access-token");


} catch (exception) {
    $call.tracer.trace("Exception message => " + exception.message + " <= in line => " + exception.stack);
    $console.debug(">sensedia => Exception", exception);
}

For more information on creating APIs in the Sensedia API Platform.

Step 3: Configuring the Custom JS interceptor in the flow of data APIs

In the Sensedia API Platform, select the data APIs eligible for MQD:

Add the Custom JS interceptor in the Flows step. If there is more than one API, place the interceptor once in each existing API.
In the Resources field, keep the "All" option.

Adding JS Interceptor in Flow

You can configure the Custom JS interceptor as follows:

const scriptName = String("MQD - Event Producer Interceptor");
const scriptVersion = String("1.1.0");

try {

    $call.tracer.trace(">Sensedia debug => Custom javascript: Name [ "+scriptName+" ] version [ "+scriptVersion+" ]." );
    $console.debug("responseStatusCode", $call.response.getStatus());

    if ($call.response.getStatus() == 200 && $call.request.getMethod() == "GET") {

        const url = String($call.request.getRequestedUrl().getPath());

        const apiName = extractApiNameByUrl(url);

        $console.debug("apiName", apiName);

        const mqdTopicUrl = String($call.environmentVariables.get('mqd-'+ apiName +'-topic-handler-url'));

        $console.debug("mqdTopicUrl", mqdTopicUrl);

        if(mqdTopicUrl) {

            let jsonResponse = JSON.parse( $call.response.getBody().getString("UTF-8") );

            jsonResponse.mqd = {};
            jsonResponse.mqd.endpointName = substituirPathPorPathId(url);
            jsonResponse.mqd.serverOrgId = $call.request.getHeader("x-organisation-id");
            jsonResponse.mqd.fapiInteractionId = $response.getHeader("x-fapi-interaction-id");

            const response =  $http.post(mqdTopicUrl, $call.response.getHeaders(), $json.stringify(jsonResponse));

            $console.debug("response", response.responseText);
            $console.debug("response status", response.getStatus());

            $call.tracer.trace("Custom javascript: Name [ "+scriptName+" ] version [ "+scriptVersion+" ].Envio para MQD Finalizado." );

        } else {
            $call.tracer.trace("Custom javascript: Name [ "+scriptName+" ] version [ "+scriptVersion+" ]. ATENCAO!  environmentVariable >'mqd-"+ apiName +"-topic-handler-url'< nao configurada! Ignorando envio para topico MQD." );
        }
    }
} catch (exception) {
    $call.tracer.trace("Exception message => " + exception.message + " <= in line => " + exception.stack);
    $console.debug(">sensedia => Exception", exception);
}


function substituirPathPorPathId(urlSearch) {

  function substituir(match, pathAnterior) {
      // Remove the last character "s" from the previous path
      const pathId = pathAnterior.slice(0, -1) + 'Id';
      // Return the pathId within braces
      return '/' + pathAnterior + '/{'+pathId+'}';
  }
  // Standard UUID regex for data API resources
  let regex = /\/([^\/]+)\/([a-f0-9-]+)(?=\/|$)/g;

  // If it's the consents API, the regex will be different
  if(apiName === "consents"){
    regex = /\/([^\/]+)\/(urn:[a-zA-Z0-9-]+:[a-f0-9-]+)(?=\/|$)/g;
  }

  // Replace the previous path name plus "Id" with the string inside braces
  const novaUrl = urlSearch.replace(regex, substituir);

  return novaUrl.match(/open-banking(.+)/)[1];
}

function extractApiNameByUrl(url) {
  const start = "open-banking/";
  const startIndex = url.indexOf(start);

  if (startIndex === -1) {
    return "";
  }

  const endIndex = url.indexOf("/", startIndex + start.length);
  if (endIndex === -1) {
    return url.substring(startIndex + start.length);
  }

  return url.substring(startIndex + start.length, endIndex);
}

In the Custom JS interceptor:

The method used will always be GET, and the response will be 200, as the JS interceptor will be triggered when the data API receives a successful response.
Configure environment variables with the URL of the Events Hub topic. Topics create publication URLs for each of them (the number of URLs varies depending on the number of APIs).

With this configuration, the interceptors capture responses eligible for MQD and publish them to an Events Hub topic.

For more information on configuring interceptors and the JS interceptor in the Sensedia API Platform.

Thanks for your feedback!

EDIT

Share your suggestions with us!
Click here and then [+ Submit idea]