aidbox-subscriptions-to-kafka

Topic-based Subscriptions to Kafka

Demo | Documentation

This example showcases Aidbox SubscriptionTopic producing data to Kafka.

Objectives:

1. Set up Aidbox and Kafka locally using Docker Compose.
2. Get FHIR QuestionnaireResponse via Aidbox Forms.
3. Learn how AidboxSubscriptionTopic and AidboxTopicDestination work with Kafka to handle the collected data.

Table of Contents

• Topic-based Subscriptions to Kafka
  • Prerequisites
  • Step 1: Set Up the Environment
    • Set Up Aidbox
    • Run Aidbox, Kafka & Kafka UI
  • Step 2: Set Up Subscription and Destination
    • Create AidboxSubscriptionTopic Resource
    • Create AidboxTopicDestination Resource
  • Step 3: Demonstration
    • Submit Form
    • Check AidboxTopicDestination Status
    • See Messages in Kafka UI
  • Example of Kubernetes Setup
  • Demo

Prerequisites

• Docker
• Cloned repository: Github: Aidbox/examples
• Working directory: aidbox-subscriptions-to-kafka

Step 1: Set Up the Environment

Set Up Aidbox

1. Copy the .env.tpl file to .env:

   cp .env.tpl .env

2. Get a self-hosted Aidbox license from the Aidbox Portal.

3. Add the license key (AIDBOX_LICENSE) to the .env file.

Run Aidbox, Kafka & Kafka UI

docker compose up

• Aidbox is be available at http://localhost:8888/
  • Username: admin
  • Password: password
• Kafka UI is be available at http://localhost:8080/
• Kafka is available at http://localhost:9092/ (no authorization required)

The Docker Compose file initializes the environment for both Kafka and Aidbox with the following configuration:

• Imports FHIR Questionnaire (see init-aidbox service).
• Creates a Kafka topic for QuestionnaireResponse (see init-kafka service).

Step 2: Set Up Subscription and Destination

Create AidboxSubscriptionTopic Resource

To create a subscription on the QuestionnaireResponse resource that has a specific status, open Aidbox UI -> APIs -> REST Console and execute the following request:

POST /fhir/AidboxSubscriptionTopic
content-type: application/json
accept: application/json

{
  "resourceType": "AidboxSubscriptionTopic",
  "url": "http://example.org/FHIR/R5/SubscriptionTopic/QuestionnaireResponse-topic",
  "status": "active",
  "trigger": [
    {
      "resource": "QuestionnaireResponse",
      "fhirPathCriteria": "status = 'completed' or status = 'amended'"
    }
  ]
}

This resource describes the data source for the subscription but doesn't execute any activities from Aidbox.

Create AidboxTopicDestination Resource

Creating this resource establishes a connection to the Kafka server. When the system produces an event, it will be processed to the specified Kafka topic.

POST /fhir/AidboxTopicDestination
content-type: application/json
accept: application/json

{
  "meta": {
    "profile": [
      "http://aidbox.app/StructureDefinition/aidboxtopicdestination-kafka-at-least-once"
    ]
  },
  "kind": "kafka-at-least-once",
  "id": "kafka-destination",
  "topic": "http://example.org/FHIR/R5/SubscriptionTopic/QuestionnaireResponse-topic",
  "parameter": [
    {
      "name": "kafkaTopic",
      "valueString": "aidbox-forms"
    },
    {
      "name": "bootstrapServers",
      "valueString": "kafka:29092"
    }
  ]
}

Step 3: Demonstration

Submit Form

Open the list of forms, click share -> enable 'allow amend' -> click attach -> copy the link -> open the link -> fill out the form, and submit it.

Check AidboxTopicDestination Status

Open the Aidbox REST Console and get the AidboxTopicDestination status:

GET /fhir/AidboxTopicDestination/kafka-destination/$status

See Messages in Kafka UI

Open Kafka UI -> Topics -> aidbox-forms -> messages and review the QuestionnaireResponse that was created after submitting the form.

Example of Kubernetes Setup

Also you can find example of k8s deployment:

• Configuration: k8s.yaml
• Also, you need to pass secrets for Aidbox and Database. See details: Deploy Aidbox with Helm Charts. We recommend to use helm.
• Configuration resource examples: k8s_resources

Demo

A deployed and configured Aidbox instance with Kafka is available for you to explore how Aidbox's SubscriptionTopic works. The SubscriptionTopic in Aidbox is set up to send QuestionnaireResponse events in the completed and amended status to Kafka.

To try it out:

1. Open Aidbox Forms
2. Share form, copy the link.
3. Open the link and fill form.
4. Open the Kafka UI to view your QuestionnaireResponse in the Kafka messages tab.