Skip to content

Latest commit

 

History

History
58 lines (37 loc) · 2.54 KB

README.md

File metadata and controls

58 lines (37 loc) · 2.54 KB
Raw

IoT-machine-monitoring-using-Apache-Kafka

An alert system made for real time process monitoring of an IoT enabled CNC machine

Motivation

This project is an implementation of a scalable data pipleine using Apache Kafka for IoT enabled CNC machines which live stream data using the MTConnect protocol.The data is streamed on the NIST Webpage which is the Kafka Producer in the implementation . The Kafka consumer consumes the messages published by the Kafka Producer.

Screenshots

The data provided by NIST is avaiable at the web page in the format as shown below. nist

The text based data generated by the IoT enabled machines by the MTConnect protocol is being interpreted by a Python programme in this implementation. The basic structure of the document is defined by the XML specification by the MTConnect standard and contains one or more XML Elements.

A sample XML schema of the data stream is shown below.

mtc

The Python programme is used to parse the XML webpage and produce the parsed information using Apache Kafka. The Kafka Consumer is also another Python programme which has functionality to detect events while consuming messages from the Kafka Producer

Tech/framework used

  1. Python 3
  2. Apache Kafka
  3. Apache Zookeeper

Packages/Modules

  1. kafka - A Python client for Apache Kafka
  2. xml.etree
  3. urllib.requests
  4. request
  5. time

Code Example

The kafka (python client for Apache Kafka) is used to produce message to the Apache Kafka server , which is a distributed data streaming platform capable of handling trillions of data events in a day. The package can be imported by a simple import statement in Python

from kafka import KafkaProducer

Installation

Apache Kafka Installation

To install Apache Kafka server , the instructions can be found on this link

Apache Zookeeper Installation

For this implementation , a single node setup of Apache Zookeeper has been used. The steps to install Apache Zookeeper can be found on this link

How to use?

Credits

  1. Data Intensive Manufacturing Environment ,NCSU,USA
  2. NIST: National Institute of Standards and Technology