An alert system made for real time process monitoring of an IoT enabled CNC machine
This project is an implementation of a scalable data pipleine using Apache Kafka for IoT enabled CNC machines which live stream data using the MTConnect protocol.The data is streamed on the NIST Webpage which is the Kafka Producer in the implementation . The Kafka consumer consumes the messages published by the Kafka Producer.
The data provided by NIST is avaiable at the web page in the format as shown below.
The text based data generated by the IoT enabled machines by the MTConnect protocol is being interpreted by a Python programme in this implementation. The basic structure of the document is defined by the XML specification by the MTConnect standard and contains one or more XML Elements.
A sample XML schema of the data stream is shown below.
The Python programme is used to parse the XML webpage and produce the parsed information using Apache Kafka. The Kafka Consumer is also another Python programme which has functionality to detect events while consuming messages from the Kafka Producer
- Python 3
- Apache Kafka
- Apache Zookeeper
- kafka - A Python client for Apache Kafka
- xml.etree
- urllib.requests
- request
- time
The kafka (python client for Apache Kafka) is used to produce message to the Apache Kafka server , which is a distributed data streaming platform capable of handling trillions of data events in a day. The package can be imported by a simple import statement in Python
from kafka import KafkaProducerTo install Apache Kafka server , the instructions can be found on this link
For this implementation , a single node setup of Apache Zookeeper has been used. The steps to install Apache Zookeeper can be found on this link