Apache Flume - Ingesting log data into Hadoop and Kafka
Description
As part of this session we will understand how we can use Apache Flume to ingest streaming real time data in detail.
- Overview of Flume
- Setting up gen_logs
- Develop first Flume Agent
- Understand Source, Sink and Channel
- Flume Multi Agent Flows
- Get data into HDFS using Flume
- Limitations and Conclusion
For this demo we will be using our Big Data developer labs. You need to have access to existing big data cluster or sign up to our labs.
Hands On demos:
- Developing simple Flume agent to get data from netcat to agent logs
- Develop multi agent flow where data from web server logs go to avro sink and then from avro source to logger
- Develop multiplex flow where data from web server logs is written to HDFS and Kafka
Udemy FREE Course for Limited Time HURRY UP
100%
Free
Course
