Data Engineering
Real-time Sentiment Pipeline
Kafka + Spark streaming sentiment analysis, fully Dockerized.
- Year
- 2024
- Role
- Solo builder
- Type
- Data Engineering
Overview
Real-time sentiment analysis infrastructure: producers scraping Hespress, Twitter and YouTube feed Kafka; Spark Streaming processes the firehose into MongoDB; Streamlit dashboards visualize results live. Every component containerized with Docker Compose.
Highlights
- Multi-source scraping producers (news, Twitter, YouTube)
- Spark Streaming consumption into MongoDB
- Live dashboards, one-command Docker deployment
Stack
- Kafka
- PySpark
- MongoDB
- Docker
- Streamlit
- Python
