Streaming Pipelines

A streaming pipeline is a system or framework that allows the processing and analysis of data in a continuous and real-time manner.   Typically the data is acquired from an API, web service, databases, file systems, etc., but it is possible for web scraping can be integrated into a streaming pipeline to extract data from websites in real-time.  

Related Links

Stream Your Data Using Nothing But Python’s Requests Library

Tutorials to Build Batch ETL Pipelines

Connect to an API with Python (Use the Reddit API and BigQuery to build a Daily News Pipeline)

Access and Manipulate URL-based Data (How to Use Python to Access Data in Multiple URLs with Rick and Morty)

(using Python to scrape a webpage)