Why Kafka ? What does it solve?
Each system can produce data, consume data or both.
Integrating between systems becomes a must.
If system A produces data, and System B & C needs this data, you will have to integrate A +B , then A+C.
Integrating systems costs a lot with this way ( i mean two by two).
- Imagine you have 4 tools, the number of possible integration between every two is : C(4,2) = 6
- Imagine you have 10 tools, the number of possible integration between every two is : C(10,2) = 45
Note: C(n,k) is a maths formula, check it here: https://www.calculatorsoup.com/calculators/discretemathematics/combinations.php
Kafka comes to avoid all these integration. Instead of integrating two by two, you just need to put Kafka as centralized place for Data Streaming : Data producers ( systems which produces data) need to report their data to Kafka & Data consumers ( systems which needs this data) they just need to subscribe to Kafka ( e.g Kafka topics,… so on)