Cost-Effective Security Log Processing: A Case Study with Fluent-bit, NATS, Elasticsearch, and S3

A medium-sized company has implemented an in-house security log processing system to handle approximately 2 terabytes of daily data from over 200 services and databases. The data flow begins with fluent-bit for log collection, then uses NATS as the messaging system to transport logs to Elasticsearch for search and alerting. Logs are also stored on Amazon S3 for long-term retention with a 7-year policy. The system processes an average of 24,000 messages per second, with peak capacity of 200,000 messages per second during incidents, and generates alerts in under one second. The system is managed by a team of six security professionals. This solution was implemented to avoid the high costs of traditional Security Information and Event Management (SIEM) solutions, which were estimated at $50,000 per month. The company also cited greater flexibility as a key advantage of their in-house system. Technically, the implementation leverages several key technologies. Fluent-bit is used for efficient log collection from diverse sources. NATS provides a high-performance messaging backbone capable of handling the substantial message throughput. Elasticsearch enables real-time search and analysis of security events, while Amazon S3 serves as a cost-effective solution for long-term log retention. This case study demonstrates that medium-sized organizations can achieve efficient and cost-effective security log processing without relying on expensive commercial SIEM solutions. The system's performance metrics and cost savings make it a compelling example for organizations seeking to optimize their security operations. For cybersecurity professionals, this implementation offers several actionable insights. It illustrates the effectiveness of combining open-source tools with cloud storage for security log processing. It demonstrates that real-time alerting can be achieved with a small team and appropriate architecture. Finally, it provides a concrete example of how to design a cost-effective security log processing system using modern technologies.