Powerful, scalable and user-friendly log and audit solution for finance industry

Problem

In complex systems, different components create various logs, making it challenging for users to search and track them effectively. A financial company required a reliable, generic approach to search these logs, as if they were just one log. However, logs with sensitive information must have specific roll-off policies to adhere to rules and regulations.

Solution

The Nubis team designed a log structure and roll-off solution using Elasticsearch. We leveraged Logstash to pipe various log sources into Elasticsearch, which sorts them according to log types, such as audit logs and debug logs.

Different storage and roll-off policies are applied per log type, depending on policies. Other policies optimized data availability speed versus database cost by automatically and seamlessly moving some logs to cheaper and slower storage and roll-off.

An advanced but easy to use query language and an underlying data storage model allowed instant search through all logs in one go, while also providing the ability to narrow down the search by log type, time period and numerous other parameters.

The solution offered the client a centralized, scalable and user-friendly approach to search their logs, making it easy for users to track activities across different components. Furthermore, different storage and roll-off policies helped the client to save database costs and adhere to regulations by ensuring that sensitive logs are securely stored and automatically deleted after the specified period, and old data is put in cheaper cold-storage.

Conclusion

In conclusion, Nubis’s audit and log solution provided the financial company a powerful solution that was reliable and cost-effective. It streamlined the search process and provided a user-friendly interface, allowing client to focus on their business operations without worrying about their logs.