Co-authored by Ed Montgomery & René Fusco, Rapid7
In today’s cybersecurity landscape, organizations need robust detection and response solutions to stay ahead of evolving threats. Rapid7’s InsightIDR, the foundation of our Managed Detection and Response (MDR) service, empowers security teams with advanced analytics, automation, and expert-led investigations. Whether used as a standalone SIEM and XDR platform or in combination with MDR, InsightIDR’s latest Log Search enhancements bring even more value across the board. These updates accelerate response times, simplify complex queries, and improve the investigation process for both our MDR clients and product-only customers.
These updates, including Simplified Query Building, Pre-Computed Queries, and Bloom Filters, enhance the speed, accuracy, and accessibility of log search for security teams, ensuring faster, more targeted threat investigations for organizations.
Let’s explore how these updates elevate the detection and response lifecycle.
Simplified Query Building: Empowering Analysts to Act Faster
A key element of any detection and response solution is the ability to quickly turn data into actionable insights. Simplified Query Building enables analysts to construct and refine log searches faster, without complex syntax or technical details. This user-friendly interface enables any InsightIDR user, regardless of technical expertise, to create advanced queries through point-and-click prompts, accessing critical data quickly to streamline investigations.
By lowering the barrier to creating queries, Simplified Query Building provides organizations with timely, data-backed insights into incidents, reducing investigation time for both Rapid7’s MDR team and InsightIDR customers. This update ensures that every security team member, regardless of tenure, can access and leverage the power of InsightIDR’s log data without becoming bogged down by technical complexities.
Pre-Computed Queries: Reducing Time-to-Response for All Investigations
Time is critical when it comes to threat response.With Pre-Computed Queries (PCQs), both MDR and product-only customers benefit from reduced log search times. PCQs enable predictably fast, near-instant access to insights by pre-calculating query results in real-time as data arrives, enhancing responsiveness for all InsightIDR users.
Customer Feedback
"As an MSSP, InsightIDR's ability to handle large amounts of data is key for identifying threats in our client environments. Pre-Computed Queries have reduced return times for complex searches by over 70%, allowing us to create more impactful insights for our clients."
— Mat Cornish, Technical Director, Longwall Security
While InsightIDR already supports saving queries for reuse, PCQs take it further by pre-computing results, helping analysts to instantly identify patterns or gather evidence. Additionally, the Log Search home tab organizes queries by “Recent,” “Saved,” and “Pre-computed,” enabling users to quickly find what they need for streamlined incident handling. Whether you’re a customer conducting an in-house investigation or part of Rapid7’s MDR team, PCQs ensure faster insights and more efficient incident response.
Bloom Filters: Accelerating Key Value Pair Searches for Precise Threat Hunts
Not all queries can be pre-calculated in advance. Security teams are frequently asked questions about potential exposure to specific indicators of compromise (IoCs), such as flagged IP addresses or hash values. With Bloom Filters, both MDR and product-only customers gain a performance boost in search time for precise threat hunts by reducing unnecessary data processing.
For exact match searches, like identifying a compromised IP address or hunting for a suspicious hash value where(hash.sha="..."), Bloom Filters optimize search time by ruling out irrelevant data - enabling the algorithm to skip logs that would not have matches. This enhancement is implemented on the backend and occurs automatically for any search that contains an exact match key-value pair. Reducing the search space means accelerating analysts’ ability to hone in on the exact information they need, cutting down investigation time dramatically.
A recent research effort into InsightIDR’s new indexing approach, which leverages Bloom Filters, showed impressive results with:
- Improved Efficiency: Approximately 40-60% of all searches have experienced noticeable speed improvements since deployment.
- Increased Precision: The new index has enabled applicable queries to skip irrelevant data three to four times more effectively, leading to shorter search durations for even more efficient investigations.
Bringing It All Together: Faster, More Effective Detection and Response
Whether you’re a Rapid7 MDR customer or an InsightIDR product-only user, these Log Search updates significantly enhance detection and response capabilities. By reducing search times, simplifying complex queries, and pinpointing threats with greater accuracy, we provide every InsightIDR user with faster, more effective security outcomes.
This means:
- Faster Detection: Pre-Computed Queries and Bloom Filters accelerate search processes, enabling quicker response to incidents across both MDR and product-only use cases.
- Improved Visibility: Simplified Query Building ensures analysts can quickly refine searches and access the data needed for comprehensive investigations.
- Targeted Threat Hunts: Optimized key-value pair searches focus on the most relevant data, delivering quicker results for security teams.
Want to see these improvements in action? Contact us today to learn how Rapid7’s MDR service can protect your organization. You can also try InsightIDR for free with a 30-day trial.
Source: Rapid7
Source Link: https://blog.rapid7.com/2024/11/15/new-idr-log-search-enhancements-accelerate-streamline-and-simplify-investigations/