Rails Log Analyzer

A high-performance command-line tool for analyzing Ruby on Rails log files, built in Rust for maximum speed and efficiency.

Features

• High Performance: Parse 1GB+ log files in seconds using memory-mapped files and parallel processing
• Problem Detection: Identify errors, exceptions, timeouts, and performance bottlenecks
• Performance Analysis: Extract detailed metrics including response times, database query times, and memory allocations
• Route Intelligence: Analyze route-specific performance patterns and identify slow endpoints
• JSON/CSV Conversion: Transform plain text logs to structured formats
• Flexible Configuration: YAML-based configuration for thresholds and filters

Installation

# Clone the repository
git clone <repository-url>
cd rails-log-analyzer

# Build the project
cargo build --release

# The binary will be available at target/release/rails-log-analyzer

Usage

Basic Analysis

# Analyze a single log file
rails-log-analyzer analyze production.log

# Analyze multiple files
rails-log-analyzer analyze app.log api.log background.log

Performance Analysis

# Focus on performance with custom slow threshold
rails-log-analyzer perf production.log --slow-threshold 500

# Show top 20 slowest routes
rails-log-analyzer perf production.log --top-routes 20

Problem Detection

# Find critical issues
rails-log-analyzer problems production.log --errors --timeouts --memory

Format Conversion

# Convert to JSON
rails-log-analyzer convert production.log --format json --output structured.json

# Convert to CSV
rails-log-analyzer convert production.log --format csv --output data.csv

Real-time Monitoring

# Monitor a log file (basic implementation)
rails-log-analyzer monitor production.log --follow

Advanced Options

# Use custom configuration
rails-log-analyzer analyze production.log --config rails-log-analyzer.yml

# Filter by time range
rails-log-analyzer analyze production.log --from "2024-01-15 10:00" --to "2024-01-15 11:00"

# Output in different formats
rails-log-analyzer analyze production.log --format json
rails-log-analyzer analyze production.log --format csv

Configuration

Create a rails-log-analyzer.yml file to customize analysis:

thresholds:
  slow_request_ms: 500.0
  slow_db_query_ms: 100.0
  error_rate_warning_pct: 5.0
  memory_allocation_warning: 50000

filters:
  exclude_paths:
    - "/health"
    - "/metrics"
    - "/favicon.ico"
  include_methods:
    - "GET"
    - "POST"
    - "PUT"
    - "DELETE"
  include_status_codes:
    - 200
    - 201
    - 301
    - 302
    - 400
    - 404
    - 422
    - 500
    - 502
    - 503

output:
  format: "table"
  max_routes: 50
  time_buckets: "1h"
  show_params: false
  truncate_long_paths: true

analysis:
  detect_n_plus_one: true
  track_memory_leaks: true
  identify_bot_traffic: true

Supported Log Formats

The analyzer supports standard Rails log formats:

Started GET "/users/123" for 192.168.1.1 at 2024-01-15 10:30:45 +0000
Processing by UsersController#show as HTML
Parameters: {"id"=>"123"}
User Load (2.3ms)  SELECT "users".* FROM "users" WHERE "users"."id" = $1 LIMIT $2
Rendered users/show.html.erb within layouts/application (89.1ms)
Completed 200 OK in 145ms (Views: 89.1ms | ActiveRecord: 23.4ms | Allocations: 15420)

Also handles:

• Heroku router logs
• Custom tagged logging formats
• Multi-line exception traces
• Background job logs (Sidekiq, etc.)

Output Examples

Summary Report

Rails Log Analysis Report
========================
Total Requests: 45,231
Error Rate: 2.3%
Avg Response Time: 145ms
P95 Response Time: 890ms

Issues Found:
  [CRITICAL] High error rate: 12.0% (count: 543)
    Route: /api/orders/process
  [WARNING] Slow P95 response time: 2018.0ms (count: 1)

Slowest Routes:
Route                    | Requests | Avg Time | P95 Time | Error %
/api/reports/generate    |      156 |   2.3s   |   4.1s   |   12%
/users/dashboard         |    3,421 |   650ms  |   1.2s   |    3%
/orders/process          |      89  |   890ms  |   1.8s   |    8%

JSON Output

{
  "timestamp": "2024-01-15T10:30:45.123Z",
  "request_id": "abc123",
  "method": "GET",
  "path": "/users/123",
  "controller": "UsersController",
  "action": "show",
  "status": 200,
  "duration_ms": 145.2,
  "db_time_ms": 23.4,
  "view_time_ms": 89.1,
  "allocations": 15420,
  "params": {"id": "123"},
  "ip": "192.168.1.1",
  "errors": [],
  "warnings": []
}

Performance Benchmarks

The tool is designed for high performance:

• Parse 10,000 log entries in ~40ms
• Analyze 50,000 entries in ~5ms
• Memory usage stays reasonable even for large files
• Utilizes parallel processing for multi-core systems

Testing

Run the test suite:

# Run all tests
cargo test

# Run specific test modules
cargo test --test lib

# Run benchmarks
cargo bench

Architecture

src/
├── parser/          # Log parsing engine with regex patterns
├── analyzer/        # Performance analysis and issue detection
├── reporter/        # Output formatting (table, JSON, CSV)
├── cli/            # Command-line interface using clap
└── config/         # YAML configuration handling

Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass: cargo test
5. Submit a pull request

License

MIT License - see LICENSE file for details.