/
SC Log Aggregation Tool

SC Log Aggregation Tool

Table of Contents

Overview

The SC Log Aggregation tool (SC_log_aggregation.bash) is a utility designed to assist in collecting and preparing log files from your Syndeia Cloud deployment for analysis or support purposes. It automatically gathers logs from various components of the SC-stack (Cassandra, Kafka, Zookeeper, JanusGraph, and Syndeia Cloud microservices), sanitizes sensitive information, and optionally compresses the logs for easier sharing.

Disclaimer

IMPORTANT: While this tool makes a best-effort attempt to sanitize potentially sensitive information from logs, it does not guarantee complete removal of all private, confidential, or security-related data. The sanitization process is provided as a convenience measure only and is not intended to be a comprehensive security solution. Users are advised to review any collected logs prior to sharing with the Intercax helpdesk. By using this tool, you acknowledge that you are responsible for ensuring compliance with your organization's data protection policies and applicable regulations.

Location

This script is included in your Syndeia Deployment files at:

syndeia-cloud-3.6-SP2\bin\SC_log_aggregation.bash

Key Features

  • Comprehensive Log Collection: Gathers logs from all SC-stack components:

    • Cassandra

    • Kafka

    • Zookeeper

    • JanusGraph

    • Syndeia Cloud microservices

  • Data Privacy: By default, sanitizes sensitive information like IP addresses, port numbers, and hostnames

  • Flexible Options: Configurable to include non-text files and control sanitization

  • Organized Output: Creates a structured directory of collected logs

  • Easy Sharing: Option to compress logs into a single ZIP file

  • Detailed Logging: Maintains its own log file for troubleshooting

Usage

Basic Usage

To run the script with default settings:

./SC_log_aggregation.bash

This will run in interactive mode, prompting you for preferences about sanitization, non-text files, output location, and compression.

Advanced Usage

The script supports various command-line options for more control or automated usage:

./SC_log_aggregation.bash [LOG_LEVEL] [options]

Where:

  • LOG_LEVEL is optional and can be DEBUG, INFO, WARN, or ERROR (default is INFO)

Available Options

Option

Description

Option

Description

-h or --help

Show help message and exit

-nt or --include-non-text

Include non-text files (not sanitized) without prompting

-d or --dest-dir <path>

Specify the destination folder for logs

-nc or --no-compress

Skip the compression step

-ns or --no-sanitize

Skip sanitization of IPs ports and hostnames in text files

-ni or --non-interactive

Run in non-interactive mode using defaults for unspecified options

Default values if running in non-interactive mode:

  • Log level: INFO

  • Non-text files: Excluded (only text files will be collected)

  • Output location: Current working directory

  • Compression: Enabled (logs will be compressed into a ZIP file)

  • Sanitization: Enabled (IPs, ports, and hostnames will be sanitized)

Example Usage

# Run with detailed logging output ./SC_log_aggregation.bash DEBUG # Show help information ./SC_log_aggregation.bash -h # Run with warning-level logging, including non-text files, saving to /tmp/logs, # without compression or sanitization ./SC_log_aggregation.bash WARN -nt -d /tmp/logs -nc -ns # Run non-interactively with default settings, but specify destination directory ./SC_log_aggregation.bash -ni -d /tmp/logs

Understanding the Output

After running the script, you will find:

  1. Log Collection Directory: A directory named SC-stack-logs-YYYYMMDDHHMMSS containing all collected logs, organized by service.

  2. Script Log File: A detailed log of the script's execution at sc-stack-collector-YYYYMMDDHHMMSS.log.

  3. Hostname Mapping File (if sanitization is enabled): A file showing the mapping between original and sanitized hostnames.

  4. ZIP Archive (if compression is enabled): A ZIP file containing all collected logs.

Directory Structure

SC-stack-logs-YYYYMMDDHHMMSS/ ├── cassandra/ ├── zookeeper/ ├── kafka/ ├── JG/ (JanusGraph logs) └── SC/ (Syndeia Cloud logs) ├── service1/ ├── service2/ └── ...

Privacy Considerations

By default, the script sanitizes text files to protect sensitive information:

  • IP addresses are replaced with "SERVERIP".

  • Port numbers for known services are replaced with service identifiers.

  • Local hostname is replaced with "SERVERNAME".

  • External hostnames are replaced with unique identifiers like "OBFUSCATED_HOST_000001".

A mapping file is created to allow reference back to original values if needed.

Non-Text Files Explained

Non-text files are any files that cannot be processed as plain text, including:

  • Binary log files

  • Memory dumps

  • Database files

  • Compressed archives

  • Image files

  • Executable files

  • Empty files

These files cannot be sanitized using text processing methods and may contain sensitive information such as:

  • IP addresses

  • Hostnames

  • Credentials

  • Configuration data

When the script asks whether to include non-text files, consider whether these files are necessary for troubleshooting the specific issue you're experiencing. If you choose to include them, be aware that they will be collected as-is without sanitization.

Performance Note

For processing large log sets more efficiently:

  • If the Linux utility parallel is installed, the script will use it to process files concurrently.

  • Without parallel, the script uses awk for sequential processing.

To install parallel :

  • On CentOS/RHEL: sudo yum install parallel

  • On Ubuntu/Debian: sudo apt-get install parallel

Troubleshooting

If the script encounters an error, it will:

  1. Log detailed information to the script log file.

  2. Display a message indicating where to find more information.

  3. Exit with an appropriate error code.

Common issues:

  • Insufficient permissions: Ensure you have read access to log directories.

  • Disk space: Ensure sufficient space for log collection and compression.

  • Missing dependencies: The script requires common utilities like 'find', 'sed', and optionally 'zip'.

Typical Support Scenario

When to Provide Logs

You should gather and provide logs when:

  1. You are experiencing issues with your Syndeia Cloud deployment.

  2. You have an active support request on our helpdesk and have been asked to provide logs.

  3. You are opening a new support request and want to include diagnostic information.

Step-by-Step Process

  1. Run the log aggregation tool:

    # Navigate to the bin directory in your Syndeia deployment cd /opt/icx/syndeia-cloud-current/bin # Run the script (example with common options) ./SC_log_aggregation.bash -d /tmp/logs

  2. Answer the prompts (if running in interactive mode):

    • Whether to skip sanitization (recommended: "n", default: sanitization enabled)

    • Whether to include non-text files (depends on support request, default: "n" - non-text files excluded)

    • Where to store logs (default: current working directory)

    • Whether to compress logs (recommended: "y", default: "y" - compression enabled)

  3. Review the collected logs before sharing:

    • Check the sanitization mapping file to ensure sensitive data has been properly obfuscated.

    • Verify no sensitive data remains in the logs that is critical to your security.

  4. Share the logs with Syndeia Support:

    • If you have an existing ticket: Upload the logs to your existing ticket.

    • If opening a new ticket: Create a support request at intercax.com/help and attach the logs.

What Files to Share

For most support cases, provide:

  1. The compressed ZIP file (e.g., /tmp/logs/SC-stack-logs-YYYYMMDDHHMMSS.zip)

  2. Optionally: The script log file (e.g., /tmp/logs/sc-stack-collector-YYYYMMDDHHMMSS.log)

Do NOT share the hostname mapping file unless specifically requested by Intercax support team, as it contains the mapping between obfuscated and actual hostnames.

Security Reminder

Before uploading logs, ensure you have:

  • Reviewed any security-sensitive logs for remaining sensitive data.

  • Confirmed that sharing these logs complies with your organization's security policies.

  • Removed or further obfuscated any data that should not be shared.

Support

If you encounter issues with the script or need assistance interpreting the collected logs, please contact Syndeia Support by opening a ticket at intercax.com/help with:

  1. The script log file (sc-stack-collector-YYYYMMDDHHMMSS.log)

  2. A description of what you were trying to accomplish