- Created by Debesh Roy Chowdhury, last modified by Brian Miller on Oct 14, 2024
You are viewing an old version of this page. View the current version.
Compare with Current View Page History
« Previous Version 12 Current »
Overview
In this section, the Automated Deployment method for Syndeia Cloud on Linux OS is presented. In the Automated Deployment method, Syndeia admins will run scripts that download and configure Syndeia Cloud and its infrastructure components - Cassandra, JanusGraph, Kafka, and Zookeeper.
There is also an option to first download Syndeia Cloud and its infrastructure components, and then run the automated scripts to install. This is useful when deploying Syndeia in an air-gapped environment. See the Offline Installation Mode (Optional) section below to learn more.
Syndeia admins must use either the Automated Deployment (presented here) or the Semi-Automated Deployment method. See the guidance on the parent page: Deployment Methods.
Prerequisites Summary
Deployment page read and understanding of:
Hardware that meets Requirements
Software that meets Version Requirements
Supported OS & shell:
Preferred and Recommended - RHEL/CentOS/Alma Linux v7.9-8.10 with Console and/or SSH access enabled
(
INFORMATIONAL: SC Web Ports 9000 (HTTP) | 9443 (HTTPS) will be opened automatically by adding a firewallservice definition/etc/firewalld/services/syndeia.xml)bashshell is used
Software: Compatibility Layer Components
Linux:
JRE/JDK (automatically installed during Cassandra installation)
Software: Infrastructure Components (automatically downloaded OR can use offline mode, see below)
Apache Cassandra
Janusgraph
Apache Zookeeper
Apache Kafka
Software: Syndeia Cloud (SC) Components
Syndeia Cloud (SC) media file .ZIPs downloaded : Download .ZIPs from password-protected links provided in the Intercax Helpdesk request where you originally received your Syndeia Cloud license. Filenames for each are as follows.
syndeia-cloud-3.6_cassandra_zookeeper_kafka_setup.zipsyndeia-cloud-3.6_janusgraph_setup.zipsyndeia-cloud-3.6.zip
Installation Logging
Before proceeding with any deployment steps in the CLI it is highly recommended you first enable input & output logging of the terminal as the shell does not do this by default.
Either enable (maximum) logging in your terminal of choice and/or use GNU script. See Appendix F6.1: Installation Logging for more details.
Offline Installation Mode (Optional)
If you are not in an offline environment, ex: air-gapped , & your server has internet access, simply skip to the next step.
If you are in an air-gapped environment, or wish to do an offline installation, please see the Offline Installation Mode (Optional) page.
Steps
Extract SC Media
Launch a Terminal with
bash(ensure you are in your home directory)Place all SC packages in your home directory
Unzip all SC packages:
unzip syndeia-cloud-3.6*.zip
Deploy Apache Cassandra
cd to the
cassandra_zookeeper_kafka_setuppackage’sbindirectory:cd ~/syndeia-cloud-3.6*_cassandra_zookeeper_kafka_setup/bin
Run the Apache Cassandra pre-setup script:
./syndeia-cloud-3.6_cassandra_pre-setup.bash
This will download Casssandra 4.1 and upgrade from 3.11.x to 4.1, or just upgrade the already downloaded Cassandra 4.1. Depending on what version you are upgrading from, you should see output similar to the below:
syndeia-cloud-3.6_cassandra_pre-setup.bash sample output
[jdoe@new-server bin]$ ./syndeia-cloud-3.6_cassandra_pre-setup.bash
+ shopt -s extglob
+ shopt -s globstar
++ readlink -f ./syndeia-cloud-3.6_cassandra_pre-setup.bash
+ SC_version_parsed=/home/jdoe/syndeia-cloud-3.6-SP1.2024-09-11_cassandra_zookeeper_kafka_setup/bin/syndeia-cloud-3.6_cassandra_pre-setup.bash
+ SC_version_parsed=/home/jdoe/syndeia-cloud-3.6-SP1.2024-09-11
+ SC_version_parsed=3.6-SP1.2024-09-11
+ SC_version=3.6-SP1.2024-09-11
+ service_name=cassandra
+ Cassandra_build_ver=4.1.0
++ uname
+ OS=Linux
+ [[ Linux =~ CYGWIN ]]
+ systemctl=systemctl
+ sudo=sudo
+ cassandra_yaml=/etc/cassandra/default.conf/cassandra.yaml
+ cassandra_jar_path=/usr/share/cassandra
+ [[ -d /usr/share/cassandra ]]
++ ls -1 /usr/share/cassandra/apache-cassandra-3.11.13.jar
+ cassandra_ver_current=/usr/share/cassandra/apache-cassandra-3.11.13.jar
+ cassandra_ver_current=/usr/share/cassandra/apache-cassandra-3.11.13
+ cassandra_ver_current=3.11.13
+ [[ ! 3.11.13 == 4.1.0 ]]
+ C_upgrade=true
++ egrep -Iwi '^cluster_name' /etc/cassandra/default.conf/cassandra.yaml
++ sed 's/cluster_name: //;s/'\''//g;'
+ cluster_name='SC35SP2TO36SP1 Cluster'
++ egrep -Iwi '^num_tokens' /etc/cassandra/default.conf/cassandra.yaml
++ sed 's/num_tokens: //;'
+ num_tokens=256
+ listen_address=localhost
+ rpc_address=localhost
+ write_request_timeout_in_ms=20000
+ batch_size_fail_threshold_in_kb=300
+ umask 0002
+ pushd .
~/syndeia-cloud-3.6-SP1.2024-09-11_cassandra_zookeeper_kafka_setup/bin ~/syndeia-cloud-3.6-SP1.2024-09-11_cassandra_zookeeper_kafka_setup/bin
+ cd /home/jdoe/
+ [[ '' == \-\d ]]
+ [[ Linux =~ CYGWIN ]]
+ [[ -e /etc/systemd/system/cassandra.service ]]
++ sudo systemctl is-active --quiet cassandra
+ [[ -n '' ]]
++ which wget
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot wget
+ [[ ! -n /usr/bin/wget ]]
++ which python
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot python
/usr/bin/which: no python in (/home/jdoe/.local/bin:/home/jdoe/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin)
+ [[ ! -n '' ]]
+ sudo yum install -y python2
Last metadata expiration check: 1:57:05 ago on Mon 16 Sep 2024 09:45:45 AM EDT.
Package python2-2.7.18-17.module_el8.10.0+3783+2756348e.alma.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
+ sudo yum install -y yum-plugin-versionlock
Last metadata expiration check: 1:57:07 ago on Mon 16 Sep 2024 09:45:45 AM EDT.
Package python3-dnf-plugin-versionlock-4.0.21-25.el8.noarch is already installed.
Dependencies resolved.
Nothing to do.
Complete!
+ sudo yum install -y java-11-openjdk-headless-11.0.23.0.9-3.el8
Last metadata expiration check: 1:57:08 ago on Mon 16 Sep 2024 09:45:45 AM EDT.
Dependencies resolved.
========================================================================================================================================================
Package Architecture Version Repository Size
========================================================================================================================================================
Installing:
java-11-openjdk-headless x86_64 1:11.0.23.0.9-3.el8 appstream 42 M
Transaction Summary
========================================================================================================================================================
Install 1 Package
Total download size: 42 M
Installed size: 178 M
Downloading Packages:
java-11-openjdk-headless-11.0.23.0.9-3.el8.x86_64.rpm 16 MB/s | 42 MB 00:02
--------------------------------------------------------------------------------------------------------------------------------------------------------
Total 16 MB/s | 42 MB 00:02
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Running scriptlet: java-11-openjdk-headless-1:11.0.23.0.9-3.el8.x86_64 1/1
Preparing : 1/1
Installing : java-11-openjdk-headless-1:11.0.23.0.9-3.el8.x86_64 1/1
Running scriptlet: java-11-openjdk-headless-1:11.0.23.0.9-3.el8.x86_64 1/1
Verifying : java-11-openjdk-headless-1:11.0.23.0.9-3.el8.x86_64 1/1
Installed:
java-11-openjdk-headless-1:11.0.23.0.9-3.el8.x86_64
Complete!
++ sudo alternatives --list
++ awk '/java-11-openjdk/{ print $3; exit }'
+ JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.23.0.9-3.el8.x86_64
+ sudo alternatives --set java /usr/lib/jvm/java-11-openjdk-11.0.23.0.9-3.el8.x86_64/bin/java
+ sudo yum versionlock java-11-openjdk-headless
Last metadata expiration check: 1:57:19 ago on Mon 16 Sep 2024 09:45:45 AM EDT.
Adding versionlock on: java-11-openjdk-headless-1:11.0.23.0.9-3.el8.*
+ [[ ! -e cassandra-4.1.0-1.noarch.rpm ]]
+ wget https://apache.jfrog.io/artifactory/cassandra-rpm/41x/cassandra-4.1.0-1.noarch.rpm
--2024-09-16 11:43:05-- https://apache.jfrog.io/artifactory/cassandra-rpm/41x/cassandra-4.1.0-1.noarch.rpm
Resolving apache.jfrog.io (apache.jfrog.io)... 54.185.186.5, 44.226.59.123, 100.21.156.46
Connecting to apache.jfrog.io (apache.jfrog.io)|54.185.186.5|:443... connected.
HTTP request sent, awaiting response... 302
Location: https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-apache/filestore/c1/c1d1d16277c04361547cdfa6123b3edd7c5aab83?X-Artifactory-username=anonymous&X-Artifactory-repoType=local&X-Artifactory-repositoryKey=cassandra-rpm&X-Artifactory-originPackageType=yum&X-Artifactory-packageType=yum&X-Artifactory-artifactPath=41x%2Fcassandra-4.1.0-1.noarch.rpm&X-Artifactory-originProjectKey=cassan&X-Artifactory-projectKey=cassan&X-Artifactory-originRepoType=local&X-Artifactory-originRepositoryKey=cassandra-rpm&x-jf-traceId=ae53672e8ee87d1d&response-content-disposition=attachment%3Bfilename%3D%22cassandra-4.1.0-1.noarch.rpm%22&response-content-type=application%2Fx-rpm&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEN%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJIMEYCIQC6DjEe%2Fw49YcU%2F9YJe6ZnyEmF0xHm9UBA0Z9pqRh9S%2FwIhALmZjh2yzqq8apsTQAv9mHngWesA3M7wm1TqCaMOwbedKoIFCBgQAhoMOTk5OTM0NzkwNDE0Igw%2BqQATIuO5eUYS2rwq3wSrUTYSVgSmib2bM3ttqIWb8v%2FUZnuU8iOo7ZuJlttYaFKuWlDLp5D3ANTOHfsdVHL2g%2FU4cC7sCK%2FbEhcK0rjIoldE5NQjMpghzvkdRk3q1gcLJoDONNNZ9D2ggg%2FE0qL8DHihEQeFkf%2FaNhs9wU6zkAqRHsTpq5rV1pFpOF3aQQXolRp6hhOySIzBqY0eb0%2Bl9F%2BE47rxC3bbtJjfRRztM4AeVaVSPh5a%2FSCWXkZRdj9VFlUu6%2F89uO9C6APKcJc8OjRSYAkJ8VCqtbM15ZZqn%2FvdhohMEAjgNmLBi82psGpMJCtcLealFLADbo9THCWS4bA%2F%2BRA0IYhO%2BTO6waHjwUQ94c3F%2FJ2FgLU745%2FzA4KWiCREQQt5uhfvglwDvQs3C8QMslAmcUaiMRBkP5F2P11%2FxoKY5mMZVRfvsEmW9UY%2F0eTjcV8kyWXD91wJkTUxXyqYhkg23Y21YN6zdM1jrHsgOGGW%2BR53LRhRWNkyoR5QBMTGcdWzPllERC7pVNVKL2j%2Fg%2FFdpjkGlVIGnVrx1ixEdOJAwepV2RLK0e0Dvnw06Db%2BS9n32ILYx4wv3iTlIrYrwg12MzjH%2B5RlvHTeHDp7X%2Bg2j1VUxGUlUu1dLDXMaLANVCBApZIaTpgSd6lsfbJ2a1gFexx7suXrpKFN3Lgb0xzr6guPUd41W6nWa0jaEMO%2BUX1Mwff76Atwa54c%2BE%2FluDjXv42yCkCPuDXwlt%2FXGby9SQzAoiAX5nrAwKPCdwya50%2FD3smBkjQBqaPioMzG%2FoLu7%2FNJgHCqzMtatQ9dBikE20BwpCgvPbTZMKKQobcGOpkB4Po52wZ25VaeBP95c4etBgT4UmS9l1nYCiUt%2F0gqpQwlsUMMdcNEAIgF22PC1pjXVFK4UxVCkXjRX7FI5P4G0uFGd16JsqBgHlSAoURafimDaOrHTYT%2FBWHuBvXQEFHy36ueq3IEslQ%2F88zNDzbquH7xjy7kJGGfDkKuqAk2f3%2FZk0PifKxxq1cmXgMgQDaFTm%2BqmqC10EK0&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20240916T154305Z&X-Amz-SignedHeaders=host&X-Amz-Expires=600&X-Amz-Credential=ASIA6RUGCBMHOPFV7DHC%2F20240916%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=e593d0c7bb16c610b0aef3c4643667872225a64531c95dae4d74b141585620fe [following]
--2024-09-16 11:43:05-- https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-apache/filestore/c1/c1d1d16277c04361547cdfa6123b3edd7c5aab83?X-Artifactory-username=anonymous&X-Artifactory-repoType=local&X-Artifactory-repositoryKey=cassandra-rpm&X-Artifactory-originPackageType=yum&X-Artifactory-packageType=yum&X-Artifactory-artifactPath=41x%2Fcassandra-4.1.0-1.noarch.rpm&X-Artifactory-originProjectKey=cassan&X-Artifactory-projectKey=cassan&X-Artifactory-originRepoType=local&X-Artifactory-originRepositoryKey=cassandra-rpm&x-jf-traceId=ae53672e8ee87d1d&response-content-disposition=attachment%3Bfilename%3D%22cassandra-4.1.0-1.noarch.rpm%22&response-content-type=application%2Fx-rpm&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEN%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJIMEYCIQC6DjEe%2Fw49YcU%2F9YJe6ZnyEmF0xHm9UBA0Z9pqRh9S%2FwIhALmZjh2yzqq8apsTQAv9mHngWesA3M7wm1TqCaMOwbedKoIFCBgQAhoMOTk5OTM0NzkwNDE0Igw%2BqQATIuO5eUYS2rwq3wSrUTYSVgSmib2bM3ttqIWb8v%2FUZnuU8iOo7ZuJlttYaFKuWlDLp5D3ANTOHfsdVHL2g%2FU4cC7sCK%2FbEhcK0rjIoldE5NQjMpghzvkdRk3q1gcLJoDONNNZ9D2ggg%2FE0qL8DHihEQeFkf%2FaNhs9wU6zkAqRHsTpq5rV1pFpOF3aQQXolRp6hhOySIzBqY0eb0%2Bl9F%2BE47rxC3bbtJjfRRztM4AeVaVSPh5a%2FSCWXkZRdj9VFlUu6%2F89uO9C6APKcJc8OjRSYAkJ8VCqtbM15ZZqn%2FvdhohMEAjgNmLBi82psGpMJCtcLealFLADbo9THCWS4bA%2F%2BRA0IYhO%2BTO6waHjwUQ94c3F%2FJ2FgLU745%2FzA4KWiCREQQt5uhfvglwDvQs3C8QMslAmcUaiMRBkP5F2P11%2FxoKY5mMZVRfvsEmW9UY%2F0eTjcV8kyWXD91wJkTUxXyqYhkg23Y21YN6zdM1jrHsgOGGW%2BR53LRhRWNkyoR5QBMTGcdWzPllERC7pVNVKL2j%2Fg%2FFdpjkGlVIGnVrx1ixEdOJAwepV2RLK0e0Dvnw06Db%2BS9n32ILYx4wv3iTlIrYrwg12MzjH%2B5RlvHTeHDp7X%2Bg2j1VUxGUlUu1dLDXMaLANVCBApZIaTpgSd6lsfbJ2a1gFexx7suXrpKFN3Lgb0xzr6guPUd41W6nWa0jaEMO%2BUX1Mwff76Atwa54c%2BE%2FluDjXv42yCkCPuDXwlt%2FXGby9SQzAoiAX5nrAwKPCdwya50%2FD3smBkjQBqaPioMzG%2FoLu7%2FNJgHCqzMtatQ9dBikE20BwpCgvPbTZMKKQobcGOpkB4Po52wZ25VaeBP95c4etBgT4UmS9l1nYCiUt%2F0gqpQwlsUMMdcNEAIgF22PC1pjXVFK4UxVCkXjRX7FI5P4G0uFGd16JsqBgHlSAoURafimDaOrHTYT%2FBWHuBvXQEFHy36ueq3IEslQ%2F88zNDzbquH7xjy7kJGGfDkKuqAk2f3%2FZk0PifKxxq1cmXgMgQDaFTm%2BqmqC10EK0&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20240916T154305Z&X-Amz-SignedHeaders=host&X-Amz-Expires=600&X-Amz-Credential=ASIA6RUGCBMHOPFV7DHC%2F20240916%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=e593d0c7bb16c610b0aef3c4643667872225a64531c95dae4d74b141585620fe
Resolving jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com (jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com)... 52.92.211.225, 52.92.165.65, 52.92.193.113, ...
Connecting to jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com (jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com)|52.92.211.225|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 50792764 (48M) [application/x-rpm]
Saving to: ‘cassandra-4.1.0-1.noarch.rpm’
cassandra-4.1.0-1.noarch.rpm 100%[=========================================================================>] 48.44M 7.56MB/s in 16s
2024-09-16 11:43:22 (3.10 MB/s) - ‘cassandra-4.1.0-1.noarch.rpm’ saved [50792764/50792764]
+ sudo yum install -y cassandra-4.1.0-1.noarch.rpm
Last metadata expiration check: 1:57:37 ago on Mon 16 Sep 2024 09:45:45 AM EDT.
Dependencies resolved.
========================================================================================================================================================
Package Architecture Version Repository Size
========================================================================================================================================================
Upgrading:
cassandra noarch 4.1.0-1 @commandline 48 M
Transaction Summary
========================================================================================================================================================
Upgrade 1 Package
Total size: 48 M
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: cassandra-4.1.0-1.noarch 1/2
Upgrading : cassandra-4.1.0-1.noarch 1/2
warning: /etc/cassandra/default.conf/cassandra.yaml created as /etc/cassandra/default.conf/cassandra.yaml.rpmnew
Running scriptlet: cassandra-4.1.0-1.noarch 1/2
Running scriptlet: cassandra-3.11.13-1.noarch 2/2
Cleanup : cassandra-3.11.13-1.noarch 2/2
Running scriptlet: cassandra-3.11.13-1.noarch 2/2
Verifying : cassandra-4.1.0-1.noarch 1/2
Verifying : cassandra-3.11.13-1.noarch 2/2
Upgraded:
cassandra-4.1.0-1.noarch
Complete!
+ [[ true == \t\r\u\e ]]
+ sudo mv /etc/cassandra/default.conf/cassandra.yaml /etc/cassandra/default.conf/cassandra.yaml.bak
+ sudo cp /etc/cassandra/default.conf/cassandra.yaml.rpmnew /etc/cassandra/default.conf/cassandra.yaml
+ sudo sed -i 's/cluster_name: '\''Test Cluster'\''/cluster_name: '\''SC35SP2TO36SP1 Cluster'\''/;s/num_tokens: 16/num_tokens: 256/;s/authenticator: AllowAllAuthenticator/authenticator: PasswordAuthenticator/;s/authorizer: AllowAllAuthorizer/authorizer: CassandraAuthorizer/;s/write_request_timeout_in_ms: 2000/write_request_timeout_in_ms: 20000/;s/batch_size_fail_threshold_in_kb: 50/batch_size_fail_threshold_in_kb: 300/;s/sasi_indexes_enabled: false/sasi_indexes_enabled: true/' /etc/cassandra/default.conf/cassandra.yaml
+ [[ Linux =~ CYGWIN ]]
+ sudo cp syndeia-cloud-3.6-SP1.2024-09-11_cassandra_zookeeper_kafka_setup/conf/tmpfiles.d/cassandra.conf /etc/tmpfiles.d/.
+ sudo systemd-tmpfiles --create --boot /etc/tmpfiles.d/cassandra.conf
+ sudo cp syndeia-cloud-3.6-SP1.2024-09-11_cassandra_zookeeper_kafka_setup/conf/init/systemd/cassandra.service /etc/systemd/system/.
+ sudo systemctl daemon-reload
+ sudo systemctl enable cassandra
Synchronizing state of cassandra.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable cassandra
+ sudo systemctl start cassandra
Verification:
Verify Cassandra is up and functioning by running
nodetool status:
If you are on Linux:nodetool status
You should get output similar to the following:
Datacenter: dc1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.1.3 644.57 KiB 256 100.0% 141c21db-4f79-476b-b818-ee6d2da16d7d rack1
Deploy Apache Zookeeper (ZK)
Run the Apache Zookeeper (ZK) pre-setup script:
./syndeia-cloud-3.6_zookeeper_pre-setup.bash
syndeia-cloud-3.6_zookeeper_pre-setup.bash sample output
[jdoe@new-server bin]$ ./syndeia-cloud-3.6_zookeeper_pre-setup.bash + shopt -s extglob + shopt -s globstar ++ readlink -f ./syndeia-cloud-3.6_zookeeper_pre-setup.bash + SC_version_parsed=/home/jdoe/syndeia-cloud-3.6-SP1.2024-09-02_103957_cassandra_zookeeper_kafka_setup/bin/syndeia-cloud-3.6_zookeeper_pre-setup.bash + SC_version_parsed=/home/jdoe/syndeia-cloud-3.6-SP1.2024-09-02_103957 + SC_version_parsed=3.6-SP1.2024-09-02_103957 + SC_version=3.6-SP1.2024-09-02_103957 + service_name=zookeeper + ZK_build_ver=3.8.4 ++ uname + OS=Linux + [[ Linux =~ CYGWIN ]] + systemctl=systemctl + sudo=sudo + ln='ln -nfs' + ZK_upgrade=false + [[ -e /opt/zookeeper-current ]] + umask 0002 + pushd . ~/syndeia-cloud-3.6-SP1.2024-09-02_103957_cassandra_zookeeper_kafka_setup/bin ~/syndeia-cloud-3.6-SP1.2024-09-02_103957_cassandra_zookeeper_kafka_setup/bin + cd /home/jdoe/ + [[ '' == \-\d ]] + [[ Linux =~ CYGWIN ]] + [[ -e /etc/systemd/system/zookeeper.service ]] + [[ -e /opt/zookeeper-3.8.4 ]] + [[ ! -e zookeeper-3.8.4-bin.tar.gz ]] + wget https://archive.apache.org/dist/zookeeper/zookeeper-3.8.4/apache-zookeeper-3.8.4-bin.tar.gz -O zookeeper-3.8.4-bin.tar.gz --2024-09-03 19:16:44-- https://archive.apache.org/dist/zookeeper/zookeeper-3.8.4/apache-zookeeper-3.8.4-bin.tar.gz Resolving archive.apache.org (archive.apache.org)... 2a01:4f9:1a:a084::2, 65.108.204.189 Connecting to archive.apache.org (archive.apache.org)|2a01:4f9:1a:a084::2|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 14609453 (14M) [application/x-gzip] Saving to: ‘zookeeper-3.8.4-bin.tar.gz’ zookeeper-3.8.4-bin.tar.gz 100%[==================================================================================================================================>] 13.93M 9.78MB/s in 1.4s 2024-09-03 19:16:46 (9.78 MB/s) - ‘zookeeper-3.8.4-bin.tar.gz’ saved [14609453/14609453] ++ which tar ++ alias ++ eval declare -f +++ declare -f ++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot tar /usr/bin/which: no tar in (/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) + [[ ! -n '' ]] + sudo yum install -y tar Updating Subscription Management repositories. Last metadata expiration check: 0:03:27 ago on Tue 03 Sep 2024 07:13:23 PM EDT. Dependencies resolved. ==================================================================================================================================================================================================================================== Package Architecture Version Repository Size ==================================================================================================================================================================================================================================== Installing: tar x86_64 2:1.30-9.el8 rhel-8-for-x86_64-baseos-rpms 839 k Transaction Summary ==================================================================================================================================================================================================================================== Install 1 Package Total download size: 839 k Installed size: 2.7 M Downloading Packages: tar-1.30-9.el8.x86_64.rpm 4.8 MB/s | 839 kB 00:00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 4.8 MB/s | 839 kB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : tar-2:1.30-9.el8.x86_64 1/1 Running scriptlet: tar-2:1.30-9.el8.x86_64 1/1 Verifying : tar-2:1.30-9.el8.x86_64 1/1 Installed products updated. Installed: tar-2:1.30-9.el8.x86_64 Complete! + sudo tar -xzf zookeeper-3.8.4-bin.tar.gz -C /opt/ + sudo mv /opt/apache-zookeeper-3.8.4-bin /opt/zookeeper-3.8.4 + [[ false == \f\a\l\s\e ]] + sudo mkdir -p /opt/zookeeper-3.8.4/logs /var/lib/zookeeper + sudo ln -nfs /opt/zookeeper-3.8.4 /opt/zookeeper-current + pushd . ~ ~ ~/syndeia-cloud-3.6-SP1.2024-09-02_103957_cassandra_zookeeper_kafka_setup/bin + cd /opt/zookeeper-3.8.4/logs/ + sudo ln -nfs ./zookeeper-zookeeper-server-syndeia36-autotest.intercax.com.log zookeeper.log + popd ~ ~/syndeia-cloud-3.6-SP1.2024-09-02_103957_cassandra_zookeeper_kafka_setup/bin + sudo groupadd --system kafka-zookeeper + sudo chown -R zookeeper:kafka-zookeeper /opt/zookeeper-3.8.4 /opt/zookeeper-current /var/lib/zookeeper + init_zoo_config + sudo mkdir -p /etc/zookeeper/conf/ + echo 1 + sudo tee /etc/zookeeper/conf/myid + sudo tee /etc/zookeeper/conf/zoo.cfg + [[ false == \t\r\u\e ]] + sudo cp syndeia-cloud-3.6-SP1.2024-09-02_103957_cassandra_zookeeper_kafka_setup/conf/init/systemd/zookeeper.service /etc/systemd/system/. + sudo systemctl daemon-reload + sudo systemctl enable zookeeper Created symlink /etc/systemd/system/multi-user.target.wants/zookeeper.service → /etc/systemd/system/zookeeper.service. + sudo systemctl start zookeeper + printf ruok ./syndeia-cloud-3.6_zookeeper_pre-setup.bash: connect: Connection refused ./syndeia-cloud-3.6_zookeeper_pre-setup.bash: line 224: /dev/tcp/localhost/2181: Connection refused + sleep 2 + printf ruok + printf '\n' + [[ false == \t\r\u\e ]]
Verification
Verify Zookeeper is up and functioning by running
zkCli.sh:sudo -u zookeeper /opt/zookeeper-current/bin/zkCli.sh -server localhost:2181
You should get output similar to the following:
Connecting to localhost:2181 Welcome to ZooKeeper! JLine support is enabled WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: localhost:2181(CONNECTED) 0]
Deploy Apache Kafka
Run the Apache Kafka pre-setup script. This will download Kafka 3.7.0 and install it, or will install an already downloaded Kafka 3.7.0 :
./syndeia-cloud-3.6_kafka_pre-setup.bash
syndeia-cloud-3.6_kafka_pre-setup.bash sample output
[jdoe@new-server bin]$ ./syndeia-cloud-3.6_kafka_pre-setup.bash + shopt -s extglob + shopt -s globstar ++ readlink -f ./syndeia-cloud-3.6_kafka_pre-setup.bash + SC_version_parsed=/home/icxadmin/syndeia-cloud-3.6-SP1.2024-09-02_103957_cassandra_zookeeper_kafka_setup/bin/syndeia-cloud-3.6_kafka_pre-setup.bash + SC_version_parsed=/home/icxadmin/syndeia-cloud-3.6-SP1.2024-09-02_103957 + SC_version_parsed=3.6-SP1.2024-09-02_103957 + SC_version=3.6-SP1.2024-09-02_103957 + service_name=kafka + Scala_ver=2.13 + Kafka_build_ver=3.7.0 ++ uname + OS=Linux + [[ Linux =~ CYGWIN ]] + systemctl=systemctl + sudo=sudo + ln='ln -nfs' + Kafka_upgrade=false + [[ -e /opt/kafka-current ]] + umask 0002 + pushd . ~/syndeia-cloud-3.6-SP1.2024-09-02_103957_cassandra_zookeeper_kafka_setup/bin ~/syndeia-cloud-3.6-SP1.2024-09-02_103957_cassandra_zookeeper_kafka_setup/bin + cd /home/icxadmin/ + [[ '' == \-\d ]] + [[ Linux =~ CYGWIN ]] + [[ false == \t\r\u\e ]] + [[ -e /opt/kafka- ]] + [[ ! -e kafka_2.13-3.7.0.tgz ]] + wget https://archive.apache.org/dist/kafka/3.7.0/kafka_2.13-3.7.0.tgz --2024-09-03 19:19:11-- https://archive.apache.org/dist/kafka/3.7.0/kafka_2.13-3.7.0.tgz Resolving archive.apache.org (archive.apache.org)... 2a01:4f9:1a:a084::2, 65.108.204.189 Connecting to archive.apache.org (archive.apache.org)|2a01:4f9:1a:a084::2|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 119028138 (114M) [application/x-gzip] Saving to: ‘kafka_2.13-3.7.0.tgz’ kafka_2.13-3.7.0.tgz 100%[==================================================================================================================================>] 113.51M 8.50MB/s in 9.0s 2024-09-03 19:19:21 (12.6 MB/s) - ‘kafka_2.13-3.7.0.tgz’ saved [119028138/119028138] ++ which tar ++ alias ++ eval declare -f +++ declare -f ++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot tar + [[ ! -n /usr/bin/tar ]] + sudo tar -xvzf kafka_2.13-3.7.0.tgz -C /opt/ kafka_2.13-3.7.0/ kafka_2.13-3.7.0/LICENSE kafka_2.13-3.7.0/NOTICE kafka_2.13-3.7.0/bin/ kafka_2.13-3.7.0/bin/kafka-delete-records.sh kafka_2.13-3.7.0/bin/trogdor.sh kafka_2.13-3.7.0/bin/kafka-jmx.sh kafka_2.13-3.7.0/bin/connect-mirror-maker.sh kafka_2.13-3.7.0/bin/kafka-console-consumer.sh kafka_2.13-3.7.0/bin/kafka-consumer-perf-test.sh kafka_2.13-3.7.0/bin/kafka-log-dirs.sh kafka_2.13-3.7.0/bin/kafka-metadata-quorum.sh kafka_2.13-3.7.0/bin/zookeeper-server-stop.sh kafka_2.13-3.7.0/bin/kafka-verifiable-consumer.sh kafka_2.13-3.7.0/bin/kafka-features.sh kafka_2.13-3.7.0/bin/kafka-acls.sh kafka_2.13-3.7.0/bin/zookeeper-server-start.sh kafka_2.13-3.7.0/bin/kafka-server-stop.sh kafka_2.13-3.7.0/bin/kafka-configs.sh kafka_2.13-3.7.0/bin/kafka-reassign-partitions.sh kafka_2.13-3.7.0/bin/connect-plugin-path.sh kafka_2.13-3.7.0/bin/kafka-leader-election.sh kafka_2.13-3.7.0/bin/kafka-producer-perf-test.sh kafka_2.13-3.7.0/bin/kafka-transactions.sh kafka_2.13-3.7.0/bin/kafka-topics.sh kafka_2.13-3.7.0/bin/connect-standalone.sh kafka_2.13-3.7.0/bin/kafka-e2e-latency.sh kafka_2.13-3.7.0/bin/kafka-metadata-shell.sh kafka_2.13-3.7.0/bin/kafka-get-offsets.sh kafka_2.13-3.7.0/bin/kafka-dump-log.sh kafka_2.13-3.7.0/bin/kafka-broker-api-versions.sh kafka_2.13-3.7.0/bin/kafka-consumer-groups.sh kafka_2.13-3.7.0/bin/connect-distributed.sh kafka_2.13-3.7.0/bin/kafka-delegation-tokens.sh kafka_2.13-3.7.0/bin/kafka-run-class.sh kafka_2.13-3.7.0/bin/kafka-replica-verification.sh kafka_2.13-3.7.0/bin/kafka-console-producer.sh kafka_2.13-3.7.0/bin/zookeeper-shell.sh kafka_2.13-3.7.0/bin/kafka-client-metrics.sh kafka_2.13-3.7.0/bin/kafka-cluster.sh kafka_2.13-3.7.0/bin/windows/ kafka_2.13-3.7.0/bin/windows/kafka-e2e-latency.bat kafka_2.13-3.7.0/bin/windows/kafka-log-dirs.bat kafka_2.13-3.7.0/bin/windows/zookeeper-server-stop.bat kafka_2.13-3.7.0/bin/windows/connect-distributed.bat kafka_2.13-3.7.0/bin/windows/kafka-configs.bat kafka_2.13-3.7.0/bin/windows/kafka-console-producer.bat kafka_2.13-3.7.0/bin/windows/kafka-delete-records.bat kafka_2.13-3.7.0/bin/windows/kafka-topics.bat kafka_2.13-3.7.0/bin/windows/kafka-dump-log.bat kafka_2.13-3.7.0/bin/windows/kafka-console-consumer.bat kafka_2.13-3.7.0/bin/windows/kafka-server-start.bat kafka_2.13-3.7.0/bin/windows/kafka-consumer-groups.bat kafka_2.13-3.7.0/bin/windows/kafka-mirror-maker.bat kafka_2.13-3.7.0/bin/windows/kafka-features.bat kafka_2.13-3.7.0/bin/windows/kafka-reassign-partitions.bat kafka_2.13-3.7.0/bin/windows/kafka-client-metrics.bat kafka_2.13-3.7.0/bin/windows/kafka-cluster.bat kafka_2.13-3.7.0/bin/windows/kafka-producer-perf-test.bat kafka_2.13-3.7.0/bin/windows/kafka-jmx.bat kafka_2.13-3.7.0/bin/windows/zookeeper-server-start.bat kafka_2.13-3.7.0/bin/windows/kafka-server-stop.bat kafka_2.13-3.7.0/bin/windows/kafka-replica-verification.bat kafka_2.13-3.7.0/bin/windows/kafka-run-class.bat kafka_2.13-3.7.0/bin/windows/kafka-acls.bat kafka_2.13-3.7.0/bin/windows/kafka-delegation-tokens.bat kafka_2.13-3.7.0/bin/windows/kafka-broker-api-versions.bat kafka_2.13-3.7.0/bin/windows/kafka-metadata-quorum.bat kafka_2.13-3.7.0/bin/windows/kafka-transactions.bat kafka_2.13-3.7.0/bin/windows/kafka-storage.bat kafka_2.13-3.7.0/bin/windows/kafka-leader-election.bat kafka_2.13-3.7.0/bin/windows/kafka-streams-application-reset.bat kafka_2.13-3.7.0/bin/windows/kafka-get-offsets.bat kafka_2.13-3.7.0/bin/windows/connect-plugin-path.bat kafka_2.13-3.7.0/bin/windows/zookeeper-shell.bat kafka_2.13-3.7.0/bin/windows/connect-standalone.bat kafka_2.13-3.7.0/bin/windows/kafka-consumer-perf-test.bat kafka_2.13-3.7.0/bin/kafka-verifiable-producer.sh kafka_2.13-3.7.0/bin/kafka-server-start.sh kafka_2.13-3.7.0/bin/kafka-mirror-maker.sh kafka_2.13-3.7.0/bin/kafka-storage.sh kafka_2.13-3.7.0/bin/kafka-streams-application-reset.sh kafka_2.13-3.7.0/bin/zookeeper-security-migration.sh kafka_2.13-3.7.0/config/ kafka_2.13-3.7.0/config/consumer.properties kafka_2.13-3.7.0/config/connect-mirror-maker.properties kafka_2.13-3.7.0/config/zookeeper.properties kafka_2.13-3.7.0/config/server.properties kafka_2.13-3.7.0/config/producer.properties kafka_2.13-3.7.0/config/trogdor.conf kafka_2.13-3.7.0/config/connect-console-sink.properties kafka_2.13-3.7.0/config/connect-log4j.properties kafka_2.13-3.7.0/config/connect-standalone.properties kafka_2.13-3.7.0/config/connect-file-source.properties kafka_2.13-3.7.0/config/connect-console-source.properties kafka_2.13-3.7.0/config/connect-distributed.properties kafka_2.13-3.7.0/config/tools-log4j.properties kafka_2.13-3.7.0/config/connect-file-sink.properties kafka_2.13-3.7.0/config/kraft/ kafka_2.13-3.7.0/config/kraft/server.properties kafka_2.13-3.7.0/config/kraft/broker.properties kafka_2.13-3.7.0/config/kraft/controller.properties kafka_2.13-3.7.0/config/log4j.properties kafka_2.13-3.7.0/licenses/ kafka_2.13-3.7.0/licenses/pcollections-MIT kafka_2.13-3.7.0/licenses/jopt-simple-MIT kafka_2.13-3.7.0/licenses/protobuf-java-BSD-3-clause kafka_2.13-3.7.0/licenses/DWTFYWTPL kafka_2.13-3.7.0/licenses/eclipse-distribution-license-1.0 kafka_2.13-3.7.0/licenses/slf4j-MIT kafka_2.13-3.7.0/licenses/jline-BSD-3-clause kafka_2.13-3.7.0/licenses/jsr305-BSD-3-clause kafka_2.13-3.7.0/licenses/eclipse-public-license-2.0 kafka_2.13-3.7.0/licenses/paranamer-BSD-3-clause kafka_2.13-3.7.0/licenses/argparse-MIT kafka_2.13-3.7.0/licenses/zstd-jni-BSD-2-clause kafka_2.13-3.7.0/licenses/checker-qual-MIT kafka_2.13-3.7.0/licenses/CDDL+GPL-1.1 kafka_2.13-3.7.0/libs/ kafka_2.13-3.7.0/libs/kafka-group-coordinator-3.7.0.jar kafka_2.13-3.7.0/libs/kafka-metadata-3.7.0.jar kafka_2.13-3.7.0/libs/kafka-storage-3.7.0.jar kafka_2.13-3.7.0/libs/kafka-storage-api-3.7.0.jar kafka_2.13-3.7.0/libs/kafka-raft-3.7.0.jar kafka_2.13-3.7.0/libs/kafka-server-3.7.0.jar kafka_2.13-3.7.0/libs/kafka-server-common-3.7.0.jar kafka_2.13-3.7.0/libs/kafka-tools-api-3.7.0.jar kafka_2.13-3.7.0/libs/kafka-clients-3.7.0.jar kafka_2.13-3.7.0/libs/jackson-dataformat-csv-2.16.0.jar kafka_2.13-3.7.0/libs/jackson-datatype-jdk8-2.16.0.jar kafka_2.13-3.7.0/libs/jackson-databind-2.16.0.jar kafka_2.13-3.7.0/libs/jackson-annotations-2.16.0.jar kafka_2.13-3.7.0/libs/jackson-core-2.16.0.jar kafka_2.13-3.7.0/libs/jackson-module-scala_2.13-2.16.0.jar kafka_2.13-3.7.0/libs/scala-collection-compat_2.13-2.10.0.jar kafka_2.13-3.7.0/libs/scala-java8-compat_2.13-1.0.2.jar kafka_2.13-3.7.0/libs/scala-logging_2.13-3.9.4.jar kafka_2.13-3.7.0/libs/scala-reflect-2.13.12.jar kafka_2.13-3.7.0/libs/scala-library-2.13.12.jar kafka_2.13-3.7.0/libs/argparse4j-0.7.0.jar kafka_2.13-3.7.0/libs/commons-validator-1.7.jar kafka_2.13-3.7.0/libs/jopt-simple-5.0.4.jar kafka_2.13-3.7.0/libs/jose4j-0.9.4.jar kafka_2.13-3.7.0/libs/metrics-core-2.2.0.jar kafka_2.13-3.7.0/libs/metrics-core-4.1.12.1.jar kafka_2.13-3.7.0/libs/zookeeper-3.8.3.jar kafka_2.13-3.7.0/libs/slf4j-api-1.7.36.jar kafka_2.13-3.7.0/libs/commons-cli-1.4.jar kafka_2.13-3.7.0/libs/commons-beanutils-1.9.4.jar kafka_2.13-3.7.0/libs/commons-logging-1.2.jar kafka_2.13-3.7.0/libs/commons-collections-3.2.2.jar kafka_2.13-3.7.0/libs/commons-digester-2.1.jar kafka_2.13-3.7.0/libs/paranamer-2.8.jar kafka_2.13-3.7.0/libs/zookeeper-jute-3.8.3.jar kafka_2.13-3.7.0/libs/audience-annotations-0.12.0.jar kafka_2.13-3.7.0/libs/netty-handler-4.1.100.Final.jar kafka_2.13-3.7.0/libs/netty-transport-native-epoll-4.1.100.Final.jar kafka_2.13-3.7.0/libs/netty-transport-classes-epoll-4.1.100.Final.jar kafka_2.13-3.7.0/libs/netty-transport-native-unix-common-4.1.100.Final.jar kafka_2.13-3.7.0/libs/netty-codec-4.1.100.Final.jar kafka_2.13-3.7.0/libs/netty-transport-4.1.100.Final.jar kafka_2.13-3.7.0/libs/netty-resolver-4.1.100.Final.jar kafka_2.13-3.7.0/libs/netty-buffer-4.1.100.Final.jar kafka_2.13-3.7.0/libs/netty-common-4.1.100.Final.jar kafka_2.13-3.7.0/libs/commons-io-2.11.0.jar kafka_2.13-3.7.0/libs/zstd-jni-1.5.5-6.jar kafka_2.13-3.7.0/libs/lz4-java-1.8.0.jar kafka_2.13-3.7.0/libs/snappy-java-1.1.10.5.jar kafka_2.13-3.7.0/libs/opentelemetry-proto-1.0.0-alpha.jar kafka_2.13-3.7.0/libs/pcollections-4.0.1.jar kafka_2.13-3.7.0/libs/caffeine-2.9.3.jar kafka_2.13-3.7.0/libs/protobuf-java-3.23.4.jar kafka_2.13-3.7.0/libs/checker-qual-3.19.0.jar kafka_2.13-3.7.0/libs/error_prone_annotations-2.10.0.jar kafka_2.13-3.7.0/libs/kafka_2.13-3.7.0.jar kafka_2.13-3.7.0/site-docs/ kafka_2.13-3.7.0/site-docs/kafka_2.13-3.7.0-site-docs.tgz kafka_2.13-3.7.0/libs/kafka-tools-3.7.0.jar kafka_2.13-3.7.0/libs/connect-runtime-3.7.0.jar kafka_2.13-3.7.0/libs/kafka-log4j-appender-3.7.0.jar kafka_2.13-3.7.0/libs/connect-json-3.7.0.jar kafka_2.13-3.7.0/libs/connect-transforms-3.7.0.jar kafka_2.13-3.7.0/libs/connect-api-3.7.0.jar kafka_2.13-3.7.0/libs/reflections-0.10.2.jar kafka_2.13-3.7.0/libs/slf4j-reload4j-1.7.36.jar kafka_2.13-3.7.0/libs/reload4j-1.2.25.jar kafka_2.13-3.7.0/libs/jackson-jaxrs-base-2.16.0.jar kafka_2.13-3.7.0/libs/jackson-module-jaxb-annotations-2.16.0.jar kafka_2.13-3.7.0/libs/jackson-jaxrs-json-provider-2.16.0.jar kafka_2.13-3.7.0/libs/jakarta.xml.bind-api-2.3.3.jar kafka_2.13-3.7.0/libs/jakarta.activation-api-1.2.2.jar kafka_2.13-3.7.0/libs/jersey-container-servlet-2.39.1.jar kafka_2.13-3.7.0/libs/jersey-hk2-2.39.1.jar kafka_2.13-3.7.0/libs/jaxb-api-2.3.1.jar kafka_2.13-3.7.0/libs/activation-1.1.1.jar kafka_2.13-3.7.0/libs/jetty-servlet-9.4.53.v20231009.jar kafka_2.13-3.7.0/libs/jetty-security-9.4.53.v20231009.jar kafka_2.13-3.7.0/libs/jetty-server-9.4.53.v20231009.jar kafka_2.13-3.7.0/libs/jetty-servlets-9.4.53.v20231009.jar kafka_2.13-3.7.0/libs/jetty-client-9.4.53.v20231009.jar kafka_2.13-3.7.0/libs/maven-artifact-3.8.8.jar kafka_2.13-3.7.0/libs/swagger-annotations-2.2.8.jar kafka_2.13-3.7.0/libs/javax.ws.rs-api-2.1.1.jar kafka_2.13-3.7.0/libs/jersey-container-servlet-core-2.39.1.jar kafka_2.13-3.7.0/libs/jersey-server-2.39.1.jar kafka_2.13-3.7.0/libs/jersey-client-2.39.1.jar kafka_2.13-3.7.0/libs/jersey-common-2.39.1.jar kafka_2.13-3.7.0/libs/jakarta.ws.rs-api-2.1.6.jar kafka_2.13-3.7.0/libs/hk2-locator-2.6.1.jar kafka_2.13-3.7.0/libs/javassist-3.29.2-GA.jar kafka_2.13-3.7.0/libs/javax.activation-api-1.2.0.jar kafka_2.13-3.7.0/libs/javax.servlet-api-3.1.0.jar kafka_2.13-3.7.0/libs/jetty-http-9.4.53.v20231009.jar kafka_2.13-3.7.0/libs/jetty-io-9.4.53.v20231009.jar kafka_2.13-3.7.0/libs/jetty-util-ajax-9.4.53.v20231009.jar kafka_2.13-3.7.0/libs/jetty-continuation-9.4.53.v20231009.jar kafka_2.13-3.7.0/libs/jetty-util-9.4.53.v20231009.jar kafka_2.13-3.7.0/libs/jsr305-3.0.2.jar kafka_2.13-3.7.0/libs/plexus-utils-3.3.1.jar kafka_2.13-3.7.0/libs/commons-lang3-3.8.1.jar kafka_2.13-3.7.0/libs/hk2-api-2.6.1.jar kafka_2.13-3.7.0/libs/hk2-utils-2.6.1.jar kafka_2.13-3.7.0/libs/jakarta.inject-2.6.1.jar kafka_2.13-3.7.0/libs/jakarta.annotation-api-1.3.5.jar kafka_2.13-3.7.0/libs/osgi-resource-locator-1.0.3.jar kafka_2.13-3.7.0/libs/jakarta.validation-api-2.0.2.jar kafka_2.13-3.7.0/libs/aopalliance-repackaged-2.6.1.jar kafka_2.13-3.7.0/libs/trogdor-3.7.0.jar kafka_2.13-3.7.0/libs/kafka-shell-3.7.0.jar kafka_2.13-3.7.0/libs/jline-3.22.0.jar kafka_2.13-3.7.0/libs/connect-file-3.7.0.jar kafka_2.13-3.7.0/libs/connect-basic-auth-extension-3.7.0.jar kafka_2.13-3.7.0/libs/javax.annotation-api-1.3.2.jar kafka_2.13-3.7.0/libs/connect-mirror-3.7.0.jar kafka_2.13-3.7.0/libs/connect-mirror-client-3.7.0.jar kafka_2.13-3.7.0/libs/kafka-streams-3.7.0.jar kafka_2.13-3.7.0/libs/rocksdbjni-7.9.2.jar kafka_2.13-3.7.0/libs/kafka-streams-scala_2.13-3.7.0.jar kafka_2.13-3.7.0/libs/kafka-streams-test-utils-3.7.0.jar kafka_2.13-3.7.0/libs/kafka-streams-examples-3.7.0.jar + sudo groupadd --system kafka-zookeeper groupadd: group 'kafka-zookeeper' already exists + sudo useradd --system --groups kafka-zookeeper kafka + [[ false == \t\r\u\e ]] + sudo sed -i s#log.dirs=/tmp/kafka-logs#log.dirs=/opt/kafka-current/logs# /opt/kafka_2.13-3.7.0/config/server.properties + sudo chown -R kafka:kafka-zookeeper /opt/kafka_2.13-3.7.0 + sudo ln -nfs /opt/kafka_2.13-3.7.0 /opt/kafka-current + sudo chown -h kafka:kafka-zookeeper /opt/kafka-current + sudo cp syndeia-cloud-3.6-SP1.2024-09-02_103957_cassandra_zookeeper_kafka_setup/conf/init/systemd/kafka.service /etc/systemd/system/. + sudo systemctl daemon-reload + sudo systemctl enable kafka Created symlink /etc/systemd/system/multi-user.target.wants/kafka.service → /etc/systemd/system/kafka.service. + sudo systemctl start kafka + [[ false == \t\r\u\e ]]
Verification
Verify Kafka is up and functioning by creating a test topic, producer with test events and a consumer that replays them:
/opt/kafka-current/bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
… producer:
/opt/kafka-current/bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
Test event1 Test event2
… consumer:
/opt/kafka-current/bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
You should see:
Test event1 Test event2
Deploy JanusGraph (JG)
cd to the
janusgaph_setuppackage’sbindirectory:cd ~/syndeia-cloud-3.6*_janusgraph_setup/bin
Run the JanusGraph (JG) pre-setup script:
./syndeia-cloud-3.6_janusgraph_pre-setup.bash
syndeia-cloud-3.6_janusgraph_pre-setup.bash sample output
[jdoe@new-server bin]$ ./syndeia-cloud-3.6_janusgraph_pre-setup.bash + shopt -s extglob + shopt -s globstar +++ dirname ./syndeia-cloud-3.6_janusgraph_pre-setup.bash ++ cd -P . ++ pwd + SCRIPT_DIR=/home/icxadmin/syndeia-cloud-3.6-SP1.2024-09-02_103957_janusgraph_setup/bin ++ readlink -f ./syndeia-cloud-3.6_janusgraph_pre-setup.bash + SC_version_parsed=/home/icxadmin/syndeia-cloud-3.6-SP1.2024-09-02_103957_janusgraph_setup/bin/syndeia-cloud-3.6_janusgraph_pre-setup.bash + SC_version_parsed=/home/icxadmin/syndeia-cloud-3.6-SP1.2024-09-02_103957 + SC_version_parsed=3.6-SP1.2024-09-02_103957 + SC_version=3.6-SP1.2024-09-02_103957 + service_name=janusgraph + JG_build_ver=1.0.0 ++ uname + OS=Linux + [[ Linux =~ CYGWIN ]] + systemctl=systemctl + sudo=sudo + ln='ln -nfs' + JG_upgrade=false + [[ -e /opt/janusgraph-current ]] + umask 0002 + pushd . ~/syndeia-cloud-3.6-SP1.2024-09-02_103957_janusgraph_setup/bin ~/syndeia-cloud-3.6-SP1.2024-09-02_103957_janusgraph_setup/bin + cd /home/icxadmin/ + [[ '' == \-\d ]] + [[ Linux =~ CYGWIN ]] + [[ -e /etc/systemd/system/janusgraph.service ]] + [[ ! -e janusgraph-full-1.0.0.zip ]] + wget https://github.com/JanusGraph/janusgraph/releases/download/v1.0.0/janusgraph-full-1.0.0.zip --2024-09-03 19:20:52-- https://github.com/JanusGraph/janusgraph/releases/download/v1.0.0/janusgraph-full-1.0.0.zip Resolving github.com (github.com)... 140.82.112.3 Connecting to github.com (github.com)|140.82.112.3|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/77385607/4db62a12-dfef-4589-adba-f1ed3564af56?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240903%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240903T232052Z&X-Amz-Expires=300&X-Amz-Signature=000fdb2c84bcc1085f00149666abf7fb29b8b897e5deda9ed9bd87864867a694&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=77385607&response-content-disposition=attachment%3B%20filename%3Djanusgraph-full-1.0.0.zip&response-content-type=application%2Foctet-stream [following] --2024-09-03 19:20:52-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/77385607/4db62a12-dfef-4589-adba-f1ed3564af56?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240903%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240903T232052Z&X-Amz-Expires=300&X-Amz-Signature=000fdb2c84bcc1085f00149666abf7fb29b8b897e5deda9ed9bd87864867a694&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=77385607&response-content-disposition=attachment%3B%20filename%3Djanusgraph-full-1.0.0.zip&response-content-type=application%2Foctet-stream Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.110.133, ... Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 743882909 (709M) [application/octet-stream] Saving to: ‘janusgraph-full-1.0.0.zip’ janusgraph-full-1.0.0.zip 100%[==================================================================================================================================>] 709.42M 48.6MB/s in 28s 2024-09-03 19:21:21 (25.5 MB/s) - ‘janusgraph-full-1.0.0.zip’ saved [743882909/743882909] ++ which unzip ++ alias ++ eval declare -f +++ declare -f ++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot unzip + [[ ! -n /usr/bin/unzip ]] + sudo bash -c 'umask 0022 && unzip -q janusgraph-full-1.0.0.zip -d /opt/' + sudo useradd --system --user-group janusgraph + [[ false == \t\r\u\e ]] + sudo chown -R janusgraph:janusgraph /opt/janusgraph-full-1.0.0 + sudo ln -nfs /opt/janusgraph-full-1.0.0 /opt/janusgraph-current + sudo cp syndeia-cloud-3.6-SP1.2024-09-02_103957_janusgraph_setup/conf/init/systemd/janusgraph.service /etc/systemd/system/. + sudo systemctl daemon-reload + sudo systemctl enable janusgraph Created symlink /etc/systemd/system/multi-user.target.wants/janusgraph.service → /etc/systemd/system/janusgraph.service. + [[ false == \t\r\u\e ]]
Run the (main) JanusGraph (JG) setup script, ie:
./syndeia-cloud-3.6_janusgraph_setup.bash
syndeia-cloud-3.6_janusgraph_setup.bash sample output
[jdoe@new-server bin]$ ./syndeia-cloud-3.6_janusgraph_setup.bash
PURPOSE: This script will configure JanusGraph for use with Syndeia Cloud 3.6 Specifically this will:
1. Create/update janusgraph-current symlink to specified version, default = 1.0.0
2. Create a new syndeia_admin superuser in Cassandra with a password you specify
3. Create a new syndeia_cloud_graph and syndeia_cloud_graph_config keyspaces
4. GRANT ALL PERMISSIONS ON KEYSPACE syndeia_cloud_graph TO syndeia_admin
5. GRANT ALL PERMISSIONS ON KEYSPACE syndeia_cloud_graph_config TO syndeia_admin
6. Create a renamed copy of the file /opt/janusgraph-<release_ver>/conf/janusgraph-cql-configurationgraph.properties as janusgraph-cql-configurationgraph-syndeia.properties with:
graph.graphname=syndeia_cloud_graph_config
Add storage.username=syndeia_admin
Add storage.password=<password_specified>
storage.hostname=<your_cassandra_jg_host>
6. If you installed Elasticsearch on the same machine, add index.search.backend=elasticsearch and index.search.hostname=localhost to use Elasticsearch for search indexing
7. Create a renamed copy of the file /opt/janusgraph-<release_ver>/conf/gremlin-server/gremlin-server-configuration.yaml as gremlin-server-configuration-syndeia.yaml and set ConfigurationManagementGraph to point to janusgraph-cql-configurationgraph-syndeia.properties.
9. Install systemd .service file for JanusGraph service
10. Start JanusGraph service
11. Run a Groovy JanusGraph setup script to set storage params for your graph and build indexes
REQUIREMENTS: BASH v4.4.12(1)+
+ Apache Cassandra 3.11.1+ installed and running
+ JanusGraph 1.0.0 installed
+ systemd 219+
+ systemd .service files installed for Cassandra
+ GNU sed 4.2.2+
GENERAL USAGE: syndeia-cloud-3.6_janusgraph_setup.bash [--JG_version=|-jg_v <JG_version>] [--JG_RF=|-jg_rf <JG_RF>] [--upgrade|-u] [--multi_node|-m] [--automated|-a]
... where:
[<JG_version>] = optional JanusGraph version, defaults to 1.0.0 if unspecified
[<JG_RF>] = Replication Factor for JanusGraph keyspaces, ie: # of nodes to have copies of, defaults to 1 if unspecified
[--upgrade|-u] = optional, default is to assume fresh install; if enabled, this will avoid re-initializing the syndeia_cloud_graph and syndeia_cloud_graph_config keyspaces
[--multi_node|-m] = optional, default is to assume single-node; if enabled, this will patch systemd .service files to not assume a single-node topology deployment
[--automated|-a] = optional, default is to run in (mostly) interactive mode; if enabled, this will run non-interactively and use exported environment vars instead, if any (see sc-jg_automated-settings-rc template file)
(i) Note: you may be prompted for sudo authentication, you will also be prompted for you Cassandra superuser password (to create and set your syndeia_admin password) and the FQDN of your Cassandra host.
(!) Avoid any of the following special characters: \?*[]+#&.{}$'" when setting your syndeia_admin password.
VERSIONS:
2019-06-06, BKM: v0.1: Initial
2019-12-10, BKM: v0.9: Small fixes
2019-12-10, BKM: v1.1: Update for SC 3.4
2020-01-28, BKM: v1.2: Fix for SYN-1202 (max frame length exceeded error)
2020-11-04, BKM: v1.3: Redefine cassandra_jg_host definition & prompt
2021-01-13, BKM: v1.4: Set SC_HOST in janusgraph.service as it now will wait for Cassandra DB to be up first
2021-05-27, BKM: v1.5: Configure remote.yaml also (since it is more heavily used, ex: in SC 3.4 migration + JG validation step)
2021-05-31, BKM: v1.6: Clarify options syntax in USAGE
2021-05-31, BKM: v1.7: Add --automated|-a and configure_elasticsearch options; change options processing into key-value params
2021-06-02, BKM: v1.8: SC 3.4 RC: final polishing
2022-02-07, BKM: v1.9: SC 3.5 M1: Update default JG version to 0.5.3
2022-04-18, BKM: v2.0: SC 3.5 M2+: Add upgrade logic to support upgrading from 3.4.X to 3.5 (since JG "must" be ugpraded due to security requirements)
2022-04-18, BKM: v2.1: SC 3.5 M2+: Add --multi-node|-m mode
2022-04-18, BKM: v2.2: SC 3.5 M2+: Make JG service start on boot
2022-04-28, BKM: v2.3: SC 3.5 M2+: SYN-2444: add timestamps to log output
2022-08-25, BKM: v2.4: SC 3.5: SYN-2856: poll role creation to resolve occasional race condition with syndeia_admin creation
2022-09-21, BKM: v2.5: SC 3.5: update default JG_HOME location
2023-03-10, BKM: v2.6: SC 3.6: update all version refs to 3.6
2024-01-31, BKM: v2.8: SC 3.6: SYN-4357: Update to support JG 1.0.0
[press any key to continue or hit ^C to cancel]
- Enter the current password for the cassandra admin (note, avoid any of the following special characters \?*[]+#&.{}$'" ): cassandra
- Enter a password to set for the syndeia_admin user (note, avoid any of the following special characters \?*[]+#&.{}$'" ): syn45ia
- Enter Cassandra host / FQDN, ie: localhost, cassandra.mydomain.com: localhost
- Enter JG host / FQDN, ie: localhost, 0.0.0.0, jg.mydomain.com: localhost
- Enter Janusgraph installed path for v1.0.0, ie: /opt/janusgraph-full-1.0.0: /opt/janusgraph-full-1.0.0
Tue Sep 3 19:22:10 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Symlinking /opt/janusgraph-full-1.0.0 as /opt/janusgraph-current...
Tue Sep 3 19:22:10 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Updating template schema and user generation .CQL with user provided values...
Tue Sep 3 19:22:10 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Now creating syndeia_admin user + syndeia_cloud_graph + syndeia_cloud_graph_config keyspaces...
Warning: Using a password on the command line interface can be insecure.
Recommendation: use the credentials file to securely provide the password.
Warning: Using a password on the command line interface can be insecure.
Recommendation: use the credentials file to securely provide the password.
- Query with filter met
Tue Sep 3 19:22:11 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Successfully ran verification of syndeia_admin user creation
Tue Sep 3 19:22:11 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Successfully created user
Warning: Using a password on the command line interface can be insecure.
Recommendation: use the credentials file to securely provide the password.
Warnings :
Role 'syndeia_admin' was already granted CREATE, ALTER, DROP, SELECT, MODIFY, AUTHORIZE on <keyspace syndeia_cloud_graph>
Warnings :
Role 'syndeia_admin' was already granted CREATE, ALTER, DROP, SELECT, MODIFY, AUTHORIZE on <keyspace syndeia_cloud_graph_config>
Tue Sep 3 19:22:12 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Successfully created keyspaces
- Do you have Elasticsearch installed on the same machine? [Y/N]:
1) Y
2) N
#? n
Tue Sep 3 19:22:16 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Install and configure Syndeia Cloud Janusgraph [upgrade] .groovy init script...
Tue Sep 3 19:22:16 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Create and configure Syndeia Cloud JanusGraph configurationgraph .properties file...
Tue Sep 3 19:22:16 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Create and configure Syndeia Cloud JanusGraph Gremlin server .yaml files...
Tue Sep 3 19:22:16 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Install janusgraph.service file...
Tue Sep 3 19:22:16 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Stopping janusgraph.service (this may take a while)...
Tue Sep 3 19:22:16 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Starting janusgraph.service...
Tue Sep 3 19:22:16 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:17 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:18 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:19 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:20 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:21 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:22 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:23 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:24 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:25 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:26 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:27 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:28 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:29 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:30 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:31 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:32 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:33 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
Tue Sep 3 19:22:34 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Waiting until janusgraph port is up...
- Enable janusgraph.service to start on boot...
Tue Sep 3 19:22:35 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Run Syndeia Cloud Janusgraph .groovy init script (ignore warning about not being able to load native-hadoop lib)...
Sep 03, 2024 7:22:36 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/janusgraph-full-1.0.0/lib/log4j-slf4j-impl-2.20.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/janusgraph-full-1.0.0/lib/logback-classic-1.2.11.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
19:22:39 INFO org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.getCustomizers - HADOOP_GREMLIN_LIBS is set to: /opt/janusgraph-full-1.0.0/lib
19:22:39 WARN org.apache.hadoop.util.NativeCodeLoader.<clinit> - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Sep 03, 2024 7:22:48 PM org.yaml.snakeyaml.internal.Logger warn
WARNING: Failed to find field for org.apache.tinkerpop.gremlin.driver.Settings.serializers
19:22:48 INFO org.apache.tinkerpop.gremlin.driver.Connection.<init> - Created new connection for ws://localhost:8182/gremlin
19:22:48 INFO org.apache.tinkerpop.gremlin.driver.ConnectionPool.<init> - Opening connection pool on Host{address=localhost/127.0.0.1:8182, hostUri=ws://localhost:8182/gremlin} with core size of 1
19:22:54 INFO org.apache.tinkerpop.gremlin.driver.ConnectionPool.closeAsync - Signalled closing of connection pool on Host{address=localhost/127.0.0.1:8182, hostUri=ws://localhost:8182/gremlin} with core size of 1
19:22:54 INFO org.apache.tinkerpop.gremlin.driver.Cluster.lambda$closeAsync$0 - Closed Cluster for hosts [localhost/127.0.0.1:8182]
Tue Sep 3 19:22:55 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359:
Tue Sep 3 19:22:55 EDT 2024 [info] ./syndeia-cloud-3.6_janusgraph_setup.bash, PID=16359: - Done!
~/syndeia-cloud-3.6-SP1.2024-09-02_103957_janusgraph_setup/bin
Note, the current password for the Cassandra admin when installed via the pre-setup script is the Cassandra project’s documented default, ie: cassandra, we recommend changing this via CQLSH once SC deployment has been successfully completed (see Appendix B2.11 for instructions on how to do this), especially if the Cassandra node will (eventually) be exposed in a multi-node topology.
Verification:
Verify JG is up and functioning by running the Gremlin client, ie:
/opt/janusgraph-current/bin/gremlin.sh
Then execute the following commands after it starts up:
:remote connect tinkerpop.server conf/remote.yaml session :remote console graph = ConfiguredGraphFactory.open('syndeia_cloud_graph'); // should return: ==>standardjanusgraph[cql:[cassandra.mydomain.com]] g = graph.traversal(); g.V(); g.E(); // The last 2 commands above should not return any results since the graph (syndeia_cloud_graph) is empty - no vertices or edges.
Deploy Syndeia Cloud (SC)
cd to the
syndeia-cloudpackage’sbindirectory:cd ~/syndeia-cloud-3.6/bin
Run the Syndeia Cloud (SC) pre-setup script:
./syndeia-cloud-3.6_install_pre-setup.bash
During execution, you will be prompted to set credentials for JMX monitoring. By default Syndeia Cloud has been configured with JMX enabled & the script will prompt to set a reader & read-write credentials. If you do not wish to use JMX, you can do so pre or post-SC setup via the steps in Appendix C3.6).
syndeia-cloud-3.6_install_pre-setup.bash sample output
[jdoe@new-server bin]$ ./syndeia-cloud-3.6_install_pre-setup.bash
+ shopt -s extglob
+ shopt -s globstar
++ readlink -f ./syndeia-cloud-3.6_install_pre-setup.bash
+ SC_version_parsed=/home/icxadmin/syndeia-cloud-3.6-SP1.2024-09-02_103957/bin/syndeia-cloud-3.6_install_pre-setup.bash
+ SC_version_parsed=/home/icxadmin/syndeia-cloud-3.6-SP1.2024-09-02_103957
+ SC_version_parsed=3.6-SP1.2024-09-02_103957
+ SC_version=3.6-SP1.2024-09-02_103957
+ SC_build_ver=3.6-SP1
++ uname
+ OS=Linux
+ [[ Linux =~ CYGWIN ]]
+ systemctl=systemctl
+ sudo=sudo
+ ln='ln -nfs'
+ [[ ! -v JAVA_HOME ]]
+++ which java
+++ alias
+++ eval declare -f
++++ declare -f
+++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot java
++ readlink -f /usr/bin/java
+ JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.23.0.9-3.el8.x86_64/bin/java
+ [[ Linux =~ CYGWIN ]]
+ JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.23.0.9-3.el8.x86_64
+ export JAVA_HOME
+ SC_upgrade=false
+ [[ -e /opt/icx/syndeia-cloud-current ]]
+ [[ Linux =~ CYGWIN ]]
+ sudo useradd --system --user-group syndeia-cloud
+ init_JMX_settings
+ [[ ! -d /usr/lib/jvm/java-11-openjdk-11.0.23.0.9-3.el8.x86_64/conf/management/ ]]
+ [[ false == \f\a\l\s\e ]]
+ JMX_monitoring_pw=
+ JMX_control_pw=
+ [[ -e /usr/lib/jvm/java-11-openjdk-11.0.23.0.9-3.el8.x86_64/conf/management/jmxremote.password.template ]]
+ sudo cp /usr/lib/jvm/java-11-openjdk-11.0.23.0.9-3.el8.x86_64/conf/management/jmxremote.password.template /usr/lib/jvm/java-11-openjdk-11.0.23.0.9-3.el8.x86_64/conf/management/jmxremote.password
+ read -e -p '- Enter a password for the monitoring JMX role (note, avoid any of the following special characters \?*[]+#&.{}$'\''" ): ' -i 'MYPw!' JMX_monitoring_pw
- Enter a password for the monitoring JMX role (note, avoid any of the following special characters \?*[]+#&.{}$'" ): MYPw!
+ echo ''
+ read -e -p '- Enter a password for the control (debug) JMX role (note, avoid any of the following special characters \?*[]+#&.{}$'\''" ): ' -i 'MyDebugPw!' JMX_control_pw
- Enter a password for the control (debug) JMX role (note, avoid any of the following special characters \?*[]+#&.{}$'" ): MyDebugPw!
+ echo ''
+ echo 'monitorRole MYPw!'
+ sudo tee -a /usr/lib/jvm/java-11-openjdk-11.0.23.0.9-3.el8.x86_64/conf/management/jmxremote.password
monitorRole MYPw!
+ echo 'controlRole MyDebugPw!'
+ sudo tee -a /usr/lib/jvm/java-11-openjdk-11.0.23.0.9-3.el8.x86_64/conf/management/jmxremote.password
controlRole MyDebugPw!
+ sudo egrep '^monitorRole |^controlRole ' /usr/lib/jvm/java-11-openjdk-11.0.23.0.9-3.el8.x86_64/conf/management/jmxremote.password
controlRole MyDebugPw!
+ [[ ! Linux =~ CYGWIN ]]
+ sudo chown syndeia-cloud:syndeia-cloud /usr/lib/jvm/java-11-openjdk-11.0.23.0.9-3.el8.x86_64/conf/management/jmxremote.password
+ sudo chown syndeia-cloud:syndeia-cloud /usr/lib/jvm/java-11-openjdk-11.0.23.0.9-3.el8.x86_64/conf/management/jmxremote.access
+ sudo chmod u+rw,u-x,go-rwx /usr/lib/jvm/java-11-openjdk-11.0.23.0.9-3.el8.x86_64/conf/management/jmxremote.password
+ umask 0002
+ sudo tee /etc/sudoers.d/syndeia-cloud
+ cd /home/icxadmin/
+ init_firewalld syndeia
+ service_definition=syndeia
++ which firewall-cmd
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot firewall-cmd
+ [[ -e /usr/bin/firewall-cmd ]]
+ sudo cp /home/icxadmin/syndeia-cloud-3.6-SP1.2024-09-02_103957/conf/firewalld/syndeia.xml /etc/firewalld/services/.
+ sudo systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2024-09-03 18:49:44 EDT; 34min ago
Docs: man:firewalld(1)
Main PID: 399 (firewalld)
Tasks: 2 (limit: 102192)
Memory: 55.6M
CGroup: /system.slice/firewalld.service
└─399 /usr/libexec/platform-python -s /usr/sbin/firewalld --nofork --nopid
Sep 03 18:49:44 base-image-RHEL.intercax.com systemd[1]: Starting firewalld - dynamic firewall daemon...
Sep 03 18:49:44 base-image-RHEL.intercax.com systemd[1]: Started firewalld - dynamic firewall daemon.
Sep 03 18:49:44 base-image-RHEL.intercax.com firewalld[399]: WARNING: AllowZoneDrifting is enabled. This is considered an insecure configuration option. It will be removed in a future release. Please consider disabling it now.
+ [[ 0 == 3 ]]
+ sudo firewall-cmd --reload
success
+ sudo firewall-cmd --add-service=syndeia --permanent
success
+ sudo firewall-cmd --reload
success
+ sudo firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: eth0
sources:
Run the (main) Syndeia Cloud (SC) setup script, ie:
./syndeia-cloud-3.6_install.bash
syndeia-cloud-3.6_install.bash sample output
[jdoe@new-server bin]$ ./syndeia-cloud-3.6_install.bash
OVERVIEW: This script will install, configure, and start all Syndeia Cloud (SC) 3.6 services for a single-node deployment (multi-node deployment will require some additional customization).
GENERAL USAGE: syndeia-cloud-3.6_install.bash [--kafka_version=|-k_v <KAFKA_version>] [--SC_version=|-SC_v <SC_version>] [--skip_devops|-s] [--upgrade[_new_conf]|-u[_nc]] [--multi_node|-m] [--super_user_pw=|-su_pw <super_user_pw>] [--automated|-a] [--verbose|-v] [--debug|-d]
... where:
[<KAFKA_version>] = optional KAFKA version, defaults to "current" if unspecified
[<SC_version>] = optional, Syndeia Cloud version, defaults to 3.6 if unspecified
[--upgrade|-u] = optional, default is to drop existing keyspaces, run devops actions at end, & archive Kafka logs; if enabled, this will keep existing keyspaces & omit the following: running super.user devops action + Kafka log archiving
[--upgrade_new_conf|-u_nc] = optional, same as --upgrade|-u except use new conf files and copy in old for manual merging, any 3rd-party libs files will also need to be manually reinstalled (NOTE, keystore will still be copied over if it exists but you will need to manually re-enable it in the web-gateway's application.conf)
[--skip_devops|-s] = optional, default is run devops actions; if enabled, this will skip running the devops actions at the end
[--multi_node|-m] = optional, default is to assume single-node; if enabled, this will patch systemd .service files to not assume a single-node topology deployment
[--super_user_pw|-su_pw] = optional, switch to specify the super.user password if it has been changed
[--automated|-a] = optional, default is to run in (mostly) interactive mode; if enabled, this will run non-interactively and use exported environment vars instead, if any (see sc_automated_settings-rc template file)
[--verbose] = optional, default is off, if enabled, zip extractions will list all files extracted
[--debug] = optional, default is off, if enabled, all services are launched via a screen session with multiple windows to tail the logs, also enables JMX (WARNING: requires first setting up JVM's JMX auth (see https://docs.oracle.com/en/java/javase/11/management/monitoring-and-management-using-jmx-technology.html#GUID-805517EC-2D33-4D61-81D8-4D0FA770D1B8)), JMX port #s listed under "JMX port definitions:"
In interactive mode you will be prompted for the following:
- your syndeia_admin password,
- Cassandra/JanusGraph (JG) hostname,
- Apache Kafka hostname,
- Apache Kafka installed path,
- Syndeia Cloud (SC) hostname
(i) Note: you may be also prompted for sudo authentication
DETAIL: Specifically this script will:
- install Syndeia Cloud 3.6 to /opt/icx/syndeia-cloud-3.6 (default),
- make syndeia-cloud:syndeia-cloud the owner,
- configure the application.conf files if necessary,
- generate a concatenated schema file,
- stop any existing Syndeia Cloud 3.6, 3.5, 3.4, 3.3, or 3.2 processes,
- update/create a "current" symlink to the installed version,
- stop the Janusgraph service,
- stop the Kafka service,
- archive any old Kafka logs,
- drop any old keyspaces & generate the new schema in the DB (Cassandra),
- start Kafka service,
- start Janusgraph service,
- install a tmpfiles.d .conf file for syndeia-cloud,
- install sudoers file for syndeia-cloud,
- install systemd .target and .service files for syndeia-cloud microservices
- start all Syndeia Cloud microservices,
- create the superuser account, and
- create some sample test data.
REQUIREMENTS: BASH v4.4.12(1)+
+ Apache Cassandra 3.11.13+ installed and running
+ JanusGraph 0.5.3 installed (WILL NOT WORK WITH 0.6+)
+ Apache Zookeeper 3.6.3+ + installed and running
+ Apache Kafka 2.13-3.2.1+ installed
+ systemd 219+
+ systemd .service files installed for {Cassandra,JanusGraph,Kafka}
+ GNU sed 4.2.2+
+ GNU tr 8.2.2+ (part of GNU coreutils)
+ Info-Zip unzip 6.0+
(+ GNU screen 4.03.01+ installed, only for debug mode)
VERSIONS:
2019-06-10, BKM: v0.1: Initial
2019-12-10, BKM: v0.9: Small fixes
2020-01-22, BKM: v1.1: Update for SC 3.4
2020-01-22, BKM: v1.2: Update for SC 3.4: add new services: confluence, github, twc(loud), wc
2020-08-17, BKM: v1.3: Update for SC 3.4: add new services: bitbucket, gitlab; re-number ports
2019-09-24, BKM: v1.4: Small updates (ported improvements from 3.3 script, fixed comments/messages)
2020-10-14, BKM: v1.5: Update for SC 3.4: add new services: aras; re-number ports; remove extraneous comments
2020-10-21, BKM: v1.6: Update for SC 3.4: add new services: sysmlv2
2020-10-28, BKM: v1.7: Update for SC 3.4: add new services: testrail
2020-11-19, BKM: v1.8: Update for SC 3.4: add new services: doors
2021-01-06, BKM: v1.9: Update for SC 3.4-v2: add --upgrade|-u option
2021-01-13, BKM: v2.0: sc-auth.service: sed replace localhost with what user specifies during install; upgrade: keep Kafka logs also
2021-01-13, BKM: v2.1: Update for SC 3.4: add new services: artifactory
2021-01-14, BKM: v2.2: Update for SC 3.4: add --skip_devops action option
2021-01-21, BKM: v2.3: Update for SC 3.4: chown /opt/icx vs just /opt/icx/syndeia-cloud-<version>, disable SC 3.2 service if exists; fix skip_devops check
2021-04-02, BKM: v2.4: Update for SC 3.4: add --upgrade_new_conf|-u_nc to not copy in old conf file settings
2021-04-02, BKM: v2.4: Update for SC 3.4: add -q to zip extractions by default, add --verbose|-v option (currently only makes zip more verbose, ie: remove -q, and silences grep confirmation output), make unzips more precise to avoid "cautions" also
2021-05-25, BKM: v2.5: Update for SC 3.4: Make "1.0-SNAPSHOT" a variable
2021-05-31, BKM: v2.6: Update for SC 3.4: add --automated|-a action option for running non-interactively + template for environment vars that can be set (see sc_automated_settings-rc); quote services wildcard: 'sc-*'
2021-09-29, BKM: v2.9: SC 3.4.1 (SP1): change hardcoded /opt/kafka-current to KAFKA_HOME (some customers have non-standard custom installation path requirements to /opt/local vs /opt)
2021-10-05, BKM: v3.0: SC 3.4.1 (SP1): SYN-2336, SYN-2330: extract out janusconfigurationString.yaml too, add support for dual/multi-node deployment topologies (--multi_node|-m): configure janusconfiguration{,String}.yaml with jg_host + syndeia_admin_pw + update installed .service & .target files
2021-10-06, BKM: v3.1: SC 3.4.1 (SP1): SYN-2345: fix --upgrade|-u as it doesn't properly copy over old .conf files when snapshot version changes
2021-10-14, BKM: v3.2: SC 3.4.1 (SP1): tweak all_files_exist() logic
2021-11-15, BKM: v3.3: SC 3.5: add PLAY_SECRET since Lagom v1.6.5 seems to now require setting it something more complicated than "whatever"
2022-01-11, BKM: v3.4: SC 3.5: SYN-2336, SYN-2330: add support for dual/multi-node deployment topologies (--multi_node|-m): configure janusconfiguration{,String}.yaml with jg_host + syndeia_admin_pw + update installed .service & .target files
2022-01-27, BKM: v3.5: SC 3.5: add new services: volta
2022-01-21, BKM: v3.6: SC 3.5: add SC_snapshot_version_old, defaults to 3.4-SP1, use in --upgrade|-u and --upgrade_new_conf|-u_nc mode (previous implementation was only useful for development testing)
2022-03-11, BKM: v3.7: SC 3.5: add new services: collaborator
2022-03-24, BKM: v3.8: SC 3.5: SYN-2449: switch to using build provided /conf/ files by default except for select services
2022-03-25, BKM: v3.9: SC 3.5: SYN-2550: SC: 1. setup doesn't take into account different SC_snapshot_versions when upgrading
2022-04-18, BKM: v3.10: SC 3.5: Make SC services start on boot
2022-04-28, BKM: v3.11: SC 3.5: SYN-2444: add timestamps to log output
2022-05-18, BKM: v3.12: SC 3.5: SYN-2470: add support for new 'restful' service
2022-06-21, BKM: v3.13: SC 3.5: Set SC_snapshot_version_old default to 3.4-SP3
2022-08-18, BKM: v3.14: SC 3.5: SYN-2444, SYN-2752: Remove unused social auth providers, remove existing closing brace before EOF, add new LDAP group settings, add new SAML2 settings
2022-08-23, BKM: v3.15: SC 3.5: SYN-2444, SYN-2752, SYN-2851: --upgrade|-u: Update web-gateway's application.conf settings to add all new services from 3.4-SP3, set PLAY_SECRET, update play.filters.headers.contentSecurityPolicy, add play.filters.headers.allowActionSpecificHeaders = true, make LDAP Group + SAML upgrade sections more idempotent
2022-08-24, BKM: v3.16: SC 3.5: SYN-2848, SYN-2850 (PR #743, PR #744): fresh install + --upgrade|-u: implement corresponding changes to setup new or upgrade LDAP(S) settings
2022-08-26, BKM: v3.17: SC 3.5: SYN-2855: --upgrade|-u || --upgrade_new_conf|-u_nc: make this now run setup action since 3.4 to 3.5 introduced new integrations
2022-08-31, BKM: v4.0: SC 3.5: SYN-2858: remove dead code dealing with unnecessary stopping/starting of JG service (legacy artifact from when JG setup was part of this setup)
2022-11-30, BKM: v4.1: SC 3.6: Bump up to 3.6
2022-11-09, BKM: v4.2: SC 3.6: SYN-2917: SC setup update for password to sc-graph's janusconfiguration.yaml and janusconfigurationString.yaml should be less fragile.
2023-03-20, BKM: v4.3: SC 3.6: Update more 3.5 refs to 3.6
2023-03-20, BKM: v4.3: SC 3.6: SYN-3762: add support for new (NON-RESTful) DSE3 service
2023-04-19, KP: v4.4: SC 3.6: SYN-3847: add support for new genesys service
2023-04-19, KP: v4.5: SC 3.6: SYN-3848: add support for new (NON-RESTful) teamcenter service
2023-06-06, BKM: v4.6: SC 3.6: SYN-3166: Advanced Topologies: add support where kafka may not be on localhost
2023-06-27, KP: v4.7: SC 3.6: SYN-3944: add support for new dscr service
2023-06-27, KP: v4.8: SC 3.6: SYN-4141: add support for new digital-thread service
2024-03-05, BKM: v4.9: SC 3.6: SYN-4259: add support for porting devops configuration when HTTPS is enabled
2024-08-23, BKM: v5.0: SC 3.6-SP1: SYN-4794: add support for porting 3rd-party libs during upgrades
[press any key to when ready or hit ^C to cancel]
- Enter password for the Cassandra syndeia_admin user created during JanusGraph setup: ***REDACTED***
- Enter Cassandra host / FQDN, ie: cassandra.mydomain.com, localhost, etc.: localhost
- Enter JanusGraph host / FQDN, ie: jg.mydomain.com, localhost, etc.: localhost
- Enter Apache Kafka host / FQDN, ie: kafka.mydomain.com, localhost, etc.: localhost
- Enter Apache Kafka installed path, ie: /opt/kafka-current: /opt/kafka-current
- Do you wish to configure LDAP? [y|n]:
1) Y
2) N
#? n
- Enter Syndeia Cloud (SC) host / FQDN used for port check, ie: localhost, cassandra.mydomain.com, syndeia-cloud33.mydomain.com, sc33.mydomain.com, etc.: localhost
Mon Sep 30 12:53:12 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Checking if any SC 3.2 service file exist...
Mon Sep 30 12:53:12 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Checking if any SC 3.3/3.4/3.5/3.6 service files exist...
Mon Sep 30 12:53:12 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - /etc/systemd/system/sc-store.service not found
Mon Sep 30 12:53:12 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - /etc/systemd/system/sc-store.service not found
Mon Sep 30 12:53:12 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - /etc/systemd/system/sc-store.service not found
Mon Sep 30 12:53:12 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - /etc/systemd/system/sc-store.service not found
Mon Sep 30 12:53:12 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Check if any "syndeia-cloud" (debug) processes exist...
Mon Sep 30 12:53:12 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud not detected running.
Mon Sep 30 12:53:12 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Now creating install dir...
Mon Sep 30 12:53:12 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzip sub-packages...
Mon Sep 30 12:53:12 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping aras-impl-3.6-SP1.zip...
Mon Sep 30 12:53:12 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping artifactory-impl-3.6-SP1.zip...
Mon Sep 30 12:53:13 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping auth-impl-3.6-SP1.zip...
Mon Sep 30 12:53:14 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping bitbucket-impl-3.6-SP1.zip...
Mon Sep 30 12:53:14 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping collaborator-impl-3.6-SP1.zip...
Mon Sep 30 12:53:15 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping confluence-impl-3.6-SP1.zip...
Mon Sep 30 12:53:16 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping devops-3.6-SP1.zip...
Mon Sep 30 12:53:16 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping digital-thread-impl-3.6-SP1.zip...
Mon Sep 30 12:53:17 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping doors-impl-3.6-SP1.zip...
Mon Sep 30 12:53:18 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping dscr-impl-3.6-SP1.zip...
Mon Sep 30 12:53:18 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping dse3-impl-3.6-SP1.zip...
Mon Sep 30 12:53:19 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping genesys-impl-3.6-SP1.zip...
Mon Sep 30 12:53:20 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping github-impl-3.6-SP1.zip...
Mon Sep 30 12:53:20 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping gitlab-impl-3.6-SP1.zip...
Mon Sep 30 12:53:21 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping graph-impl-3.6-SP1.zip...
Mon Sep 30 12:53:22 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping jama-impl-3.6-SP1.zip...
Mon Sep 30 12:53:22 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping jira-impl-3.6-SP1.zip...
Mon Sep 30 12:53:23 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping restful-impl-3.6-SP1.zip...
Mon Sep 30 12:53:24 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping store-impl-3.6-SP1.zip...
Mon Sep 30 12:53:25 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping SyndeiaCloudCycloneDxBoms_3.6-SP1.zip...
Mon Sep 30 12:53:25 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping sysmlv2-impl-3.6-SP1.zip...
Mon Sep 30 12:53:25 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping teamcenter-impl-3.6-SP1.zip...
Mon Sep 30 12:53:26 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping teamworkcloud-impl-3.6-SP1.zip...
Mon Sep 30 12:53:27 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping testrail-impl-3.6-SP1.zip...
Mon Sep 30 12:53:27 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping volta-impl-3.6-SP1.zip...
Mon Sep 30 12:53:28 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping web-gateway-3.6-SP1.zip...
Mon Sep 30 12:53:30 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping windchill-impl-3.6-SP1.zip...
Mon Sep 30 12:53:30 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzip web-gateway conf files as text...
Mon Sep 30 12:53:30 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Take recursive ownership of install dir...
Mon Sep 30 12:53:30 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Create logs folder + store-impl auth-impl graph-impl aras-impl artifactory-impl bitbucket-impl collaborator-impl confluence-impl digital-thread-impl doors-impl dscr-impl dse3-impl jama-impl jira-impl genesys-impl github-impl gitlab-impl restful-impl sysmlv2-impl teamcenter-impl teamworkcloud-impl testrail-impl volta-impl windchill-impl service conf folders...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzip store-impl auth-impl graph-impl aras-impl artifactory-impl bitbucket-impl collaborator-impl confluence-impl digital-thread-impl doors-impl dscr-impl dse3-impl jama-impl jira-impl genesys-impl github-impl gitlab-impl restful-impl sysmlv2-impl teamcenter-impl teamworkcloud-impl testrail-impl volta-impl windchill-impl conf files as text...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping store-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping auth-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping graph-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping aras-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping artifactory-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping bitbucket-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping collaborator-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping confluence-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping digital-thread-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping doors-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping dscr-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping dse3-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping jama-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping jira-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping genesys-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping github-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping gitlab-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping restful-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping sysmlv2-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping teamcenter-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping teamworkcloud-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping testrail-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping volta-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping windchill-impl conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzipping devops conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Setting lagom.circuit-breaker call-timeout = 120 seconds in auth service's application.conf to avoid circuit breaker timeouts...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Unzip graph service's JanusGraph configuration .yaml file as text...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Configure all services' application.conf files...
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - setup_LDAP is false and/or upgrade is true: LDAP settings will not be updated in silhouette.conf from build
Mon Sep 30 12:53:31 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Generate concatenated gen-schema.cql file...
Mon Sep 30 12:53:32 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Apache Kafka installed locally, stopping Kafka to archive any previous logs, if any...
Mon Sep 30 12:53:33 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - tar old Apache Kafka logs folder (to prevent any interference if this is a re-install)...
Mon Sep 30 12:53:33 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: -
tar: Removing leading `/' from member names
Mon Sep 30 12:53:33 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Drop old keyspaces and import concatenated schema...
Warning: Using a password on the command line interface can be insecure.
Recommendation: use the credentials file to securely provide the password.
Warnings :
Role 'syndeia_admin' was already granted CREATE, ALTER, DROP, SELECT, MODIFY, AUTHORIZE on <keyspace syndeia_cloud_graph>
Warnings :
Role 'syndeia_admin' was already granted CREATE, ALTER, DROP, SELECT, MODIFY, AUTHORIZE on <keyspace syndeia_cloud_graph_config>
Warnings :
Role 'syndeia_admin' was already granted CREATE, ALTER, DROP, SELECT, MODIFY, AUTHORIZE on <keyspace syndeia_cloud_store>
Warnings :
SASI indexes are experimental and are not recommended for production use.
Warnings :
Role 'syndeia_admin' was already granted CREATE, ALTER, DROP, SELECT, MODIFY, AUTHORIZE on <keyspace syndeia_cloud_auth>
Warnings :
Role 'syndeia_admin' was already granted CREATE, ALTER, DROP, SELECT, MODIFY, AUTHORIZE on <keyspace syndeia_cloud_digital_threads>
Mon Sep 30 12:53:38 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Apache Kafka installed locally, starting Kafka if needed...
Mon Sep 30 12:53:38 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Confirming kafka port is up before proceeding...
Mon Sep 30 12:53:41 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Confirming janusgraph port is up before proceeding...
Mon Sep 30 12:53:41 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Install tmpfiles.d syndeia-cloud.conf file...
Mon Sep 30 12:53:41 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Install systemd syndeia-cloud .target and .service files...
Mon Sep 30 12:53:42 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Reload systemd daemons...
Mon Sep 30 12:53:42 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Check if sudoers/syndeia-cloud is installed...
Mon Sep 30 12:53:42 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - sudoers file not installed, install from source
Mon Sep 30 12:53:42 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - update sudoers file to be specific to logged in user (pj)
Mon Sep 30 12:53:42 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - systemd services for SC 3.6 detected...
Mon Sep 30 12:53:42 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Starting core Syndeia Cloud 3.6 services: sc-store sc-auth sc-graph sc-web-gateway...
Mon Sep 30 12:53:42 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Auth service port (56110) is up...
Mon Sep 30 12:53:48 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Store service port (56120) is up...
Mon Sep 30 12:53:49 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Graph service port (56180) is up...
Mon Sep 30 12:53:49 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Web-Gateway service port (9000) is up...
Mon Sep 30 12:53:49 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Enable SC services to start on boot...
Mon Sep 30 12:53:49 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Wait 30 seconds for system to stabilize...
Mon Sep 30 12:54:19 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Finish setup process (ie: create super.user, create sample data)...
Mon Sep 30 12:54:19 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: creating "super.user" account...
12:54:20.699 [info] com.intercax.syndeia.cli.core.DevopsConfig [] - root dir arg: None
12:54:20.709 [info] com.intercax.syndeia.cli.core.DevopsConfig [] - root dir config: Map()
12:54:20.910 [info] com.intercax.syndeia.cli.SyndeiaCli [] - Lagom Syndeia client
12:54:20.910 [info] com.intercax.syndeia.cli.SyndeiaCli [] - Action: superuser
12:54:20.911 [info] com.intercax.syndeia.cli.SyndeiaCli [] - User:
12:54:20.911 [info] com.intercax.syndeia.cli.SyndeiaCli [] - Mode: Prod
12:54:21.302 [info] akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
12:54:21.885 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Creating StandaloneLagomClientFactory. Running mode found in devops config: Prod
12:54:21.886 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Creating StandaloneLagomClientFactory with ConfigurationServiceLocatorComponents.
12:54:21.899 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Service locator configuration entry: [graph :: "http://127.0.0.1:56180"]
12:54:21.900 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Service locator configuration entry: [store :: "http://127.0.0.1:56120"]
12:54:21.900 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Service locator configuration entry: [doors :: "http://127.0.0.1:56113"]
12:54:21.900 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Service locator configuration entry: [jama :: "http://127.0.0.1:56114"]
12:54:21.900 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Service locator configuration entry: [web-gateway :: "http://127.0.0.1:9000"]
12:54:21.900 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Service locator configuration entry: [kafka_native :: "http://localhost:9092"]
12:54:21.900 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Service locator configuration entry: [gitlab :: "http://127.0.0.1:56123"]
12:54:21.900 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Service locator configuration entry: [windchill :: "http://127.0.0.1:56170"]
12:54:21.900 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Service locator configuration entry: [auth :: "http://127.0.0.1:56110"]
12:54:21.900 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Service locator configuration entry: [bitbucket :: "http://127.0.0.1:56124"]
12:54:21.900 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Service locator configuration entry: [aras :: "http://127.0.0.1:56111"]
12:54:21.900 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Service locator configuration entry: [sysmlv2 :: "http://127.0.0.1:56118"]
12:54:21.900 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Service locator configuration entry: [teamworkcloud :: "http://127.0.0.1:56160"]
12:54:21.900 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Service locator configuration entry: [cas_native :: "http://localhost:9042"]
12:54:21.901 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Service locator configuration value for web-gateway: http://127.0.0.1:9000
12:54:21.925 [info] akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
12:54:33.537 [info] com.intercax.syndeia.superuser.SuperUserAction [] - Sign-up super user is successful
12:54:33.597 [info] akka.actor.CoordinatedShutdown [akkaAddress=akka://application, akkaUid=1917033255123195448, sourceThread=application-akka.actor.default-dispatcher-6, akkaSource=CoordinatedShutdown(akka://application), sourceActorSystem=application, akkaTimestamp=16:54:33.596UTC] - Running CoordinatedShutdown with reason [ClientStoppedReason]
12:54:33.636 [info] com.intercax.syndeia.cli.SyndeiaCli [] - Action ran successfully (with a return code: 0).
Mon Sep 30 12:54:33 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Successfully ran DevOps (create) superuser action
Mon Sep 30 12:54:33 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: CHECK to see "super.user" account was (eventually) created...
Mon Sep 30 12:54:33 EDT 2024 [error] ./syndeia-cloud-3.6_install.bash, PID=549784: ~ DevOps (check) super.user signIn failed, waiting to try re-running DevOps (check) super.user signIn...
Mon Sep 30 12:54:39 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Successfully ran DevOps (check) super.user signIn
Mon Sep 30 12:54:39 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: creating default data set...
12:54:39.637 [info] com.intercax.syndeia.cli.core.DevopsConfig [] - root dir arg: None
12:54:39.646 [info] com.intercax.syndeia.cli.core.DevopsConfig [] - root dir config: Map()
12:54:39.831 [info] com.intercax.syndeia.cli.SyndeiaCli [] - Lagom Syndeia client
12:54:39.831 [info] com.intercax.syndeia.cli.SyndeiaCli [] - Action: setup
12:54:39.832 [info] com.intercax.syndeia.cli.SyndeiaCli [] - User: super.user
12:54:39.832 [info] com.intercax.syndeia.cli.SyndeiaCli [] - Mode: Prod
12:54:40.093 [info] akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
12:54:40.492 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Creating StandaloneLagomClientFactory. Running mode found in devops config: Prod
12:54:40.493 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Creating StandaloneLagomClientFactory with ConfigurationServiceLocatorComponents.
12:54:40.506 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Service locator configuration entry: [graph :: "http://127.0.0.1:56180"]
12:54:40.506 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Service locator configuration entry: [store :: "http://127.0.0.1:56120"]
12:54:40.506 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Service locator configuration entry: [doors :: "http://127.0.0.1:56113"]
12:54:40.506 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Service locator configuration entry: [jama :: "http://127.0.0.1:56114"]
12:54:40.506 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Service locator configuration entry: [web-gateway :: "http://127.0.0.1:9000"]
12:54:40.506 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Service locator configuration entry: [kafka_native :: "http://localhost:9092"]
12:54:40.506 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Service locator configuration entry: [gitlab :: "http://127.0.0.1:56123"]
12:54:40.506 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Service locator configuration entry: [windchill :: "http://127.0.0.1:56170"]
12:54:40.506 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Service locator configuration entry: [auth :: "http://127.0.0.1:56110"]
12:54:40.506 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Service locator configuration entry: [bitbucket :: "http://127.0.0.1:56124"]
12:54:40.506 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Service locator configuration entry: [aras :: "http://127.0.0.1:56111"]
12:54:40.507 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Service locator configuration entry: [sysmlv2 :: "http://127.0.0.1:56118"]
12:54:40.507 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Service locator configuration entry: [teamworkcloud :: "http://127.0.0.1:56160"]
12:54:40.507 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Service locator configuration entry: [cas_native :: "http://localhost:9042"]
12:54:40.507 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Service locator configuration value for web-gateway: http://127.0.0.1:9000
12:54:40.521 [info] akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
12:54:41.777 [info] com.intercax.syndeia.api.AuthApi [] - Sign in to Syndeia Cloud for super.user successful.
12:54:57.349 [info] com.intercax.syndeia.api.RepositoryApi [] - Creating repositories... with RepositoryCreate(Some(InternalIdentityForm(None,None)),Some(ExternalIdentityForm(None,None)),None,None,None,None,Syndeia Repository,Some(Syndeia Internal Repository),Some(http://intercax.com:9000/repositories?isInternal=true),None,ReferenceCreate(Some(InternalReferenceForm(REPO-TYPE1)),None),None,None)
12:54:57.973 [info] com.intercax.syndeia.api.RepositoryApi [] - Creating repositories... with RepositoryCreate(None,Some(ExternalIdentityForm(Some(http://localhost.com/repositories/sysml),Some(http://localhost.com/repositories/sysml))),None,None,None,None,SysML Repository,Some(SysML Repository),Some(http://localhost.com/repositories/sysml),None,ReferenceCreate(Some(InternalReferenceForm(REPO-TYPE18)),None),None,None)
12:54:59.348 [info] com.intercax.syndeia.setup.RelationTypeSetUp [] - Found: 0 existing relation types
12:55:32.965 [info] com.intercax.syndeia.setup.CloudSetupAction [] - Set up action is successful
12:55:33.038 [info] akka.actor.CoordinatedShutdown [akkaAddress=akka://application, akkaUid=3979485647664124636, sourceThread=application-akka.actor.default-dispatcher-5, akkaSource=CoordinatedShutdown(akka://application), sourceActorSystem=application, akkaTimestamp=16:55:33.038UTC] - Running CoordinatedShutdown with reason [ClientStoppedReason]
12:55:33.071 [info] com.intercax.syndeia.cli.SyndeiaCli [] - Action ran successfully (with a return code: 0).
Mon Sep 30 12:55:33 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Successfully ran DevOps setup action
Mon Sep 30 12:55:33 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - systemd services for SC 3.6 detected...
Mon Sep 30 12:55:33 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Starting integration Syndeia Cloud 3.6 services: aras-impl artifactory-impl bitbucket-impl collaborator-impl confluence-impl digital-thread-impl doors-impl dscr-impl dse3-impl jama-impl jira-impl genesys-impl github-impl gitlab-impl restful-impl sysmlv2-impl teamcenter-impl teamworkcloud-impl testrail-impl volta-impl windchill-impl...
Mon Sep 30 12:55:33 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Aras service port (56111) is up...
Mon Sep 30 12:56:00 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Artifactory service port (56112) is up...
Mon Sep 30 12:56:00 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Bitbucket service port (56124) is up...
Mon Sep 30 12:56:00 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Collaborator service port (56126) is up...
Mon Sep 30 12:56:00 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Confluence service port (56140) is up...
Mon Sep 30 12:56:00 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Dt service port (56132) is up...
Mon Sep 30 12:56:01 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Doors service port (56113) is up...
Mon Sep 30 12:56:01 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Dscr service port (56131) is up...
Mon Sep 30 12:56:01 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Dse3 service port (56128) is up...
Mon Sep 30 12:56:01 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Jama service port (56114) is up...
Mon Sep 30 12:56:01 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Jira service port (56130) is up...
Mon Sep 30 12:56:01 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Genesys service port (56129) is up...
Mon Sep 30 12:56:01 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Github service port (56150) is up...
Mon Sep 30 12:56:01 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Gitlab service port (56123) is up...
Mon Sep 30 12:56:01 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Restful service port (56127) is up...
Mon Sep 30 12:56:01 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Sysmlv2 service port (56118) is up...
Mon Sep 30 12:56:01 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Tc service port (56116) is up...
Mon Sep 30 12:56:01 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Twcloud service port (56160) is up...
Mon Sep 30 12:56:01 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Testrail service port (56117) is up...
Mon Sep 30 12:56:01 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Volta service port (56125) is up...
Mon Sep 30 12:56:01 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Syndeia Cloud: Waiting until SC Wc service port (56170) is up...
Mon Sep 30 12:56:01 EDT 2024 [info] ./syndeia-cloud-3.6_install.bash, PID=549784: - Done!
Syndeia Cloud (SC) 3.6 introduces new integrations for the following tools. These integrations require JAR files (libraries) to be collected and deployed.
If your organization is using these tools and your Syndeia license grants you access to these integrations, please follow the steps from the relevant page(s) linked above to gather and install these JAR files from the external repository server(s), otherwise proceed to the Verification section.
Verification
On the server and/or your local machine, launch a web browser & check the following to validate that the application is correctly running:
http://<syndeia_server_FQDN>:9000 should give you:
To log in as the default administrator and create users, see the User Management section.Once logged in, please verify you see:
a bar graph gets rendered on the Dashboard home page and
the installed version shows correctly under Help > About in the sidebar.
Congratulations! ⭐ You have a running Syndeia Cloud instance.
Review the following sections to learn about monitoring the services and locating service logs.
How to Manage Services & Check Logs
Services
Services are all managed using systemd’s systemctl command with a verb, ie: status, start, stop, restart, followed the service name, ie: cassandra, janusgraph, zookeeper, kafka, sc-SC-short-service-name(s) (or sc-* to reference all SC services). For more information run systemctl --help and/or man systemctl .
Example usage for Cassandra, Zookeeper, Kafka, JanusGraph, and Syndeia Cloud follows:
Apache Cassandra
To check the summary status:
sudo systemctl status cassandra
To start the service:
sudo systemctl start cassandra
To stop the service:
sudo systemctl stop cassandra
To restart the service:
sudo systemctl restart cassandra
Apache Zookeeper
To check the summary status:
sudo systemctl status zookeeper
To start the service:
sudo systemctl start zookeeper
To stop the service:
sudo systemctl stop zookeeper
To restart the service:
sudo systemctl restart zookeeper
Apache Kafka
To check the summary status:
sudo systemctl status kafka
To start the service:
sudo systemctl start kafka
To stop the service:
sudo systemctl stop kafka
To restart the service:
sudo systemctl restart kafka
JanusGraph
To check the summary status:
sudo systemctl status janusgraph
To start the service:
sudo systemctl start janusgraph
To stop the service:
sudo systemctl stop janusgraph
To restart the service:
sudo systemctl restart janusgraph
Syndeia Cloud
Syndeia Cloud 3.6 is defined by two sets of services:
SC 3.6 Core Services:
sc-store,sc-auth,sc-graph,sc-web-gatewaySC 3.6 Integration Services (🌟 = New!) :
sc-aras,sc-artifactory,sc-bitbucket,sc-collaborator,sc-confluence,sc-doors,sc-dscr🌟,sc-dse3🌟 ,sc-dt🌟 ,sc-github,sc-genesys🌟,sc-gitlab,sc-jama,sc-jira,sc-restful,sc-sysmlv2,sc-tc🌟,sc-testrail,sc-twcloud,sc-volta,sc-wc
To perform an operation on all services at once, you can use the wildcard sc-* to match all of the above
To check the summary status for a specific service, ex: web-gateway:
sudo systemctl status sc-web-gateway
To start a specific service, ex: web-gateway:
sudo systemctl start sc-web-gateway
To start all services with dependencies, use the specially defined SC group, use sc.target
To stop a service, ex: web-gateway:
sudo systemctl stop sc-web-gateway
To restart a service, ex: web-gateway:
sudo systemctl restart sc-web-gateway
Logs & Monitoring
Logs on Linux can be viewed using the journalctl command, ex: sudo journalctl -xeu service-name, ie: cassandra, janusgraph, zookeeper, kafka, sc-SC-short-service-name(s)
To tail a particular service’s logs add the f switch to the journalctl command, ie: sudo journalctl -xfeu service-name, or use the sudo tail -f /path/to/service.log command.
Raw log files are located in the following locations:
Apache Cassandra
/var/log/cassandra/system.log
Apache Zookeeper
Zookeeper creates a log file of the form zookeeper-accountName -server-serverFQDN .log
This is cumbersome, so for simplicity we have created a symlink to it as zookeeper.log
/opt/zookeeper-current/logs/zookeeper.log
Apache Kafka
Apache Kafka generates several files with the extension .log in its logs folder
Most of these files constitute Kafka’s DB, the only one that is of concern for diagnostics/troubleshooting is the one named server.log
/opt/kafka-current/logs/server.log
JanusGraph
/opt/janusgraph-current/logs/gremlin-server.log
Syndeia Cloud
For single-node deployments, one can use the common logs directory with symlinks to every service’s logs folder (this is useful is say one wishes to quickly archive all logs to submit for troubleshooting)
/opt/icx/syndeia-cloud-current/logs
For an individual service’s logs, log files will be under:
/opt/icx/syndeia-cloud-current/$service_name-impl-3.6*/logs/$service_name.log
… where $service_name = any one from the below two sets of services:
SC 3.6 Core Services:
store,auth,graph,web-gatewaySC 3.6 Integration Services (🌟 = New!) :
aras,artifactory,bitbucket,collaborator,confluence,doors,dscr🌟,dse3🌟 ,digital-thread🌟 ,github,genesys🌟,gitlab,jama,jira,restful,sysmlv2,teamcenter🌟,testrail,twcloud,volta,windchill Note, teamcenterhas an additional logs subfolder underlogs/SOAfor the logs generated by the Siemens SOA client libs.
Troubleshooting
The pre-setup scripts have been tested on the documented platforms so most scripts should execute reliably (ideally flawlessly).
However having said that, it’s always possible that:
you may have missed a step (or steps)
you have a non-standard environment
there’s a genuine bug
If you do experience any issues, please perform the following before opening that Helpdesk ticket:
Review the output: if you notice an error, 🛑 stop! 🖐 VS simply moving forward (this is one of the reasons trace mode was (currently) left enabled VS hiding the output).
Uninstall & try again: Try following the below instructions to reset to a clean state and start over (in case a step (or steps) were missed) :
Uninstall
WARNING: The following will uninstall SC and all dependencies
Save the below file named SC_stack_uninstall.bash to /opt/, set execute permission on it (ie: chmod ug+x), then run it from a bash Terminal, ie: ./SC_stack_uninstall.bash (on Windows this would be a Cygwin Terminal, on Linux this would be a bash shell prompt)
Tips
Ensure you DON’T run ANY of the setup scripts with
rootorsudo!Ensure you DO have enough space allocated to
/opt, /var, /tmp( check via sudo df -h),Ensure you DON'T have a
noexecmount option for/tmp( check via mount, note, it may not be directly under/and could be a sub-dir under/).Ensure you DO have enough CPU cores, ie: check via:
cores=$(tail -28 /proc/cpuinfo | grep processor | awk '{ print $3 }'); cores=$((cores+1)); echo $coresIf that # is < the minimum requirements (14 for SC 3.6), please bump this up value.
Ensure you DO have the SC media downloaded to the home dir and NOT a subdirectory or anywhere else
- No labels