/
Stage 2 - Cloning Source Syndeia Cloud Server

Stage 2 - Cloning Source Syndeia Cloud Server

Objective

Cloning Syndeia Cloud (SC) deployment with single-node Cassandra to a Target Server to prepare for migration to 3.6 SP2.

From this point forward, the following is assumed:

  • Steps that need to be run on the “Target Server” / “Source Server” will be marked as such to clearly distinguish which node one needs to perform the step on.

  • Awareness of the location of configuration of the main Cassandra config file (Typically /etc/cassandra/cassandra.yaml).

  • All the steps, unless indicated otherwise, should be performed from a bash shell.

  • This is a single-node deployment on a RHEL or RHEL derived Linux system. See Upgrade (3.5 SP2 -> 3.6 SP2, 3.6 -> 3.6 SP2).

Prerequisites

  1. Metrics were successfully collected for the Syndeia Cloud server in Stage 1.

  2. Basic familiarity with Cassandra DB, service management & log file locations

  3. Important - If using HTTPS, you will need to request, have issued/generated, and downloaded a target cert for the newly cloned server. If using a commercial CA this process can sometimes take a while (days?) so plan accordingly to have this ready! (see Securing Syndeia Cloud for basic information on this)

  4. Important - Ensure that the target server meets all requirements specified here: Deployment

Steps

1. Pre-Cloning Preparation Steps (Source Server)

Before cloning your machine, you will need to confirm certain settings and prepare it to ensure that there is no interference with the Source Server when the clone is deployed.

Step 1.1 - If using HTTPS - Verify HTTPS settings in devops config file

View contents of /opt/icx/syndeia-cloud-current/devops-*/conf/application.conf to ensure that your web-gateway is updated as shown below. Replace <FQDN> with the actual fully qualified domain name (FQDN) if not.

lagom.services { cas_native = "http://localhost:9042" [...] web-gateway = "https://<FQDN>:9443" [...] }

Step 1.2 - Verify that your cassandra.yaml file has the below mentioned settings.

listen_address: localhost rpc_address: localhost seeds: "127.0.0.1"

Step 1.3 - Disable auto-start of all services.

cd /etc/systemd/system sudo systemctl disable cassandra janusgraph zookeeper kafka sc-*

WARNING, failure to do this (especially if Step 2 is not true) will cause Cassandra to auto-start and attempt to join it's source cluster, causing a node-collision!

Step 1.4 - Stop all SC + infrastructure services (Cassandra + Janusgraph + Zookeeper + Kafka)

cd /etc/systemd/system sudo systemctl stop sc-* && sudo systemctl stop janusgraph kafka zookeeper && sudo systemctl stop cassandra

2. Cloning Steps (Source Server)

Step 2.1 - Clone the Source Server. You may use your choice of cloning software depending on your hosting service, e.g.

  • AWS (volume snapshot / AMIs)

  • Linode (Linode snapshot / backups)

  • GCP (…)

  • Azure (…)

  • VM (Virtual Box, VMware, KVM, etc.: Clone option, or copying VM folder)

  • Physical hardware (Altiris rdeploy, Symantec Ghost, etc.)

Step 2.2 - Deploy image onto Target Server & boot it.

Please see the Deployment section to confirm you are deploying onto an instance that meets the new requirements.

3. Reconfiguration Steps (Target Server)

3.1 Target Server Preparation Steps

The steps in this subsection 3.1 describe the process of modifying the hostname of the target server. This is done to prevent several issues that can occur when two servers with the same hostname are running simultaneously.

If you wish for the target server to have the same hostname as the source server, we recommend shutting down the source server and ensuring that both servers are never running simultaneously. In this case, none of the following steps are required and you can skip straight to the next stage.

Step 3.1.1 - Run the following from a bash shell to save the old hostname for subsequent steps and set new hostname.

hostname # should print full FQDN of old hostname, ex: host.domain.tld hostname -s # should print just the hostname, i.e.: WITHOUT Fully Qualified Domain Name (FQDN), ex: host export old_cassandra_JG_host=$(hostname); export old_cassandra_JG_host_short=$(hostname -s); export new_cassandra_JG_host=localhost # set this to locahost to avoid the network hop

Step 3.1.2 - Update the hostname and reload the bash shell:

sudo hostnamectl set-hostname <new_node_name_FQDN> exec bash

Step 3.1.3 - Update any other critical system-related settings you deem relevant, i.e. for services besides JG and SC, you MUST update all other references of the old hostname / FQDN on the system. Run the below to identify any significant files.

# Find any dirs with old hostname in it sudo find / -xdev -type d -not -path '*spool*' -not -name '*mail*' -not -path '*mail*' -not -path '*log*' -not -name '*log*' -not -name 'swap*' -not -path '/opt/icx/*' -not -path '/opt/janusgraph*' -not -name '*.bak' -not -name '*history' -iname '*'${old_cassandra_JG_host_short}'*'
# Find any files with old hostname in it sudo find / -xdev -type f -not -path '*spool*' -not -name '*mail*' -not -path '*mail*' -not -path '*log*' -not -name '*log*' -not -name 'swap*' -not -name '*cassandra.yaml' -not -path '/opt/icx/*' -not -path '/opt/janusgraph*' -not -name '*.bak' -not -name '*history' -exec grep -Iin --color=always ${old_cassandra_JG_host_short} {} +

For each reference found (there should only be a few), use the command below to update it, replacing <source_server_name>, <target_server_name>, and <filename_to_update> with the appropriate values.

sudo sed -i 's/<source_server_name>/<target_server_name>/g' <filename_to_update>

If you are using LetsEncrypt.org’s certbot, you will need to reinstall (ie: remove certbot and /etc/letsecnrypt) and re-request any certificates

3.2 Cassandra Config Update Steps

Step 3.2.1 - Start Cassandra:

sudo systemctl start cassandra

Step 3.2.2 - Confirm successful startup i.e. the following command should show status Active in green -

sudo systemctl status cassandra

Step 3.2.3 - The following command should not show any errors -

sudo journalctl -xfeu cassandra

Step 3.2.4 - Run the following command to note original host_id -

nodetool status

Step 3.2.5 - Start CQLSH with the correct username by running the following command (where <username> = a super (user) account, e.g.: syndeia_admin or default = cassandra account) -

cqlsh -u <username> localhost

Step 3.2.6 - Assign a new host_id (in cqlsh - this prevents future collisions if node were to ever join into a cluster with Source Server) -

UPDATE system.local SET host_id = uuid() WHERE key = 'local';

Step 3.2.7 - Note source cluster name via cqlsh:

SELECT cluster_name FROM system.local WHERE key='local';

Step 3.2.8 - Update cluster name via cqlsh, where New_Cluster_Name = new cluster name to assign (must be different from source cluster) -

UPDATE system.local SET cluster_name = 'New_Cluster_Name Cluster' WHERE key='local';

Step 3.2.9 - Run the following command to exit out of cqlsh -

exit

Step 3.2.10 - Run the following command in shell (bash)-

nodetool flush system

Step 3.2.11 - Update the following line in the cassandra.yaml file (where ____ = new cluster name) -

cluster_name: '____ Cluster'

Step 3.2.12 - Restart the cassandra service to pick up changes to cassandra.yaml -

sudo systemctl restart cassandra

Step 3.2.13 - Run the following command to verify updated host_id -

nodetool status

It should look something like the below:

Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 127.0.0.1 224.93 MiB 256 100.0% 74ecd874-b5d8-4ebc-97ef-6889bd6c21c0 rack1

Step 3.2.14 - Run the following command to verify updated cluster name -

nodetool describecluster

It should look something like the below:

Cluster Information: Name: New_Cluster_Name Cluster Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch Partitioner: org.apache.cassandra.dht.Murmur3Partitioner Schema versions: f12cee57-b7c1-38c6-ab00-bfab205cbd83: [127.0.0.1]

3.3 JanusGraph Config Update Steps

Step 3.3.1 - Show all JG related settings that need updating (note, it’s possible you may not have any that need updating)-

egrep -IRin --color=always "${old_cassandra_JG_host_short}|0\.0\.0\.0" /opt/janusgraph-current/conf/{janusgraph-cql-configurationgraph-syndeia.properties,syndeia-cloud-3.?_janusgraph_init.groovy,gremlin-server/gremlin-server-configuration-syndeia.yaml}

Step 3.3.2 - Update JG config file by running the following command from a bash shell -

sudo -v && \ JG_HOME=/opt/janusgraph-current && \ sudo sed -i "s/storage.hostname=${old_cassandra_JG_host}/storage.hostname=${new_cassandra_JG_host}/g" ${JG_HOME}/conf/janusgraph-cql-configurationgraph-syndeia.properties && \ sudo sed -i "s/host: 0.0.0.0/host: ${new_cassandra_JG_host}/g" ${JG_HOME}/conf/gremlin-server/gremlin-server-configuration-syndeia.yaml && \ sudo sed -i "s/${old_cassandra_JG_host}/${new_cassandra_JG_host}/g" /opt/janusgraph-current/conf/syndeia-cloud-3.?_janusgraph_init.groovy

Step 3.3.3 - Start JG service by running the following command to ensure it started with no errors -

sudo systemctl start janusgraph

You may initially see messages like the ones below, ignore them as the DB is still configured to attempt to talk to the old DB, which is what we will change in the next step.

Aug 29 17:05:40 myhost.domain.tld gremlin-server.sh[15785]: 6137624 [pool-12-thread-1] ERROR org.janusgraph.graphdb.management.JanusGraphManager - Failed to open graph syndeia_cloud_graph with the following error: Aug 29 17:05:40 myhost.domain.tld gremlin-server.sh[15785]: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager. Aug 29 17:05:40 myhost.domain.tld gremlin-server.sh[15785]: Thus, it and its traversal will not be bound on this server.

Step 3.3.4 - Update JG config DB: update configured_graph JG meta graph object to point to localhost by running the Gremlin client:

${JG_HOME}/bin/gremlin.sh

… and then running each of the following:

:remote connect tinkerpop.server conf/remote.yaml session :remote console
map = new HashMap(); map.put('storage.hostname', 'localhost'); // should return: ==>null
ConfiguredGraphFactory.updateConfiguration('syndeia_cloud_graph', new MapConfiguration(map)); ConfiguredGraphFactory.getConfiguration('syndeia_cloud_graph'); // should return: // ==>storage.username=syndeia_admin // ==>graph.graphname=syndeia_cloud_graph // ==>storage.backend=cql // ==>storage.hostname=localhost // ==>storage.password=yourPw // ==>Template_Configuration=false

Forgetting to do the above could result in JG’s syndeia_cloud_graph_config meta DB pointing to the Original Server’s JG DB.

graph = ConfiguredGraphFactory.open('syndeia_cloud_graph'); // should return: // ==>standardjanusgraph[cql:[localhost]]
graph.tx().commit(); // should return: // ==>null

To quit type:

:quit

Step 3.3.6 - Restart JG service by running the following command -

sudo systemctl restart janusgraph

3.4 Syndeia Cloud (SC) Config Update Steps

For SC we need to set the SC version we are upgrading from and the build artifact (service/utility) names from that version.

Depending on which release of SC you are upgrading from, execute the appropriate section below from a bash shell.

Step 3.4.1 - Set the SC snapshot version and build artifact (service/utility) names.

If source server is SC 3.5 SP2:

SC_snapshot_version=3.5-SP2 sc_lagom_service_artifact_names=(auth-impl store-impl graph-impl aras-impl artifactory-impl bitbucket-impl collaborator-impl confluence-impl doors-impl jama-impl jira-impl github-impl gitlab-impl restful-impl sysmlv2-impl testrail-impl teamworkcloud-impl volta-impl windchill-impl) sc_build_artifact_names=("${sc_lagom_service_artifact_names[@]}" devops web-gateway)

If source server is SC 3.6:

SC_snapshot_version=3.6 sc_lagom_service_artifact_names=(auth-impl store-impl graph-impl aras-impl artifactory-impl bitbucket-impl collaborator-impl confluence-impl digital-thread-impl doors-impl dscr-impl dse3-impl genesys-impl jama-impl jira-impl github-impl gitlab-impl restful-impl sysmlv2-impl teamcenter-impl testrail-impl teamworkcloud-impl volta-impl windchill-impl) sc_build_artifact_names=("${sc_lagom_service_artifact_names[@]}" devops web-gateway)

Step 3.4.3 - Update SC’s application.conf files + graph-impl’s janusconfiguration{,String}.yaml files:

sudo -v && \ SC_HOME=/opt/icx/syndeia-cloud-current && \ for i in "${sc_build_artifact_names[@]}"; do sudo -u syndeia-cloud sed -i.build 's/cas_native\s*=\s*"http:\/\/'${old_cassandra_JG_host}'/cas_native = "http:\/\/localhost/;' ${SC_HOME}/$i-${SC_snapshot_version}/conf/application.conf && \ # We also need to make sure that we update any devops' (+ other?) application.conf files that had their lagom.services' web-gateway entry updated to deal with HTTPS-enabled servers sudo -u syndeia-cloud sed -i.build 's/web-gateway\s*=\s*"\(https\?:\/\/\)'${old_cassandra_JG_host}'\(:[0-9]\{4\}\)/web-gateway = "\1'$(hostname)'\2/;' ${SC_HOME}/$i-${SC_snapshot_version}/conf/application.conf && \ sudo -u syndeia-cloud sed -i.build 's/${old_cassandra_JG_host}/'$(hostname)'/;' /opt/icx/syndeia-cloud-current/web-gateway-${SC_snapshot_version}/conf/HTTPS.conf && \ sudo -u syndeia-cloud sed -i.build 's/${old_cassandra_JG_host}/'$(hostname)'/;' /opt/icx/syndeia-cloud-current/web-gateway-${SC_snapshot_version}/conf/silhouette.conf done && \ sudo sed -i "s/hosts: \[0.0.0.0\]/hosts: [${new_cassandra_JG_host}]/g" "${SC_HOME}"/graph-impl-${SC_snapshot_version}/conf/janusconfiguration.yaml && \ sudo sed -i "s/hosts: \[0.0.0.0\]/hosts: [${new_cassandra_JG_host}]/g" "${SC_HOME}"/graph-impl-${SC_snapshot_version}/conf/janusconfigurationString.yaml

4 Enable and Restart All Services Steps (Source Server)

On the Source Server, re-enable services to start automatically on boot and restart all services. You can skip these steps if you are certain that this server will not be used again during or after the upgrade.

Step 4.1 - Re-enable auto-start for all services:

cd /etc/systemd/system sudo systemctl enable cassandra janusgraph zookeeper kafka sc-*

Step 4.2 - Start all services:

sudo systemctl start cassandra janusgraph zookeeper kafka sc.target

Verification (Target Server)

  1. Cloned & reconfigured server should be able to exist independently of the source SC deployment, ie: server should NOT show up on source server’s Cassandra nodetool status output. To be 100% certain, as a sanity test you can run Stage 1 - Collect statistics from Source Syndeia Cloud Server again to verify metrics still have not changed.

  2. If cloned & reconfigured server was using HTTPS, then verify a new cert was properly provisioned for the new FQDN defined during Server Reconfiguration.