Deployment

Owned by Ambarish Ganesh (Unlicensed)
Last updated: Sept 23, 2022 by Brian MillerVersion comment
5 min read

Architecture

A Syndeia Cloud (SC) 3.4 installation has the following stack of component dependencies (#1 = bottom of stack, #5 = top of stack):  

  1. Apache (or DataStax) Cassandra:   NoSQL Database (DB) layer
  2. Janusgraph (JG):  Gremlin query language and graph framework (depends on #1 for DB)
  3. Apache Zookeeper (ZK):  Cluster broker (even in single-node deployments)
  4. Apache Kafka:  Event processing queue for streams (depends on #3, even in single-node deployments)- used by the SC graph service
  5. Syndeia Cloud (SC):  Server-based platform to create a queryable, visualizable & extensible federated digital thread among/within various engineering PLM tools.  
    1. SC core services:  core services that make up the SC framework (ideally should be started in below order)  
      1. sc-store:  communicates with backend DB (depends on #1 (Cassandra)),
      2. sc-auth:  handles authentication related requests (depends on #1 (Cassandra)),
      3. sc-graph:  handles graph visualization tasks, ie: conversion of data from main DB into graph-friendly DB format, processing Gremlin graph queries, etc. (depends on #1 (Cassandra) & #4 (Apache Kafka)),
      4. sc-web-gateway:  web dashboard + service request router (no service start dependencies but other services should be available for this service to be useful)
    2. SC integration services:  integration-specific service for each PLM tool being integrated with (ex: Aras, jFrog Artifactory, Bitbucket (BB), Atlassian Confluence, IBM Jazz CLM- includes Doors NG (DNG), Github, Gitlab, Jama, JIRA, SysMLv2, TestRail, NoMagic Teamwork Cloud (TWCloud), PTC Windchill (WC)).   (no service start dependencies but core services should be available for these services to be accessible)



Version Compatibility

Syndeia Cloud v3.4 has been tested with the following software versions:  

  • RHEL/CentOS/AlmaLinux 7.7 ~ 7.9 ( (star) Linux strongly recommended)
  • Windows Server 2012 ~ 2016
  • Java (Oracle or Open)JDK/JRE v1.8.0_322
  • Apache Cassandra v3.11.1 ~ 10
  • Janusgraph v0.3.1
  • Zookeeper v3.4.8.1
  • Kafka v2.11-1.1.0

(warning) Note, newer versions may or may not work (in particular note the following change to JNDI parsing in JDK/JRE v1.8.0_331+, which requires Cassandra v3.11.13).  

Requirements

(info)  Currently we recommend any machine that will be running Syndeia Cloud (SC), Cassandra, Janusgraph, Zookeeper, and Kafka be dedicated (ie: not shared with other vendor software, ex:  sharing Cassandra DB with Teamwork Cloud, etc.). 

A word about hosting infrastructure:  

(warning) WARNING:  attempting to run on a non-dedicated node (ie: default shared cloud/VPS/AWS instance or (overcommitted) internal IT VM hosts) may subject you to unknown "noisy guest neighbors".  Depending on your relationship with the hosting provider (ie: internal or external 3rd-party) and the monitoring tools provided, you may or may not have any visibility or control over this.  This could cause intermittent CQLSH timeouts or SC Circuit Breaker errors.  If you experience this, please increase (and or reserve) the resources allocated to your node.  (lightbulb) Linux guests have a bit more visibility by allowing examination of the steal metric, ex: by running iostat 1 10 and/or running top | htop (from epel repository). 

Having said the above we have the following hardware sizing requirements:

Minimum

Single Node Deployment Topology

  • CPU cores:  8 cores *
  • RAM:  16GB of RAM *
  • HD space:  100GB ( (info) Caution regarding partitioning layouts on Linux:  Some installers will by default suggest a partition layout where the bulk of disk space is allocated to a users' home folders (ie: /home).  Please ensure this is NOT the case.  The majority of disk space should be allocated to SC software, infrastructure components, DB data files, and log files, ie:  /opt, /var/{lib,log})

Ideal

Single-Node Deployment Topology (Recommended)

  • CPU cores:  21 (4 total, 1 for each dependency (Cassandra, JG, Zookeeper, Kafka) + 4 total for SC core services (sc-store, sc-auth, sc-graph, sc-web-gateway) + 1 per integration (ex: Aras, Artifactory, etc.) | or 13 for all integrations running simultaneously) *
  • RAM:  32GB of RAM **
  • HD space:  100GB ***

Multi-Node Deployment Topologies

  • CPU cores:  (varies depending on topology, see Single-Node Deployment for core breakdown)
  • RAM:  32GB of RAM **
  • HD space:  100GB ***

(warning)  Note, multi-node topologies are an advanced deployment with cluster management responsibilities, we recommend beginning with a single-node deployment to start with and re-evaluate as your usage requirements grow

Sizing References

* Adjust per your expected CCU requirements.  As a reference point, internal SC 3.4 scalability testing showed the below summary results:  

  1. 500 concurrent users (CCUs) with a max latency of 6-9k ms, min 100ms, avg 1500 ms and max response time of 267ms, min 0 ms, avg of 31ms, without encountering errors
  2. 100 concurrent users (CCUs) with a max latency of 1.5k ms, min 66ms, avg 586 ms and max response time of 69ms, min 0 ms, avg of 7.1ms, without encountering errors   

... with the following hardware setup:

  • server = cloud "(org) dedicated" 8 core instance running on an AMD EPYC 7501 32-Core Processor @ 2GHz w/ 512KB of L2 cache + 16GB of RAM + Linux Ubuntu 16.04 LTS + Cassandra 3.11.5 + Syndeia Cloud v3.4.2020-10-01
  • client = Apple MBP (Retina, 15-inch, Mid 2015) 4 core x 2.8 GHz Intel Core i7 + 16 GB of RAM (1600 MHz DDR3) + Mac OSX 10.13.X + jMeter v5.2.1
  • network = WiFi (802.11ac + WPA2 Personal encryption) with an average network latency of 20ms

... and jMeter test case:

  1. HTTP GET /signIn
    1. Assert HTTP 2XX was received
    2. Use JSON Extractor to extract token for use in subsequent calls
  2. HTTP GET /repositories
    1. Assert HTTP 2XX was received
  3. HTTP GET /containers
    1. Assert HTTP 2XX was received
  4. HTTP POST /graph/query/raw (g.V().count())
    1. Assert HTTP 2XX was received

** Adjust per your expected expected graph sizing requirements.

*** As a reference point, SC itself when extracted requires ~1.8GB and a sample environment with 9.2k relations, 66 relation types, ~16.4k artifacts, 285 artifact types, ~1k containers, 101 container types, 71 repositories, and 18 repository types requires ~306MB at the Cassandra database data layer (Replication Factor (RF) = 1 ) and ~281MB at the Kafka stream log layer on an EXT4 file system on a single-node Cassandra deployment.   Note, we recommend periodically monitoring the size of the Kafka stream "logs" & gremlin-server.log file(s) as they have a tendency to grow (the latter can grow up to 1GB in a year with the default slf4jReporter metric enabled and logging every 180000 ms) 


Overview

The following pages describe how to quickly deploy Syndeia Cloud + its dependencies on the following platforms & setup services to start on boot: 


.....Page break.....