We are glad that you are reading this page because that means you’re already convinced of the power of Syndeia 3.3 with all the cool, new features and you’ve either already installed Syndeia Cloud 3.3, or are in the process of doing so, and you are wondering what happens to the data that you’ve painstakingly created with Syndeia Cloud 3.2? Well, since Syndeia 3.3 was such a big release, a lot has changed behind the scenes, and that means, unavoidably, you’ll have to migrate your data so that it complies with the data format expected by Syndeia 3.3.

This page explains what the entire migration process entails and the steps you perform to successfully transition from Syndeia 3.2 to Syndeia 3.3.

Pre-requisites

1. Syndeia 3.2 installed and running on a server. Syndeia 3.2 requires the Cassandra database and a syndeia keyspace to hold 3.2 data. This is our source server (database).

2. Syndeia 3.3 installed and running on a server. Syndeia 3.3 also requires the Cassandra database and syndeia_cloud_store & syndeia_cloud_auth keyspaces to hold 3.3 data. Additionally, Syndeia 3. 3 uses JanusGraph behind the scenes, which requires additional syndeia_cloud_graph & syndeia_cloud_graph_config keyspaces to hold graph data. This is our target server (database).

3. The superuser and the setup steps should’ve already been run on the Syndeia 3.3 server.The migration process requires an additional syndeia_cloud_devops keyspace on the target 3.3 server. The schema/gen-schema.cql file contained in this utility has the required CQL statements to create the schema and tables. Please run this utility from a CQL shell or using a tool similar to DataStax Devcenter. This keyspace has three sets of tables - a pre_stage_* set, a migrate_stage_* set, and a duplicate_* set of tables.

4. Since we might be dealing with huge data, we also need to tweak two Cassandra timeout variables in conf/cassandra.The yaml file namely:
   read_request_timeout_in_ms (how long the coordinator should wait for read operations to complete) - whose default value is 5000, but we have to increase it to 60000, and,
   request_timeout_in_ms (the default timeout for other, miscellaneous operations) - whose default value is 10000, but we have to increase it to 120000

5. Download the Syndeia-Migration-3.3 utility-Utility from the Syndeia Cloud download page that you were provided with your Syndeia 3.3 licenses.

6. Download the Syndeia, 3.3 Standalone client from the Syndeia client download page that you were provided with your Syndeia 3.3 licenses.

7. The user who is doing the migration should be an admin user on Syndeia Cloud 3.2 and 3.3. Additionally, he the user should have the credentials of each external repository (server) because one of the steps requires that you run the Syndeia dashboard and connect to each of the repositories using your credentials. Also, this user should have permission to view all the data in any of the external repositories (servers)with permission to read all the data in that repository that was connected using Syndeia. For example, if Syndeia Cloud 3.2 has Syndeia relations to Jira issues, then the admin running the migration utility should have permissions to access the given Jira project and the issues. This is required during the enhancement phase of the migration.

...

Overview of Syndeia Cloud 3.2 → 3.3 Migration Process

The migration process works in the following stages:

1. Replicating (copying) data from the source (3.2) database to the migration (pre_stage) _* tables

2. Enhancing data in pre_stage_* tables by connecting to Slim (so that it is in a form which can be loaded in 3.3external repositories (e.g. Jira and Jama)

3. Staging data from pre_stage_* tables to migrate_stage_* tables

4. Loading data in from migrate_stage_* tables to target (3.3) database tables by calling the server using Syndeia Cloud API for data from migrate_stage tables.3.3 API

The details of the various stages are as follows:

• Replicate - first copy all data from the source (3.2) database to pre_stage_* tables (note that the 3.2 database has only 4 tables for repositories, containers, artifacts, and relations). Once the data has been copied from the 3.2 server to the pre_stage_* tables, we can safely disconnect from the 3.2 server and its role ends there. When the data is copied into the corresponding pre_stage_* tables, the value of the status column is set to -2.

• Enhance - In 3.3, we also have type related tables, which were not there in 3.2. Hence, in this step, we also have to connect to a running instance of Slim. Using this running instance, we can get external repository types and internal relation types (which is like hard-coded data, made up by us) which we first insert in pre_stage tables. Then for containers and artifacts, we need to connect to the actual repositories, get the type information for each of the container and artifact, and populate the corresponding pre_stage tablesvarious external repositories (e.g. Jama, JIRA, etc.) to get the “type” data for containers and artifacts in the external repositories. Once, all data has been enhanced, we modify the value of the status column to -1.

• Stage - Once all the data has been enhanced, we just insert all data from pre_stage_* tables to migrate_stage_* tables. During the insertion, we first update After the enhanced data has been copied from the pre_stage_* tables to the migrate_stage_* tables, the value of the status column to 0 in the pre_stage_* tables and then set this same value 0 in migrate_stage tables (because the final part of the migration process only picks up those records whose status is 0)_* tables is set to 0.

• Load - Load all the data from various migrate_stage_* tables (whose status is 0) , create the corresponding *CreateForm(s), or *UpdateForm(s) for each piece of data and call the relevant POST / PUT Cloud APIs on the Syndeia 3.3 serverto Syndeia Cloud 3.3 using its REST API.

...

Syndeia Cloud Migration Utility

A Syndeia-Migration utility is provided for the migration from Syndeia Cloud 3.2 to 3.3.

Syndeia Migration Utility

This utility is responsible for replicating data from Syndeia Cloud 3.2, enhancing the data, staging it for loading in Syndeia 3.3, and finally loading the data in Syndeia 3.3. This utility is a command-line application and accepts the following command-line arguments:

• -r (or --replicate), with the following possible values - None, Repositories, Containers, Artifacts, Relations, Users, AutoKeys or All

• -e (or --enhance), with the following possible values - None, Repositories, Containers, Artifacts, Relations, Users, AutoKeys, RepositoryTypes, ContainerTypes, RelationTypes or All (NOTE: artifact types are automatically enhanced when enhancing artifacts)

• -s (or --stage), with the following possible values - None, Repositories, Containers, Artifacts, Relations, Users, AutoKeys, RepositoryTypes, ContainerTypes, ArtifactTypes, RelationTypes or All

• -l (or --load), with the following possible values - None, Repositories, Containers, Artifacts, Relations, ContainerTypes, ArtifactTypes, Users, AutoKeys or All

The conf/application.conf file contained in this utility has to be supplied with the correct values for the source/target databases and whether the source server is LDAP enabled or not. It should also specify the Syndeia 3.3 server IP address, port and the port protocol (http or https) and the sign-in username and password for signing in to Syndeia Cloud 3.3. The username/password for Syndeia Cloud 3.3 should be for an admin account (preferably , this user should have admin rights so that he may create/update any data he likes). The relevant sections are -

...

super.user) that has ALL the permissions, including user management. The admin MUST have a local account in Syndeia 3.3 (basic auth, not LDAP to connect to Syndeia 3.3 and should be using that for the load steps of the migration process. The relevant sections of the conf file are:

source {
  // The Cassandra server used with Syndeia Cloud 3.2
  db {
    host = "localhost",
    username = "cassandra-username",
    password = "cassandra-pwd",
    port = 9042
    keyspace = "syndeia_cloud_devops"
  }

  // If the Syndeia Cloud 3.2 server is LDAP enabled
  server {
    isLdapEnabled = false
  }
}
source
target {
  // The Cassandra server used with Syndeia Cloud 3.3
  db {
    host = "localhost",
    username = "cassandra-username",
    password = "cassandra-pwd",
    port = 9042
    keyspace = "syndeia_cloud_devops"
  }
}

targetStore {
  // The Cassandra server used with Syndeia Cloud 3.3 (for auto keys)
  db {
    host = "localhost"
    username = ""
    password = ""
    port  isLdapEnabled= 9042
    keyspace = false"syndeia_cloud_store"
  }
}

// Syndeia Cloud 3.3 admin user credentials
sign-in {
  username = "super.user"
  password = "super.user.pwd"
}

// Syndeia Cloud 3.3 server host, port and protocol
server {
  ip-address = "localhost"
  port = 9000
  protocol = "http"
}

api {
  default-wait-duration = 2
}

Similarly, we need to update a few lines in the bin/syndeia-migration.bat file (or the bin/syndeia-migration file, for Linux / Mac). The following is an example of changes made to the bin/syndeia-migration.batfile; please do the similar changes in bin/syndeia-migration file.

set CFG_OPTS=-Xms1G -Xmx8G
set _JAVA_PARAMS=-Djava.library.path=lib -Dconfig.file=conf/application.conf
set "APP_CLASSPATH=%APP_LIB_DIR%\*;%APP_LIB_DIR%\com.intercax.syndeia-migration-3.3.jar"

Step-by-Step Migration Process

...

When you unzip the utility, you’ll notice empty folders called libMySQL, libTC & libWC. If you have a Syndeia license for these products and you’ve used Syndeia to create connections between data from any of these repositories, then you’ll need to copy the jar files required to connect to each of these in the respective empty folder. The details of which jar files are needed is mentioned in the below links -

Using Syndeia with MySQL

Using Syndeia with Teamcenter

Using Syndeia with Windchill

...

Step-by-Step Migration Process

Step 1 - Launch the Syndeia 3.3 Standalone client and connect to Syndeia Cloud 3.3 (using admin credentials)

When you launch Syndeia 3.3 Standalone client, you will need to provide credentials for your Syndeia Cloud 3.3 server. Click on the Settings button and specify the hostname, port, admin username, and password for your Syndeia Cloud 3.3 server. Click Apply.

...

Now, you will see a window for selecting or creating a Syndeia Cloud container (project). No containers are listed at this point because no data has been created in Syndeia Cloud 3.3 after installation. Create a new Syndeia container (project) for migration with the following values. This is a sample project which can be deleted later once this migration activity is complete.

• Key: MIGCONT

• Name: Migration container

• Description: Container for migrating data from Syndeia 3.2 to 3.3

… and press Create New button (as shown below):

...

Once you click Create New this Syndeia project should be created.

...

Step 2 - Replicate all data from Syndeia Cloud 3.2

bin\syndeia-migration.bat -r All

Step

...

3 - Enhance, stage, and load users

bin\syndeia-migration.bat -e Users -s Users -l Users

Step

...

4 - Enhance repository, container and relation types

bin\syndeia-migration.bat -e RepositoryTypes

...

bin\syndeia-migration.bat -e RelationTypes

Step

...

5 - Stage repository and relation types

bin\syndeia-migration.bat -s RepositoryTypes

bin\syndeia-migration.bat -s RelationTypes

Step

...

6 - Enhance, stage, and load repositories

bin\syndeia-migration.bat -e Repositories -s Repositories -l Repositories

Step

...

When you launch Syndeia 3.3 Standalone, you’ll see an empty screen because no Syndeia Projects exist yet. So create a Syndeia Project for migration with the following values:

Key: MIGCONT
Name: Migration container
Description: Container for migrating data from Syndeia 3.2 to 3.3, and press Create New button (as shown below).

...

NOTE: This is a sample project which can be deleted later once this migration activity is complete.

Once you click “Create New” this Syndeia project should be created.

...

...

7 - Connect to all the repositories using Syndeia Standalone client

Launch Syndeia standalone client (which is already pointing to the Syndeia Cloud 3.3 server). Double-click the MIGCONT Syndeia container (project) created in Step 1 and the Syndeia dashboard will launch. It will show all the repositories which were migrated from Syndeia 3.2 in the Repository Manager tab. Click on each of the repositories and you might be prompted to enter the password. If the prompt does not appear, then right-click each of the repositories and press “Update…” and enter your credentials there. Once you’ve entered the credentials for all the repositories, you should expand each of the repositories by clicking the “right-pointing triangle” icon. The repositories should look like the following:

...

The picture above shows the username which is used for connecting to each of the repositories.
NOTE: The user used to connect to any repository, must have access to data in that repository, otherwise that data won’t be migrated.

...

Close Syndeia Standalone client after connecting to all the repositories.

Step 8 - Enhance containers and stage container types & containers

bin\syndeia-migration.bat -e Containers

...

bin\syndeia-migration.bat -s Containers

Step

...

9 -

...

Load container types & containers

bin\syndeia-migration.bat -l ContainerTypes

bin\syndeia-migration.bat -l Containers

Step

...

10 - Enhance artifacts and stage artifact types & artifacts

bin\syndeia-migration.bat -e Artifacts

...

bin\syndeia-migration.bat -s Artifacts

Step

...

11 -

...

Load artifact types & artifacts

bin\syndeia-migration.bat -l ArtifactTypes

bin\syndeia-migration.bat -l Artifacts

Step

...

12 - Enhance, stage and load relations

bin\syndeia-migration.bat -e Relations -s Relations -l Relations

Step 13 - Enhance, stage and load auto keys

bin\syndeia-migration.bat -e AutoKeys -s AutoKeys -l AutoKeys

Troubleshooting

During the entire migration process, Step 2 (which is replicating all data from Syndeia 3.2) should hardly fail. Also, enhancing artifacts (Step 10) is the one that takes the most time. If for any reason, either the enhance or the stage steps fail due to network/internet issues, you can safely run them again. The only thing to be taken care of, is that you should try to load some data only after the replicate, enhance, and stage steps have been run, on that data in this order.

The Syndeia-Migration-3.3-Utility generates log files in the logs/ folder, so if something goes wrong, you may send those files to the Intercax team. The log files look like -

...