...
Kafka (via AWS MSK): you may be able to have a Kafka consumer run on another "backup node" (AWS EC2 or other) that just "listens" in the background logging all events put on the stream where you periodically archive off the logs generated per any backup tool of your choosing.
Note, the value of backing up the Kafka stream is debatable, as it is currently only used to convey the main SC data to the graphing subsystem. Theoretically one could replay the Kafka stream from any point to reconstruct the evolution of the graph over time but that isn't something we do yet. Also, the event sourcing system we use in the backend already supports thatCassandra (which is the DB for JG + Syndeia Cloud) :
WARNING: Confusingly, there are multiple ways DataStax/Apache Cassandra documents to backup/restore data, and each has pros/cons, they are listed in rough order of portability, management simplicity, and FOSS/PAID below:schema^ + CQLSH CSV Backup/Restore: Apache Cassandra CQLSH command, Intercax tested (see Backup & Restore Methods for Syndeia Cloud Keyspace in Cassandra )
Pros:
data is complete, ie: this is an entire backup vs an incremental,
data is very portable to DB even outside of Cassandra,
backup can work with any past, present or future Cassandra versions
Cons:
could potentially generate large (but compressible) files,
supposedly somewhat slow (may want to tweak CQLSH settings to avoid timeouts (see troubleshooting section))
NET = +1
schema^ +
dsbulk(CSV/JSON Backup/Export): DataStax provided accessory utility (see https://docs.datastax.com/en/dsbulk/doc/dsbulk/dsbulkAbout.html )Pros:
supposedly this is a faster performing CSV/JSON export,
Cons:
more complex, originally meant to be a tool to ingest in non-Cassandra data and not explicitly meant for backup/restores
NET = 0
schema^ +
nodetool snapshot(& optionally incremental backups^^) +nodetool refresh: Apache Cassandra included (see https://docs.datastax.com/en/archived/cassandra/3.x/cassandra/tools/toolsSnapShot.html , https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/operations/opsBackupIncremental.html , https://docs.datastax.com/en/archived/cassandra/3.x/cassandra/tools/toolsRefresh.html , https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/operations/opsBackupSnapshotRestore.html#Restoringfromlocalnodes , and https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/operations/opsSnapshotRestoreNewCluster.html )Pros:
almost instantaneous (as it creates a copy of the the live DB files),
Cons:
Cassandra version needs to be the same,
initial tokens process need to be backed up (see https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/operations/opsSnapshotRestoreNewCluster.html ),
process is node-specific and
snapshot/refreshneeds to be run simultaneously/atomically on,... multiple (potentially all) nodes
NET = -3
schema^ + token backup +
nodetool snapshot(& optionally incremental backups^^) +sstableloader: Apache Cassandra included (see https://docs.datastax.com/en/archived/cassandra/3.x/cassandra/tools/toolsSnapShot.html , https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/operations/opsBackupIncremental.html , https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/tools/toolsBulkloader.html?hl=sstableloader#toolsBulkloader__bulkloader-restoring-snapshots , https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/operations/opsBackupSnapshotRestore.html#Restoringfromcentralizedbackups , and https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/operations/opsSnapshotRestoreNewCluster.html )Pros:
almost instantaneous (as it creates a copy of the the live DB files)
don't have to deal with backup of token ranges,
can deal with a different number of nodes or replication strategy.
Cons:
Cassandra versions need to be the same,
process is node-specific and
snapshot/refreshneeds to be run simultaneously/atomically on,multiple (potentially all) nodes at the same time (depending on the replication factor)
NET = 0
schema + cassandra-data-copy-tool: 3rd-party FOSS tool https://github.com/wildengineer/cassandra-data-copy-tool , could use it as a way to periodically create a "cold" backup by just streaming from live to backup cluster
Pros:
Simplifies cluster-to-cluster copying,
The source and destination tables do not need to be on the same cluster or keyspace. All you need to ensure is that the destination table is compatible with the source table.
Cons:
Requires second cluster,
Unknown atomicity (presumably cluster is streamed to target)
NET = 0
Medusa: 3rd-party FOSS tool https://github.com/thelastpickle/cassandra-medusa
Pros:
schema automatically handled?,
tokens automatically handled?,
Cluster wide in place restore (restoring on the same cluster that was used for the backup),
Cluster wide remote restore (restoring on a different cluster than the one used for the backup),
Support for Google Cloud Storage (GCS) and AWS S3 through Apache Libcloud (can be extended to support other storage providers supported by Apache Libcloud)
Cons:
Cassandra deployments with multiple data folder directories not supported yet
Still somewhat beta-ish but endorsed by Apache Cassandra
NET = +3
DataStax OpsCenter™: 2nd-party PAID tool https://docs.datastax.com/en/opscenter/6.1/opsc/about_c.html (see also https://www.youtube.com/watch?v=HLy1rV4BTT8 )
Pros:
very simple to use GUI
handles automatic snapshotting across all nodes simultaneously
Cons:
Not compatible with OSS Apache Cassandra or DataStax Distributions of Apache Cassandra™ (DDAC) clusters, ie: requires migrating to a DataStax distribution of Cassandra
Not free
NET = 0
Datastax Astra™: 2nd-party PAID tool (see https://www.datastax.com/products/datastax-astra/pricing )
Pros:
80GB Free-tier
Cloud-based (you choose: AWS, GCP, Azure)
Reduced on-site h/w IT maintenance
Can scale on-demand
Cons:
Not free after free-tier
Somewhat complex metered rates based on credits ($1 = 1 credit)
Cloud-based (may not be viable for some security-sensitive deployments)
Credits expire after 1-year
Net = 0
...