Cassandra Commits Test

Cassandra processes data at several stages on the write path, starting with the immediate logging of a write and ending in with a write of data to disk:

• Logging data in the commit log
• Writing data to the memtable
• Flushing data from the memtable
• Storing data on disk in SSTables

When a write occurs, Cassandra database node stores the data in a memory structure called memtable, and to provide configurable durability, it also appends writes to the commit log on disk. Memtables and SSTables are maintained per table. The commit log is shared among tables. The commit log receives every write made to a Cassandra node, and these durable writes survive permanently even if power fails on a node.

The memtable is a write-back cache of data partitions that Cassandra looks up by key. The memtable stores writes in sorted order until reaching a configurable limit, and then is flushed. To flush the data, Cassandra writes the data to disk, in the memtable-sorted order.

When the memtable content exceeds the configurable threshold or the commit log space exceeds the commitlog_total_space_in_mb, the memtable is put in a queue that is flushed to disk. If the data to be flushed exceeds the memtable_cleanup_threshold, Cassandra blocks writes until the next flush succeeds. To reduce the commit log replay time, the recommended best practice is to flush the memtable before you restart the nodes. If a node stops working, replaying the commit log restores to the memtable the writes that were there before it stopped. Data in the commit log is purged after its corresponding data in the memtable is flushed to an SSTable on disk.

If the size of the commit log is inadequately configured against the commitlog_total_space_in_mb parameter, then, the data should be frequently flushed which in turn may increase the disk I/O manifold. This may cause performance bottlenecks on the target database node. Sometimes the growth of the commit log may also increase steeply. This may be primarily due to the write load on the node exceeding the ability to keep up with flushing the memtables. In the process, the older commit logs may not be deleted on time to accommodate additional data. To address these problems, it is necessary to monitor the commit logs of the target Cassandra Database node periodically. The Cassandra Commits test helps administrators in this regard!

This test monitors the commit log on the target database node and reports the current size and the growth rate of the commit log. In addition, this test reports how well the messages were written to the commit log and the rate at which messages were pending to be written to the disk from the commit log. The time spent to sync the data from the commit log to the disk is also monitored and reported. Using this test, administrators may be able to plan the capacity of the commit log based on the amount of data written to the commit log.

Target of the test : A Cassandra Database

Agent deploying the test : An external/remote agent.

Outputs of the test : One set of results for the target Cassandra Database node being monitored.

Configurable parameters for the test

Parameters	Description
Test Period	How often should the test be executed.
Host	The host for which the test is to be configured.
Port	The port on which the specified host listens. By default, this is 9042.
JMX Remote Port	Here, specify the port at which the JMX listens for requests from remote hosts. Ensure that you specify the same port that you configured in the cassandra-env.sh file (if the target Cassandra Database node is installed on a Unix host) or the cassandra-env.ps1 file (if the target Cassandra Database node is installed on a Windows host) in the <CASSANDRA_HOME> directory used by the target Cassandra Database node. To know how to specify the remote port, refer to Enabling JMX Support for JRE.
JMX User and JMX Password	If JMX requires authentication only (but no security), then ensure that the user and password parameters are configured with the credentials of a user with read-write access to JMX. To know how to create this user, refer to Configuring the eG Agent to Support JMX Authentication.
Confirm Password	Confirm the Password by retyping it in this text box.

Measurements made by the test

Measurement	Description	Measurement Unit	Interpretation
Completed	Indicates the rate at which the messages were written to the commit log since the start/restart of the database node.	Commits/sec	A high value is desired for this measure.
Commit log growth rate	Indicates the amount of data written to the commit log per second during the last measurement period.	MB/sec	A sudden/gradual increase in the value of this measure is a cause of concern. Sometimes the commit log can grow in an "out of control" fashion. Essentially, this can happen because the write load on the node exceeds Cassandra's ability to keep up with flushing the memtables (and thus, removing old commit log files). If administrators find a node with dozens of commit log files, and the number seems to keep growing, this might be a serious issue which needs to be addressed immediately.
Commit log size	Indicates the current size of the commit log.	MB	Ensure that the size of the commit log is adequately configured against the commitlog_total_space_in_mb parameter.
Pending	Indicates the rate at which commit log messages were written but are yet to be fsynced.	Commits/sec	A low value is desired for this measure.
Wait time on commit log fsync	Indicates the time spent for the data to be fsynced in the commit log.	Seconds
Commit log segment allocation wait time	Indicates the time spent waiting for a segment to be allocated on the commit log.	Seconds	Ideally, the value of this measure should be zero.