Cassandra Message Drops Test

Cassandra has a concept of back pressure. Back pressure is a technique used in staged event-driven architecture (SEDA) architectures in which, if a stage is already full of requests, it will not accept requests from earlier stages. As a result of back pressure, Cassandra will drop already timed-out requests without processing, and log an error. Since Cassandra is based on SEDA architecture, a request has to hop between multiple threadpools during processing, thereby increasing the latency. If too many message drops are noticed on the the Cassandra database node frequently, then, administrators have to check for overload conditions or figure out the real reason behind such performance lag at the earliest. To help the administrators, eG Enterprise offers the Cassandra Message Drops test.

The test auto-discovers the type of messages on the target Cassandra Database node and for each message type, reports the number of messages dropped and the time duration with which the messages were dropped from within the node and across the node. Using this test, administrators can figure out the messages of the message type that were dropped frequently and measure the maximum latency observed for the messages of each message type within the node and across nodes.

Target of the test : A Cassandra Database

Agent deploying the test : An external/remote agent.

Outputs of the test : One set of results for each message type on the target Cassandra Database node being monitored.

Configurable parameters for the test

Parameters	Description
Test Period	How often should the test be executed.
Host	The host for which the test is to be configured.
Port	The port on which the specified host listens. By default, this is 9042.
JMX Remote Port	Here, specify the port at which the JMX listens for requests from remote hosts. Ensure that you specify the same port that you configured in the cassandra-env.sh file (if the target Cassandra Database node is installed on a Unix host) or the cassandra-env.ps1 file (if the target Cassandra Database node is installed on a Windows host) in the <CASSANDRA_HOME> directory used by the target Cassandra Database node. To know how to specify the remote port, refer to Enabling JMX Support for JRE.
JMX User and JMX Password	If JMX requires authentication only (but no security), then ensure that the user and password parameters are configured with the credentials of a user with read-write access to JMX. To know how to create this user, refer to Configuring the eG Agent to Support JMX Authentication.
Confirm Password	Confirm the Password by retyping it in this text box.

Measurements made by the test

Measurement	Description	Measurement Unit	Interpretation
Dropped messages	Indicates the rate at which messages of this type were dropped during the last measurement period.	Messages/sec	Ideally, the value of this measure should always be 0. This measure is a good indicator of load on the database node. Messages are dropped when the internode messages received by a node are not processed within their proper timeout. Load shedding is part of the Cassandra architecture. If this is a persistent issue, it is generally a sign of an overloaded node or cluster.
Dropped latency across nodes	Indicates the time taken for the messages of this type to be dropped from across the nodes.	Milliseconds	Ideally, the value of this measure should be 0. Compare the value of this measure across message type to figure out the messages of which message type were more prone to be dropped from across the nodes.
Dropped latency within node	Indicates the time taken for the messages of this type to be dropped from within the node.	Milliseconds	Ideally, the value of this measure should be 0. Compare the value of this measure across message type to figure out the messages of which message type were more prone to be dropped from within the node.