English
Deutsch
Espanol
Français
Portuguese
中文
한국어
日本語
Snowflake monitoring and performance management
- Proactive monitoring and rapid diagnosis
- Optimize storage usage to control costs
- Monitor query performance and much more
Trusted by leading companies
What is Snowflake?
Snowflake is a cloud-based data warehousing platform that provides data storage, analysis, and reporting. It is designed to be highly scalable, secure, and highly available, and supports a wide range of data types and sources.
Snowflake’s cost-per-usage model is revolutionary in the cloud database ecosystem and good tools to optimize performance and costs including those associated with cloud platform, storage and K8s dependencies are essential.
Snowflake uses a unique architecture that separates storage and computation, allowing for high levels of performance and scalability. It also provides a SQL-based interface for querying data, making it easy for analysts and data scientists to work with. Key features are:
- A fully managed SaaS (software as a service) that provides a single platform for data warehousing, data lakes, data engineering, data science, secure sharing and consumption of real-time / shared data.
- Out-of-the-box separates storage and compute, has on-the-fly scalable compute, data sharing, data cloning, and third-party tools support.
- Enables data storage, processing, and analytic solutions that are faster, easier to use, and far more flexible than traditional offerings.
- A Snowflake database is where an organization’s uploaded structured and semi-structured data sets are held for processing and analysis. Snowflake automatically manages all parts of the data storage process, including organization, structure, metadata, file size, compression, and statistics.
How eG Enterprise meets key
Snowflake monitoring requirements
eG Enterprise’s integration with Snowflake enables complete visibility into the Snowflake architecture and operations, alongside any dependent cloud hosted infrastructures. Leveraging AIOps technologies, eG Enterprise offers an observability platform to help you use Snowflake at scale and assure performance, availability and cost optimizations. With eG Enterprise you can:
- Quickly pinpoint costly and slow queries and drill into precise execution details to address bottlenecks
- Optimize the storage requirements by looking at the usage patterns. Identifying spikes in storage used and large files being loaded inefficiently
- Monitor the credit usage and take necessary actions where cost can be optimized
- Detect any misconfiguration and security threats and proactively resolve them
- Monitor the performance of the data warehouse and understand if the warehouse can satisfy user requirements. Predict when to resize the warehouse
- Know if replication is enabled, and track lag between instances to get early insights into problems with replication
- Get alerts out-of-the-box from AIOps driven anomaly detection to rapidly detect deviations from historical patterns or fluctuations of key resources such as storage
- Use rich dashboards, topology maps and reports to break down communication silos across the organization
Monitor Snowflake credit usage across your organization
eG Enterprise allows you to monitor your Snowflake costs in many ways with fine granularity, allowing you to ensure Snowflake credit usage is managed and understood. You can:
Improve Snowflake warehouse performance and ensure availability
Deep insight into queries and query workloads
Query performance is key to any database or data store. Many factors can affect the performance and throughput of queries like the load on the target database, query complexity, blocked queries, etc. If the query performance is not acceptable, it will directly and immediately impact the performance of customers’ applications using Snowflake database and in the worst case might disrupt business critical services.
It is important to monitor each database under the client’s Snowflake service account to provide insights into query workload. eG Enterprise monitors each Snowflake database and provides vital statistics related to query execution like average and maximum time taken for query execution, how long the queries are being queued before execution, for how long were transactions blocked, etc. Administrators can look at these metrics and get to the root cause of degradation in query performance and clear out the blocked queries and transactions.
Single click detailed diagnostics allow instant visibility on key data such as the top queries by execution time, details of failed queries and more.
Observability on Snowflake data load and Snowpipe
Snowflake data loading is the process of copying and loading data or data sets from a source file, folder or application to a database. It is implemented by copying digital data from a source and loading the data to a data storage or processing utility.
Snowpipe is Snowflake's continuous data ingestion service. Snowpipe loads data within minutes after files are added to a stage and submitted for ingestion.
eG Enterprise allows you to proactively and continuously monitor data loading using COPY INTO (bulk loading) and pipes (continuous loading) within Snowflake. In the event of failures, detailed diagnosis information on error details is instantly available.
Monitor Snowflake tasks
When handling large and complex data pipelines with tasks or task trees, not all of them may execute SQL statements as expected and some may fail.
Snowflake includes a table function named TASK_HISTORY, which gives the task usage detail for your entire Snowflake account or a specific task. This TASK_HISTORY table function exists under the information schema where the task or task tree is created.
eG Enterprise automatically keep track of the Snowflake tasks and give you visibility into their operation with proactive alerting on failures and instant access to data from TASK_HISTORY with no need to write or action bespoke SQL queries. Detailed diagnostics on the details of failed or skipped tasks are accessible in a single-click.
Live and historical reporting allows you a comprehensive overview of task usage and success over longer periods.
Integrate Snowflake observability across the enterprise
eG Enterprise allows you to integrate your Snowflake monitoring into automated workflows and best-practice business workflows. Designed for users across the enterprise beyond the Snowflake Administrator including L1/L2 help desk operators, business management and cloud architects. eG Enterprise includes:
Get detailed cloud and Snowflake usage and cost insights in a single platform
eG Innovations supports over 650+ technology stacks, including most database technologies and public clouds such as Microsoft Azure and Amazon AWS. eG Enterprise allows you to:
Frequently asked questions (FAQs) about Snowflake monitoring
Snowflake is designed through three main components:
Cloud services: Snowflake uses ANSI SQL for cloud services empowering users to optimize their data. It eliminates the need for manual data warehouse management and tuning.
Query processing (compute): The compute layer of Snowflake is made up of virtual cloud data warehouses that let you analyze data. Each Snowflake virtual warehouse is an independent cluster and they do not compete for computing resources nor affect the performance of each other — which means workload concurrency is never a problem.
Database storage: The database storage layer holds all data loaded into Snowflake, including structured and semi structured data. Snowflake automatically manages all parts of the data storage process, including organization, structure, metadata, file size, compression, and statistics.
See: Key Concepts & Architecture — Snowflake Documentation for more detail.
Snowflake is provided as a self-managed service that runs completely on cloud infrastructure. This means that all three layers of Snowflake’s architecture (storage, compute, and cloud services) are deployed and managed entirely on a selected cloud platform.
A Snowflake account can be hosted on any of the following cloud platforms:
eG Enterprise monitors Snowflake in an agentless manner. A read-only monitoring user queries table views to monitor the health of the Snowflake databases.
Snowflake is a cloud-based relational database management system (RDBMS) that supports the SQL language. It is a columnar database, meaning it stores data in columns rather than rows, allowing for more efficient data compression and faster query performance. Snowflake also incorporates elements of NoSQL databases, such as flexible data modeling and semi-structured data storage, making it a hybrid of both traditional relational and NoSQL databases.
Snowflake is a data platform and data warehouse that supports the most common standardized version of SQL: ANSI. This means that all of the most common operations are usable within Snowflake. Snowflake also supports all of the operations that enable data warehousing operations, like create, update, insert, etc.
Snowpipe is Snowflake's continuous data ingestion service. Snowpipe loads data within minutes after files are added to a stage and submitted for ingestion. With Snowpipe's serverless compute model, Snowflake manages load capacity, ensuring optimal compute resources to meet demand.
A list of over 650+ stacks supported by eG Enterprise is available, see: End-to-End Monitoring: Applications, Cloud, Containers (eginnovations.com). All major, on-prem and cloud service, databases are supported as well as many database related products such as Redis. Information on some of the most popular options can be found via our Database Monitoring page, see: Top Database Monitoring Tools | eG Innovations.
