Database Trends 2024: The Power of Cloud, Consumption Models, and the Popularity of PostgreSQL

A large proportion of our customers rely on eG Enterprise to monitor and troubleshoot application and end-user experience problems caused by problems in underlying database dependencies. Our end-to-end unified monitoring and root-cause analysis platform supports all major database technologies. Over recent years, we have witnessed a significant shift from traditional on-premises databases to more dynamic, scalable solutions. As our customers’ database usage has evolved, we have naturally introduced support for newer database technologies as customers adopt them.

This evolution is characterized by the widespread adoption of cloud database services, the emergence of consumption-based models exemplified by Snowflake, and the rising popularity of databases such PostgreSQL. In this blog post, I’ll explore a few of the key trends we have seen in the past few years that we expect to see carry on into 2024. I will highlight the role of cloud platforms and services such as Amazon Aurora and Azure SQL, alongside the advantages of pay-as-you-consume models such as Snowflake and the versatility of PostgreSQL.

Database Trends 1. Cloud Storage and Infrastructure

The move from on-premises to cloud-based databases has become integral to modern data management strategies. Removing the overheads of maintaining physical infrastructure is a strong trend. Many organizations now have clear hybrid or cloud strategies and have moved some or even all their existing database operations into the cloud. Moving databases off physical self-managed on-prem servers to virtual servers running on VMs in public cloud (a lift and shift approach) is very common.

Learn more:

• You only have to look at the vast array of database options available via Azure to see this trend, see: Choosing Azure Database Services – What are the options?, for more details.
• Amazon AWS (Amazon Web Services) also offers a wealth of database options that have driven in to become a market leader as detailed in AWS’s Long, Steady Climb to the Top of the Database Market (substack.com).
• A recent survey by DZone for their September 2023 Database Trend report, cited that “Most organizations are using at least some cloud-based data storage solutions, with 41% of respondents saying their company stores all its data in the cloud and 33% saying their company uses a hybrid cloud approach. Only about a quarter of respondents (24%) said that their data storage was completely on-premises”. Interestingly this report also explored the drivers of cloud migration with the requirement “Maintain high availability” being the top reason users migrated to cloud, i.e. users were not simply looking to rid themselves of maintenance but actively seeking features intrinsic to cloud infrastructures.

Database Trends 2. Fully Managed Cloud Database Services

Platforms like AWS RDS, Azure SQL, and even the robust PostgreSQL in cloud environments offer organizations with scalable, managed solutions that eliminate the complexities of traditional on-premises setups. The implementation of the database itself, patching, security and so on become the responsibility of the database service provider freeing administrators not only of physical infrastructure maintenance but also maintaining the database software.

Benefits include:

• Scalability: Cloud databases facilitate seamless scaling of resources based on demand, ensuring optimal performance during peak usage.
• Managed Services: Cloud providers handle routine database management tasks, allowing organizations to focus on application development while maintaining data integrity.
• Global Accessibility: Cloud databases, including PostgreSQL in cloud deployments, offer universal data access, fostering collaboration in distributed environments.

Learn More About this Database Trend:

• The factors that influence the choice of a fully managed service such as AWS RDS are covered in AWS RDS vs EC2: How to choose the right hosted database
• Azure SQL offers on-prem Microsoft SQL Server users a number of routes to cloud both lift-and-shift and fully managed, the choices and factors that influence that choice are covered in, Azure SQL Deployment Options: Making the Right Choice.

Database Trends 3. Pay-as-You-Consume Models

Traditional database models often involved substantial upfront investments and the risk of underutilized resources. The advent of pay-as-you-consume models, as seen in platforms like Snowflake, has revolutionized cost structures by allowing organizations to pay based on actual usage. The direct link between usage and pricing is particularly useful for organizations delivering subscription-based apps to their customers as their costs scale with their user base and revenue stream.

Benefits include:

• Cost Efficiency: Pay-as-you-consume models eliminate the need for upfront investments in excess capacity, ensuring organizations only pay for the resources they use.
• Flexibility: Adaptable to changing data needs, these models provide flexibility without the commitment to fixed infrastructure.
• Performance Optimization: These pricing models encourage the optimization of queries and resource utilization for enhanced efficiency.

Learn more:

• We have added support to monitor and troubleshoot Pay-as-You-Use DB technologies such as Snowflake within eG Enterprise and have covered some details of this type of model plus some links where you can explore in more depth, see: eG Enterprise adds Advanced Performance Monitoring of Snowflake | eG Innovations and What is Snowflake (eginnovations.com).

Database Trends 4. PostgreSQL is Growing in Popularity

Amidst the evolving landscape, PostgreSQL stands out as a robust, open-source relational database management system known for its extensibility and adherence to SQL standards. Whether deployed on-premises or in the cloud, PostgreSQL continues to gain traction for its reliability, data integrity, and support for advanced features. Popularity matters to some extent when choosing DB technologies as it is usually far easier to find skilled staff and resources if a technology is widely adopted. Businesses, in general, are less enthusiastic about open-source solutions than developers, however PostgreSQL does stand out for the rate at which it is gaining traction in Enterprise.

Key features that attract many to PostgreSQL include:

• Extensibility: PostgreSQL’s extensibility allows users to define their data types, operators, and indexing methods, providing a high degree of customization.
• ACID Compliance: PostgreSQL adheres to ACID (Atomicity, Consistency, Isolation, Durability) principles, ensuring data integrity and reliability.
• Advanced Features: Support for JSON, full-text search, and spatial data types makes PostgreSQL versatile for a wide range of applications.

Learn more:

• The DB Engines Rankings are a monthly ranking of DB popularity and PostgreSQL now consistently features in the top 5 technologies listed, see: DB-Engines Ranking – popularity ranking of database management systems.
• Developers’ enthusiasm for PostgreSQL is explored in Postgres is the Undisputed Most Admired and Desired Database in Stack Overflow’s 2023 Developer Survey | EDB (enterprisedb.com)

Database Trends 5. Using Things That Aren’t Exactly Always Databases – such as Redis

I struggled to find a title to describe this trend well. Basically, we are seeing a lot of interest in our Redis monitoring technologies. In some of the DB rankings you will see Redis up in the top 10, in others completely absent. Its usage is certainly widespread. Redis (Remote Dictionary Server) is an in-memory data structure store that _can_ be used as a primary database but is often used in streaming workflows and other use cases as a cache or to facilitate messaging. Its popularity reflects the rise of microservice and distributed architectures that it is better suited to than many traditional DB technologies.

Understanding how Redis differs from traditional databases is a very useful mechanism to understand how modern application delivery is evolving. Our overview on Redis may help you understand this trend more, see: Redis – IT Glossary | eG Innovations.

Database Trends 6. On-prem Still has a Market

We still see plenty of on-prem DB technologies. The likes of IBM, SAP and Oracle are still doing a lot of business. Most surveys and reports show traditional on-prem technologies losing market share to cloud options. However, this reflects the fact that the overall DB market is growing, fueled by cheaper storage. Whilst most growth in the database market is now in cloud delivered technologies, the traditional on-prem market remains stable alibi somewhat static. For many organizations on-prem suits compliance requirements and data localization requirements for now.

One DB usage report that many place much weight on is the annual Gartner Report – Market Share Analysis: Database Management Systems, Worldwide, 2022 (gartner.com), and many analysts have covered the trends in these Gartner reports, see: New Gartner report shows massive growth in the database market, fueled by cloud | TechRepublic.

We certainly see a lot of customers looking to use eG Enterprise as their single console monitoring solution to cover all their DB technologies both on-prem and in cloud as they expand their DB footprint often adding newer cloud technologies alongside their on-prem databases.

Whether the newest on-premises cloud models such as Amazon Dedicated Local Zones will start nibbling at the traditional on-prem DB market will be one to watch.

Database Trends 7. AI and Big Data are Influencing DB Choices and Monitoring

AI and Big Data are influencing database technologies and user preferences. The advent of massive and diverse datasets associated with Big Data has driven users towards databases optimized for such data. NoSQL databases like MongoDB and Apache Cassandra are favored for their ability to manage unstructured data, aligning with the demands of Big Data applications. The need for real-time data processing in AI applications encourages the use of in-memory databases SAP HANA. The scalability demands of Big Data and AI workloads often leads users to opt for horizontally scalable databases like Apache Cassandra and Amazon DynamoDB.

These types of use case are also driving users to seek newer database monitoring and observability technologies which include AIOps (Artificial intelligence for IT operations) capabilities themselves. With dynamically scaling, heterogeneous and cloud-based database environments, traditional tools are no longer sufficient.

Database Trends 8. Containerization

Historically, we have always seen a lot of databases virtualized onto Virtual Machines (VMs) running on a hypervisor on-prem or on cloud VM instances. This is not surprising as eG Enterprise monitors most major cloud and virtualization technologies such as Citrix, VMware, Azure, AWS and so on.

We are increasingly seeing more databases run within containers. It is now very common to see MongoDB, Redis, MySQL and PostgreSQL deployed in conjunction with Docker, OpenShift, Kubernetes, VMware Tanzu and so on. Again, as eG Enterprise covers monitoring of these components it is not surprising we see those customers who do this. This is a relatively recent trend, as previously there were obstacles within many containerized / associated technologies, see Databases — Finally — Get Containerized – The New Stack for a reasonable overview. As such most of the well-established surveys and reports that poll cloud vs. on-prem usage have as yet to differentiate between containerized vs. virtualized vs. bare-metal usage.

For those looking for information on VMs vs. Containers, please see: Containers vs VM & Virtual Machines.

Learn more

• For details on eG Enterprise monitoring and diagnostics for databases, please see: Database Monitoring | eG Innovations
• For information on tuning databases for application performance, see: Slow Database Queries and Application Performance Tuning (eginnovations.com)