Before adopting any new SaaS (Software-as-a-Service) business tools, the IT department of an organization should evaluate whether it is wise to trust the SaaS product and indeed vendor. Five key questions an organization needs to know answers to before becoming reliant on a third-party SaaS application are:

  1. Can this SaaS product meet my data compliance requirements?
  2. How likely is the SaaS vendor / product to exist this time next year?
  3. Is the SaaS product secure?
  4. Are the terms and conditions of the contract sane?
  5. Is it supported? And is the support any good?

Who decides if a SaaS product is fit for purpose?

Before you even start trying to evaluate a SaaS product, an organization should ensure the correct people are making that decision, and that’s usually the IT team. The “Consumerization of IT” means that SaaS tools are frequently adopted as “Shadow IT” by individuals or non-IT teams, and often without proper answers to those key questions. Basically, everyone in an organization needs to understand that adopting a new SaaS tool needs to be done with the consent of the IT team. The rest of this article is written from the perspective that it will be an IT team or suitably qualified individual evaluating a SaaS tool.

Question 1: Can this SaaS product meet my data compliance requirements?

SaaS means the software is running on some server, in some datacenter, somewhere. Often SaaS is hosted on a big public cloud such as AWS, Microsoft Azure or Google GCP but often only in certain geographical data centers. It’s not uncommon for a SaaS product to be hosted only in a cloud data center located in the USA. For many customers this isn’t a problem, their business and the regulations they need to comply with mean that they don’t care where a SaaS tool is hosted.

However, for a significant number of organizations in regulated industries or certain geographies they must adhere to compliance and data localization regulations. Some businesses have clients who stipulate “data must not leave the country” in contracts or insist that SaaS is only delivered from a FedRAMP cloud. If a particular SaaS tool doesn’t meet your data location requirements, it’s a non-starter for your enterprise.

Question 2: How likely is the vendor / product to exist this time next year?

In my opinion, this is the next question you should ask, if the answer is unpalatable then no matter how good the product actually is, nor how good the answers to the other questions are – if there even a moderate chance a SaaS product won’t exist long term then it is probably more trouble than it is worth.

The statistics show that most SaaS startups are doomed to fail. CB Insights published a report analyzing the most common reasons for startup vendor failure, see: The 20 Reasons Startups Failed there are lots of other surveys and reports that converge on a failure rate of around 90% for startups.

Often these failed products simply fail to find an audience in a crowded market but often venture capital funding and cash-flow issues are contributing reasons for failure.

Venture capital fueled businesses are often looking for a buyer and many a hot SaaS tool has been acquired by a larger organization who has either absorbed the technologies into their own products or killed off the acquired product. The large tech giants certainly are not averse to killing off acquisitions or even their own products, as the site Google Graveyard – Killed by Google demonstrates.

So, you should ask questions about the vendor and product?

  1. How long has this product been around?
  2. Is the product profitable? Does it have a good established customer base funding the product?
  3. How long has the company been around?
  4. Is this product venture capitalist funded? How will they pay the venture capital back? Do their revenues / profits come close to covering the funds invested?
  5. Are they looking to be acquired?
  6. Have there been concerning layoffs?
  7. Whose datacenter / cloud are they using and what happens to your data if the SaaS provider goes bust?

There’s plenty of community advice and experience available, see:

Case Study: Code Spaces – a code hosting and software collaboration platform

This is a particularly cautionary tale of what can go wrong both for the SaaS provider and for their end customers, the type of case study that should be included in every undergraduate computer science degree course.

Code Spaces provided managed source code hosting for companies developing code and products, hosted on Amazon Web Services (AWS). The offering was attractive to many small and medium size code houses, they got the security and reliability of the trusted brand of AWS without the need to get up to speed on AWS via Code Spaces who put together a solution tailored to their use cases.

However, Code Spaces’ expertise in practice didn’t quite stretch to best in class security for AWS and hackers took control of Code Spaces’ AWS account and when they didn’t pay the ransom, the hackers deleted their storage and backups – blowing away their customers code and backups of it. Code Spaces rapidly went bust but the ripples impacted many of their customers who had often lost vast amounts of their products’ code base. A few of those customers went bust too, many lost man years of work, most had some sort of copy of some of their code somewhere and spent months patching it back together.

There were many detailed postmortems on the details of this vendor’s demise, see:

A key factor in the Code Spaces case was the failure to adhere to AWS’s IAM (Identity and Access Management) best practices. AWS Identity and Access Management (IAM) is a service that allows you to manage user identities and their permissions for accessing AWS resources and services securely. But how could a potential customer of Code Spaces know this? – More on that later!

Question 3 – Is the SaaS product secure?

In the case of Code Spaces, their security problems led to the product and vendors near instant demise. Whilst some may not be too bothered about a particular product’s data security, the prospect of losing that product overnight is a different and significant secondary risk. Many customers had signed-up because they were reassured by the solidity of the underlying AWS platform and backups it provided for Code Spaces. Whilst clouds such as AWS and Azure can run SaaS applications very securely, whether the vendor is doing so is a different matter.

External Audits and Cloud Verification Programs

The Code Spaces incident was a few years ago and thankfully there is a lot more help from those cloud giants these days, who somewhat recognize that have some responsibility for all those third-party SaaS offerings on their marketplaces being fit for purpose. The best of these partner accreditation programs, and application validation services can offer IT teams a lot more information on which to make data driven decisions when evaluating SaaS tools and their vendors.

Microsoft Defender for Cloud Apps

Microsoft Defender for Cloud Apps detects all your cloud services, assigns each a risk ranking, and also identifies all the users and third-party apps able to sign in. A core part of the service is a “Risk Score” from Microsoft Defender for Cloud Apps cloud app catalog of over 31,000 cloud apps. The apps are ranked and scored based on more than 90 risk factors to provide you with ongoing visibility into cloud use, Shadow IT, and the risk Shadow IT poses to your organization.

It’s worth reviewing how the Risk Scores are calculated to understand this service. With over 31,000 apps there is a large amount of automation and some self-certification by app vendors involved. It collates a lot of useful background about each app and vendor such as when the vendor was established, whether they comply to standards such as GDPR and SOC 2 and so on, plus information on individual security features – see: Working with the risk score – Microsoft Defender for Cloud Apps | Microsoft Learn. Generally, if the service hasn’t yet collated a lot of data on app it can rate it with a very low score – this doesn’t necessarily mean the app is dodgy, but it should trigger further investigations. Microsoft also regularly change the evaluation criteria meaning a vendor may have submitted their product prior to a category being added to the score, so it really is questioning missing security features or compliance coverage as highlighted by Defender for Cloud Apps.

Another caveat to consider when working with the Defender Risk Scores is that most categories are binary – a simple yes/no e.g. are “User roles supported”? This tells you nothing about whether the roles available or role configuration will actually allow you to restrict access to functionality and data at the best granularity for operating a business on a strictly “needs to know” basis.

AWS Foundational Technical Reviews, Qualified Software, Well-Architected

Amazon have a lot of vendor validation processes and programs in place, which these days would allow a customer evaluating a Code Spaces like product to make a much better assessment. Indeed, if these programs had been in place back then and Code Spaces had participated it is very likely that their flawed IAM usage and security flaws would have been identified and rectified before the hackers got it.

The AWS Foundational Technical Review (FTR) is a self-service program whereby SaaS vendors can check they are using best practices from the AWS Well-Architected Framework. It’s really worth familiarizing yourself with the criteria (particularly those pertaining to security) as it provides specific questions an IT team can ask a vendor to confirm on the architecture of their product. Of course, self-service programs provide only limited assurance and so AWS have implemented the AWS Competency programs which involve AWS architects and engineers examining the architecture and security of products as appropriate.

Having seen the program from the vendor side, it was certainly thorough and invasive in its investigations. More details on what was involved including the audits, here: eG Innovations achieves Amazon Web Services (AWS) Digital Workplace Competency status | eG Innovations. Going through an audit like this allows us as a vendor to carry the AWS “Qualified Software” badge which offers those using our AWS SaaS hosted monitoring assurance we have adhered to AWS best practice.

SOC 2 Type Audits

SOC (System and Organization Controls – formerly Service Organization Controls) audits are an independent assessment of the risks associated with using service organizations and other third parties. SOC 2 audits assess service organizations’ security, availability, processing integrity, confidentiality and privacy controls against the AICPA’s (American Institute of Certified Public Accountants) TSC (Trust Services Criteria), in accordance with SSAE 18. A SOC 2 report is generally used for existing or prospective clients. Whether an organization is SOC 2 audited is covered in Microsoft Defender for Cloud Apps Risk Score criteria.

Multi-tenant vs dedicated SaaS

Multi-tenancy is an architecture in which a single instance of a software application and its underlying resources serves multiple customers, each customer is called a tenant. Multi-tenant architectures are the foundation of most SaaS offerings. However, many SaaS products will also offer a premium option to be a sole tenant to ensure your data is stored in its own dedicated instance of the product. For the most security conscious or regulated multi-tenant solutions may not be appropriate or permitted.

If you opt for a dedicated SaaS instance you may well choose to run your own scans or penetration tests if the nature of the product makes such testing appropriate.

Question 4 – Are the terms and conditions of the contract sane?

You can often get a very good feel for the reliability of SaaS supplier via the credibility of the clauses in their terms and conditions. Some are so outstandingly flaky that it’s hard to believe any IT team or finance team would sign-off on the products use. It’s not unusual for a SaaS Vendor to retain the right to change what you get for your contract or reprice whenever they feel like it. Free and lower tiers are often particularly low on any kind of commitments – but what do you expect!

One popular project management application – let’s call it “XXXXX” to spare their embarrassment, lists Terms of Service including:

  • Your use of the Service is at your sole risk. The Service is provided on an “as is” and “as available” basis.
  • XXXXX does not warrant that (i) the Service will meet your requirements or expectations, (ii) the Service will be delivered uninterrupted, timely, secure, or error-free, (iii) the results that may be obtained from the use of the Service will be accurate or reliable, (iv) any errors in the Service will be corrected.
  • Technical support is provided on a best-effort basis and by e-mail.

So little reassurance! I half-wondered if even the vendor believed their product would be around next year! And to top it all they threw in the additional clauses:

  • Prices of all Service features are subject to change. If they change, the changes do not affect running subscriptions, but will take effect only for any subsequent subscription period.
  • XXXXX is not liable to you or to any third party for any price change.

Topped only by a final cover-their-butts clause:

  • XXXXX reserves the right to update and change the Terms of Service at any time without notice.

Evaluating the likelihood of hefty price hikes (remember most SaaS billing is monthly) is another wise step when considering if you should trust a SaaS product / vendor, see: Do You Trust Your SaaS Vendor? | LinkedIn.

It is always worth checking whether the contract covers a Disaster Recovery (DR) policy and what the commitments are around that, particularly if the loss of the data in the app would have a significant impact on the business.

Contracts are of course essential just bits of paper – no matter how watertight Code Spaces contract with its customers may have been, once the hackers deleted their storage, they simply no longer had a service or product to offer.

The fact that the above clauses are associated with what is a popular tool with a reasonably large user base is probably a testament to the points I made around Shadow IT and the Consumerization of IT. No credible enterprise IT team would have authorized this tool’s usage for a business-critical function on these terms. Good application user and employee browsing/website usage monitoring can help organizations reduce the risks by identifying Shadow IT in use and is one use case we see our customers using eG Enterprise for.

Question 5 – Is it supported? And is the support any good?

As we saw above, some SaaS products have very flaky support commitments. It is not just about the software, everything will fail at some point, and you need to ask how well the SaaS provider will support you. Support commitments and support SLAs are very important contract terms. Amazon’s Digital Workplace Competency program insists that vendors provide support but also have a support contract in place with Amazon themselves to ensure the end customer is fully supported.

Checking whether a product is contractually supported is one thing, receiving the level of support promised by the contract is another thing altogether. We all know of organizations whose support in practice is a lot less useful than a chocolate teapot. Do your own due diligence and research the reputation of the support offered – read reviews, check out social media interactions with the product’s accounts and if possible, undertake a trial or PoC including support interaction.

eG Enterprise SaaS

eG Enterprise is offered as fully managed SaaS service in addition to on-prem and cloud self-hosted options. Our SaaS offerings are currently available in several geographic AWS regions, including Australia, Singapore, Europe (Germany), and USA; enabling customers to comply with government and industry regulatory requirements, such as the European GDPR, and the Australian Privacy Act.

Assurances that we offer customers include:

Offering a SaaS monitoring product, we of course also get asked specific questions associated with secure monitoring architectures for cloud such as whether we need to open new ports across firewalls etc. Any SaaS product that collects data from your organizations or facilitates remote access to your organizations will need additional scrutiny.

eG Enterprise is an Observability solution for Modern IT. Monitor digital workspaces,
web applications, SaaS services, cloud and containers from a single pane of glass.

Free Trial  See the platform

About the Author

Rachel has worked as developer, product manager and marketing manager at Cloud, EUC, application and hardware vendors such as Citrix, IBM, NVIDIA and Siemens PLM. Rachel now works on technical content and engineering and partner liaison for eG Enterprise