The @Dharmesh Test: 16 Questions For Better SaaS Companies

The Dharmesh Test is a quick, simple set of yes/no questions to gauge the likelihood of long-term success of a SaaS (Software as a Service) company.  It is inspired by the highly popular “Joel Test” for measuring software development teams.  [Disclosure: I am an angel investor in Joel’s company Stack Overflow]

Now, if you know me, and you know Joel, you know that I’m no Joel.  He’s smarter and is a better writer.  So, why did I have the audacity to write the “Dharmesh Test”?  Because I’m passionate about SaaS companies and have spent the better part of the last 6 years studying them and learning from folks that should have written this article.  And, I’ve taken what I’ve learned and applied it to my own company, HubSpot — which is doing very well and scores a respectable 13/16 on the test (yes, I’ve got work to do).

This is alpha Version 0.80 of this test.  Will be iterating on it based on community feedback.

The Dharmesh Test: 16 Questions For Better SaaS Companies

1. Is there exactly one version of your software that services all users?

Or stated differently, have you resisted the temptation to have custom code for an individual customer or small group of customers?  Ideally, in steady state, the same master code-base would be servicing all customers.  Any customizations are done through application configuration or via APIs and plugins.  If you have different versions running temporarily to test new versions with a subset of customers, that’s fine.

2. Do you have a free trial?

Not only does a free trial increase your likelihood of getting more customers, it often improves the simplicity of the product.  Many companies can’t have free trials because the product is just too complicated and requires too much human assistance to really “try” it. That’s a bad thing.  Freemium (where you have a free version of your product that’s feature-limited, not time-limited) counts too. 

3. Do you track key metrics like CAC, LTV and cancellation rate?

CAC = Customer Acquisition Cost. This is the total sales expense and marketing expense (salaries, programs, etc.) over a specific period (lets say a month) divided by the number of customers acquired in that month.

Cancellation Rate / Churn Rate:  A percentage calculated as the number of customers that cancel in a month divided by the number of customer at the beginning of the month.

LTV =  Lifetime Value of a customer.  Based on how long the average customer is expected to stay (a function of cancellation rate) and the average revenue.

The reason these numbers are important is simple:  They are the key metrics that will help you drive a SaaS company.  It answers questions around how much capital you need, how much you should be investing in product, whether customers are generally happy (or not), etc.

Example:  KissMetrics gets this stuff (because that’s what they do.  And, if you’re not yet reading David Skok’s For Entrepreneurs blog, you should be.  Brilliant material on this topic.

4. Are your prices published on your website?

It is decidedly sub-optimal if people have to get a custom quote or talk to a human just to know what your price is.  Increasingly, the market values simplicity and transparency. If you’re not providing easy access to information, potential customers will go elsewhere.

5. Can people start paying you (become a customer) without interacting with a human?

The way of the world is frictionless web sales.  There’s nothing quite like the joy of having your revenues go up at 4:00 a.m. in the morning on a Saturday — without anyone having to talk to anyone.

6. Do you make more money from users that are getting more value?

Is there some “variable” component to your pricing?  Do you charge based on number of users, storage, frequency of use or some other metric?  If you have the ability to make more revenue from customers over time (without having to upsell them), that’s high margin business and very desirable.  By the way, the ideal state is when there are “in product” upgrade paths.  Customer reaches a certain limit and “upgrades” (starts paying you more) without you having to do anything.

7. Does your application have a simple, clean API (Application Programming Interface)?

The web is not just about connecting static websites — it’s now about connecting applications.  Without an API, your application is an island.  It’s unlikely that you will be able to meet all the needs of your customers.  An API allows you to integrate with third-parties (and they to you) to deliver a more complete solution for the customer. These kinds of integrations are also great ways to spread the word about your product (the more people that integrate with you, the more reach you have).

8. Can customers export their data out of your system?

To get more people to use/buy your product, you need to reduce not just the barrier to entry, but also the barrier to exit.  The product should be what keeps customers to stay with you — not data lock-in.  If you try to use data lock-in, fewer people will pick you in the first place.

9. Can you deploy (and rollback) a new version of the software in one-step without downtime?

This is to measure the effectiveness of your technical operations and software release tooling. Companies that have easy ways to deploy/rollback software tend to have more confidence and release more often.  Companies that release often tend to do better than those that don’t.

10. Is there an online community where users can interact with your team and with other users?

Ideally , everyone in your company should be interacting with customers.  Especially developers.  And, customers should be able to talk about you (even if it’s negative) online and interact with other customers.  [Note: I’m not talking here about “automatic” online venues that form — but something that you are actively fostering and encouraging]

11. Do new developers commit code on their first day at work?

This is partly to measure how difficult your development environment is to setup and what controls you have in place.  If it takes 4 days for a developer to get their development environment setup, something’s wrong.  If you’re too scared to have a rookie commit code to production, because it might bring the entire system down, something’s wrong (you haven’t built a strong enough safety net yet).  It shouldn’t be that easy to break your system.  [Nod to Eric Ries for this one].

Example:  Cheezburger Network doesn’t show new employees where the bathroom is until they’ve committed code.

12. Do you release code to production at least once a month?

This is to check whether or not you’re agile.  The challenge these days is that everyone thinks they’re agile, so asking “are you agile” is ineffective.  The proof is in the pudding.  Real agile teams release code — frequently.  Everything else is secondary.

13. Do you maintain a centralized backlog of new product ideas, bugs and issues?

Ideally, not only would you have a central database of ideas, but you’d provide the ability for customers to submit ideas and vote on other people’s ideas. 

14. Do you regularly run A/B (split) tests?

Great SaaS companies are great at testing and deciding with data.  You should be running regular tests (as in several times a month).  Example tests: Should we remove this feature (does anyone care)?  Should we increase our price to $75?  

Example: A company that does this really well is FreshBooks. If you run into Mike McDerment, ask him about how they built their product to facilitate testing.  (He probably won’t tell you, but doesn’t hurt to try).  I also think SEOmoz (hi Rand!) does a great job at this.

15. Do you invest in design and user experience?

Customer expectations (even for B2B companies) are high on the web.  Having crappy design might be OK if you happen to have a product that people can’t help but use — but most of us are not in such forgiving circumstances.  Great design not only helps you attract customers, it helps you keep them — and get referrals.  Profitable design is when you spend effort in design that is centered around your business objectives. 

Examples:  I like Campaign Monitor a lot.  Heck, even their API documentation site is beautiful.  But there are other examples like the ever-popular 37signals

16. Does your company or product have a personality?

This is the squishiest of the questions — but it’s important.  Great software companies today have a personality and a voice.  They’re distinctive and remarkable.  Whether it’s through their blog, through the copywriting in the product (like error messages), or through in-person interactions, some companies just have personality.

Examples: Take a look at what Wufoo or ZenDesk does.  Quirky?  Yes.  Effective?  Absolutely.

———-

So, how did you do?  Any questions you think are unimportant and should have been left out?  Any you’d add? 

Looking for other startup fanatics?  Request access to the OnStartups LinkedIn Group.  130,000+ members and growing daily.

Oh, and by the way, you should follow me on twitter: @dharmesh.