The Matrix Foundation is in charge of various activities revolving around Matrix. One of the most important activities to make Matrix a mainstream protocol is lowering the barrier to entry for the general public. The Matrix.org website is a critical step of the onboarding: this is what people will stumble upon when they look up “what is Matrix chat” or “chatting on Matrix” in a search engine.

When the general public stumbles upon a page on a topic they’re not familiar with, they need to quickly find valuable information or they will move on to their next task. This is not only in true in a marketing context: it is also true when trying to onboard someone. High quality, clear, focused pages help users navigate their way to a platform.

Producing such pages is an iterative process: it requires trial and error, testing, and above all one or several ways to figure out whether the changes work or not. One excellent way to do it is by conducting user research. Unfortunately this is also an expensive method: it requires to conduct the interview, and time to make sense out of the answers.

The Foundation doesn’t have the resource to tackle user research on the matrix.org website for now, although it would love to do it eventually. Fortunately, there is a much more affordable solution: web analytics.

Analytics are a kitchen knife

Analytics are usually a hot topic in privacy-minded spheres, and rightfully so. Reckless usage of analytics is not only plain creepy: it’s a surveillance enabler and privacy disaster. Giving away people’s information when they browse matrix.org would be unethical and a massive betrayal of our audience.

What makes analytics dangerous is the surveillance side of it: the possibility to build a user profile, to know who in particular is doing what. The various scandals that took place by weaponising analytics must not make us throw the baby with the bathwater though.

A screenshot from The Simpsons. A person says "Oh won't somebody please think of the children?"

Analytics can be useful to product makers all while being respectful of people using them. It is valuable for a website maintainer to know if a page is never seen. It is valuable to know if a link is never clicked. Analytics are akin to kitchen knifes: just because they can be used for nefarious purposes, they shouldn’t be banned entirely. Their use must be ethical. I have defined minimum requirements to consider a solution as ethical, goals we’re following by adding web analytics, and how the ethical solutions allow us to meet them.

Comparing solutions

For an analytics suite to be considered ethical, it needs to be:

  • Open source. On a topic as sensitive as tracking users, we cannot afford to use closed-source scripts. I consider minified javascript is closed source in practice.
  • Self-hostable. This doesn’t mean we want to self-host the solution: it means we want to be able to do so, or to move to another provider should we want it.
  • Privacy focused. More particularly, we don’t want to have the technical possibility to track individual users, build user profiles, and track them across websites.
  • GDPR and CCPA compliant. Those are the standard regulations in terms of privacy.

This first screening on the list of usual analytics suites left me with the following solutions.

MatomoPlausibleUmamiGoogle AnalyticsCloudflare AnalyticsFathom ProFathom Lite
Open SourceYesYesYesNoNoNoYes
Self-hostableYesYesYesNoNoNoYes
Privacy-focusedYesYesYesNoYesYesYes
GDPR & CCPA compliantYesYesYesNoUnclearYesNo

Only Matomo, Plausible and Umami can be considered ethical enough for the Matrix Foundation to consider them. I then examined the following criteria:

Is the UI reasonably modern, and responsive? This is rather subjective and can sound trivial, but it’s a good indicator of whether product designers are involved or not, and whether the code base is in a state that allows the implementation of features product designers pushed for.

Is the product opinionated? Here again this is a subjective opinion. The Foundation is a small team, and we don’t want to become experts in advanced web analytics. I want a simple tool that allows us to assess whether our content reached its intended audience or not. I explicitly don’t want a tool that embeds everything but the kitchen sink.

What metrics can be tracked and more particularly:

  • Visitors. How many “persons” have visited the website? A same person with two devices will appear as two different persons, and that’s a trade off I’m willing to make to preserve users’ privacy.
  • Views. How many pages visitors view on average?
  • Average visit time. A short visit time shows that the website failed to deliver a message early on, and didn’t meet the user’s expectation.
  • Bounce rate. It’s the percentage of visitors who click on a page and leave early without navigating elsewhere. Bounce rates should usually be no higher than 40 to 60%. Higher bounce rates can either indicate that users stumbled upon the exact page they were looking for and found the exact information they needed immediately, or more realistically that the website was so confusing they didn’t feel like exploring was worth their time.
  • Device type. Are visitors primarily browsing from a computer or a phone?
  • Country. Knowing where our audience is coming from allows us to either tailor a message better suited for them, or on the contrary to measure the efficiency of our tentatives to diversify our audience.
  • Custom Events & Funnels. Following a specific visitor’s journey is very intrusive and I explicitly want to be unable to do so. I want to be able to measure if a layout or component has a positive impact in a specific context. I want to know if information is easy to find, and how. I want to track what works and what doesn’t so I can improve the website. I don’t want to know who in particular is using the website. Custom events are a basic way to track what visitors interact with, but journeys/funnels allow to track how people get to a certain point or where they drop before reaching it.

How lightweight is the script?. The usage of analytics must not slow down the matrix.org website. I want to be mindful of users’ mobile plans.

Is there a SaaS offering? Even if I want the solution to be self-hostable in case we need it, the Foundation doesn’t delivers the most value by spending resources hosting its own tools, particularly as a very small team. I’m also keen on supporting financially the stack we rely on, and SaaS offerings are regularly a good way to do so.

Here is the table of results of my findings

MatomoPlausibleUmami
Modern & Responsive UIYesYesYes
Opinionated UXNoYesYes
Tracks number of visitorsYesYesYes
Tracks number of viewsYesYesYes
Tracks average visit timeYesYesYes
Tracks bounce rateYesYesYes
Tracks device typeYesYesYes
Tracks countryYesYesYes
Tracks custom eventsYesYesYes
Tracks funnelsSaaS or paid pluginYesNo
SaaS offeringYesYesYes
Lightweight22.8 KB<1KB2.7KB

Picking a solution

While Matomo has been the historical replacement for Google Analytics, it is probably too much aimed at power users: its UX is not the most intuitive, its script is quite heavy, and the funnel feature I’m interested cannot be self-hosted “out of the box” without paying an additional plugin.

Umami is a very nice analytics solutions: it’s UI simple and beautiful, to the point that it might be too barebones. It offers nice metrics but doesn’t allow me to track funnels, leaving me with stats about the popular pages but not where people get stuck.

Plausible seems to be the better fit for what we’re after: a privacy preserving web analytics solution that still delivers valuable insights on where to improve.

I encourage people interested in the topics of privacy-preserving and self-hostable analytics to browse the awesome-selfhosted list, and probably consider server log parsing which is less invasive but more limited than client-side analytics.

Counting Ghosts was another interesting resource, and very inspirational to keep me on track. My decision was different from the author’s: I think web analytics can make sense, as long as we keep using as one tool in a toolset, and not as an absolute compass.

Implementation

The Matrix Foundation is going to launch its fundraising campaign around FOSDEM, and it will last until the beginning of April at the very least. It will be a good opportunity for us to measure the impact of our campaign and see how we can improve.

The deployment on the matrix.org website will start as soon as the privacy policy including Plausible is ready to be deployed. Plausible is not collecting any Personal Identifiable Information (PII), which means that the Foundation is not required to add Plausible to its privacy policy. We decided to do it nonetheless out of transparency towards our audience.