Glean uses a single-tenant model (providing every customer a dedicated GCP project) to deploy our services and provide the best possible security experience. One big challenge we face, however, is monitoring and operating multiple GCP projects as our customer base grows. Multiple alerts for separate GCP projects can fire simultaneously, making it a challenge to manage.
Google’s Error Reporting faces the same issue – where the same application exception may happen in multiple GCP projects at the same time. In this post, we’ll discuss how we use our Notification Dispatcher tool to scale our monitoring ability in Cloud.
In order to manage multiple GCP projects effectively, we need to maintain a global view of alert notifications and application exceptions across all our GCP projects. This helps us to understand how widely a production issue has spread globally, and allows us to triage production issues in an efficient way.
Cloud Monitoring Alerting supports many different types of notification channels, but it lacks Jira support. Error Reporting didn’t support notifications when the Notification Dispatcher tool was developed. Since Glean is a search-oriented company, we want to connect these important GCP production-related services to other SaaS applications like Jira – and more importantly, make the information searchable and turn resolutions into shareable knowledge.
On top of that, we develop many home-grown features that enable us to manage the same issues for multiple projects together.
Notification Dispatcher is the internal tool at Glean for managing GCP alert notifications and Error Reporting errors. It is integrated with Cloud Monitoring and Error Reporting as the notification source, and is integrated with Slack, Jira, and OpsGenie as notification destinations.
This tool has a few components:
We’ve built some other cool features that enable us to scale operations in the Cloud:
The Notification Dispatcher tool has improved our productivity dramatically since it was launched. It provides a mechanism for our engineers to consolidate production signals from all of our GCP projects and focus on important production issues. Once those notifications are forwarded to Jira or Slack, engineers can Glean the information and turn the resolutions or discussions into knowledge.
Fill out the details below to get the full report delivered to your inbox.
Glean is an incredible opportunity to revolutionize the way everyone works. Check out why we chose to join Glean and tackle the exciting mission of growing its presence in the European market.
Glean is officially partnering with Google Cloud to deliver and scale the leading enterprise search solution for the new era of digital work.