Google BigQuery Omni release helps in a deeper understanding of all your organization's data
Google BigQuery Omni release helps in a deeper understanding of all your organization’s data
We were in a quarterly business review meeting with the Director of Finance of one of our major customers. During the meeting, the Director informed us that the usability and performance of the Google App Engine applications that we had delivered for them were quite fast and responsive, while the performance of the dashboards of the ERP (a world-famous one at) that they were using, was performing very slow and took way to much time to show the required results. He wanted to show us the problem that he was facing, so he fired up the ERP screen put some filters, and clicked ‘go’ to show the dashboard!! The spinner came and the screen was loading…loading…….loading! So we moved to other topics of discussion while the dashboard could get loaded! After 10 mins or so, we completed the meeting and we were about to leave…took a quick glance at the screen and it was still…..loading! 🙂 Case, proven.
Our application did not have the same load as the ERP, but still had several dozen millions of rows and 100+ terabytes of data to process to show the various dashboards and analytics. It was quite a data-intensive application. With several 100s of users analyzing their data each day of this business crucial application, how could we have made the application responsive while still scaling dynamically – it was due to the choice of the Google BigQuery database that we had used as the underlying data source for analytics.
Over the last 3-4 years, we have been in love with the choice of BigQuery in some of our heavy data processing applications. Being fully managed, highly scalable and always giving results in seconds (most often between 4 to 8 seconds) on massive data sets with complex queries, it has allowed us to focus more time on the application features, rather than all the technical aspects of fine-tuning the database and dealing with the infrastructure challenges. Traditional RDBMS databases would not have fit the need. It was a key to our customer satisfaction.
What does the Google BigQuery Omni release offer?
The recent announcement of the private alpha launch of Google Omni is exciting since it solves some customer real-life problems for us. For instance, data from a Warranty Management application would be moving to an Incident Management application and visa-versa with strong interconnections between the data in the 2 different systems. Similarly, there are strong connections between data in Appraisal applications and data in the organization’s Competency Management application and Bonus calculation applications. Often, due to history, one of the applications would be an application that would have a built-in Google App Engine with Big Query as the analytical data source, but the other applications might still be running on other technologies and other clouds. This always limited the analytical capabilities that we could offer the customers to only data coming from our own applications and often not combine it with the data from applications running on other clouds (though it’s possible to do a periodic database synchronization, which is quite expensive and not real-time).
Now with the Google BigQuery Omni release, we can combine data from applications running in different clouds and provide real-time data analytics to users – and this can immensely help users to understand their data much deeper, faster, and in real time (and not to miss the point that the IS team will love the cost optimization of this approach). This helps our applications that use BigQuery to transform from being just a data source to a true and optimized data lake.
Positive changes to Anthos
BigQuery Omni runs on Google Anthos. Anthos is a game-changer in the inevitable multi-cloud strategy adopted by many organizations. During the launch, Anthos seemingly being priced at a minimum starting cost of $10000 per month for 100 vCPUs, was quite expensive even for multi-billion revenue organizations, unless the organization has 1000s of CPUs to manage in a multi-cloud setting. For scenarios where data from a few applications spread over multiple clouds need to be accessed in a data lake setting, the Anthos pricing was seen as a blocking point for cost-effectiveness.
With recent changes to Anthos pricing (effective September 1, 2020) and aligning it with the same very attractive pay-as-you-use policy for many of its products such as the Google App Engine, Firestore, and BigQuery, Google has made Anthos usage very compelling.
With BigQuery Omni using Anthos, it’s a great move from Google to have made changes to the Anthos pricing policy to be a lot more attractive along with its BigQuery Omni release. With these changes, the BigQuery beta release might be one of the most exciting launches to look forward to in the coming months.