Big Data needs no introduction. Its changing the way businesses connect with customers, and the Utilities industry is no exception. With the introduction of intelligent grids and smart meters, energy companies can now gather, aggregate, and analyze terabytes of customer usage data, and get accurate insights on consumer usage patterns. This intelligent analytics enables energy companies to drive attractive pricing, intelligent load sharing, distribution efficiency, and even loyalty programs. In this post, I will build on these typical use cases, list a few Big Data challenges for utilities companies, and also recommend a solution.
Can Big Data be Intelligent Energys Driving Force? Lets begin with the source of energy dissemination the power grid that routes energy to consumer receiving stations. With the Internet of Things (IoT) – connected devices, sensors and data, power grids today, operate a lot more intelligently taking data from multiple sources, analyzing energy usage and routing patterns, and feeding results into enterprise systems, for further processing, which in turn, result in intelligent insights, operational efficiency, and customer delight. Simply put, todays connected systems dont just transmit energy to consumers. Devices, sensors and cables also carry meta-data about that energy. This meta-data is the real driving force of Intelligent Energy.
At the receiving end, the intelligence is manifested through smart meters, which in addition to receiving and routing power to consumer end-points, can also read, interpret, analyze and report data. For energy companies, this meter smartness could translate to improved consumer usage tracking, proactive energy savings initiatives, and even top line improvement, through demand and location-based surge pricing. It even opens up the possibility that in the future, energy companies could adopt best practices from the travel industrys pricing, rewards and loyalty programs. Big Data from smart meters is also an innovative way to pre-empt and investigate power theft a major pain area for energy companies in developing countries.
And then, theres social media and the rise of the connected customer. By using social media not just as an information dissemination tool, but as a customer engagement platform, energy companies can push boundaries of customer engagement to newer extremes, and take customer experience to altogether new levels Retargeting, Audience Profiling, Gamification, and Digital Marketing are just a few enablers for enhancing customer experience.
Recognizing Big Datas potential to deliver measurable and positive business results, utilities businesses are investing in Big Data solutions. However, shrinking IT budgets dont allow for clean-slate Big Data implementations. Also, businesses have made recent large investments in Data Warehousing (DW) and Business Intelligence (BI) infrastructure. This, combined with the sizeable legacy IT landscape that cannot be discarded overnight, is pushing utilities companies towards adding a Big Data solution layer over existing BI systems. However, Big Datas sheer volume and velocity poses challenges, rendering traditional, on-premise BI and DW infrastructure inadequate and ineffective for Big Data.
The first challenge for CIOs lies in ensuring the seamless integration of existing BI frameworks with Big Data components. Big Data is not just about processing structured data sets from grids, meters and applications. To unlock Big Datas real potential and value, unstructured data from maintenance records, notes and even Twitter feeds, is equally important. This unstructured data does not lend itself to traditional data analytics, and requires special data listening tools and processing algorithms. Varied data formats create differences in data and search algorithm architectures between the DW and Big Data. This operational disparity between theoretically independent BI and Big Data systems compromises overall system quality.
Second, with unconventional and unstructured input data from varied sources, Big Data systems often lack precise rules for filtering or scrubbing bad (or noise) data. Current Big Data processing frameworks are not agnostic enough to accurately accommodate varied and unstructured data points for analysis.
Finally, Big Data environments cannot accurately predict the large volume data sets, and lack compatible adapters for real-time data processing in large clusters or nodes. Most Big Data frameworks process data in smaller clusters, in batches, and not real-time, compromising the BI-friendliness of the high density data volume.
When poorly integrated with BI and DW frameworks, Big Data systems have weak data scrubbing rules, and cannot process data in real-time. As such, these systems are not failsafe, and pose cost prohibitive business consequences.
These challenges have also been summarized and articulated in an earlier blog post in four terms – Volume, Variety, Velocity, and Veracity -popularly, Big Datas 4V. When designing Big Data processing systems and infrastructure, its important to consider these four dimensions, and provision adequate infrastructure.
Despite its processing and infrastructure challenges, Big Data presents new transformation opportunities for energy companies. Its biggest potential lies in improving operational efficiency of utilities businesses while enhancing the understanding of the consumers consumption needs. However, given the numerous questions and plethora of technology options, utilities CIOs find it difficult to navigate the endless Big Data ocean. Thenext post will recommend an Assurance framework for addressing the top challenges Big Data poses to this industry sector.