The Challenges of Big Data
Big data, the massive amounts of data that today’s organizations collect, store and analyze, can be valuable in many places and many ways. In our benchmark research, six in 10 organizations (59%) using data lakes as a storage method for their data report that big data helps them gain a competitive advantage, and nearly one-half (45%) say that their use of a data lake helps them lower costs. Organizations also report it helps them improve the customer experience and respond faster to both opportunities and threats.
There was a time when embracing big data was expensive. But the economics have changed over time, making big data technology more attractive an investment. The cost of storage has declined significantly with the use of distributed systems based on clusters of inexpensive servers the collective storage of which can be accessed as a single system. In addition, the advent of low-cost cloud-based object stores has also reduced the cost of storing large amounts of data. The cost of computing power has also declined thanks to the availability of increasingly powerful CPUs and with better utilization of those CPUs via virtualization that allows the sharing of workloads. With both more data and more processing power available, organizations can now perform advanced analytics on large amounts of data, including more sophisticated and valuable analyses relying on artificial intelligence and machine learning (AI/ML). The changing economics mean that what was once a rare capability has now become a competitive necessity.