All sessions of Transformation 2021 are now available on demand. Now look.
Enterprises increasingly rely on unorganized data for rules, analytical and decision-making purposes. Unstructured data will power analytics, machine learning and business intelligence.
According to the latest figures from research firm ITC, the volume of unstructured data is projected to increase from 33 zetabytes in 2018 to 175 zetabytes, or 175 billion terabytes, by 2025. There has to be some kind of data management so that organizations have the right. Data is available at the right time. Krishna Subramaniam, President and COO of Data Management Software Provider Enterprise, sat down with VentureBit to discuss the business benefits and challenges associated with unstructured data.
Venture: Does an enterprise IT organization know how much unstructured data they have and how fast it is growing?
Krishna Subramaniam: Apparently they know that a lot is unorganized and it is growing in double digits, but they do not know exactly how much they have and how fast it is growing. We know that 0-90% of the world’s data is unstructured.
VentureBeat: What’s wrong with the growth of this data – now, after all, there’s endless cloud storage, right?
Subramaniam: The big issue is the cost – two-thirds of the cost of data is not in storage, but in its active management. For each piece of data, companies usually keep a few backup copies and a copy of the copy for disaster recovery. If you think your data is growing by 30%, that factor is 90-100% higher when you factor in all the copies of the data. It is also wise to consider that cloud storage is not necessarily cheap. For instance, AWS itself today offers more than 16 layered file and object budget storage. If you don’t put your data in the right place and e.g. If you control the cost, you can pay more than you used to store on campus because you will be charged every time you read the data. The key here is that more than 80% of the data is not really active ces access and is cool. This cold data can be stored on cheap storage and does not require the same level of backup and replication. Therefore, you need to actively manage hot data and cold data that are rarely used separately. Just one example, Pfizer researchers generate between 8TB and 10TB a day, and they ran from a datacenter distance. They were able to use the data management product to identify cold data and remove it by moving their expensive storage, backups and duplicates to low-cost-resilient storage in the cloud and taking it out of active management. The company cut 75% of their data storage and backup costs, without noticing any changes to all users. The hard thing about data growth is that many organizations do not like to delete data. You never know when you might need it. And when you do, you want to be able to find it easily. And users and apps don’t need to change their behavior when you move data around. Previously, by archiving in tape, it was not possible, but now it is with cloud storage and with data management software.
Venture: Why it’s important to be strategic about how you manage it, store it – it’s just a B.I. Not about making sure you can find for the team?
Subramaniam: Today, data is a valuable corporate asset. You have to be strategic with it because it’s not just for your BI teams, but for R&D and customer success teams. They need historical data to create new products or improve what they already have. This is very relevant in this product, such as in the semiconductor chip industry, but also in other industries such as pharmaceuticals that are so important to our economy. Covid researchers rely on the access of SARS data when developing vaccines and treatments. Data often becomes valuable again later, and what if you don’t know what you have or you can’t find it? We have clients in the media and entertainment business and in the past when they wanted to find an old show, they needed a tape archive. Then, they needed a property tag to locate the tape. It can be very difficult, and that is why archiving is not popular. Live archive solutions available today make archived data instantly accessible and transparent layered so that users can easily find files and access them at any time.
Venture: How will the tools and practice be developed to better benefit IT departments to better benefit this unstructured data for organization / business users? What is needed, where are the gaps?
Subramaniam: You need a storage-independent way to view data across all your storage technologies, not just to move data to the right place, but to help businesses extract value from data. Gartner calls this category “data management software”, and includes companies such as Cirrus Data for block data and Compress for file and object budget data. The ultimate goal is to help business users take advantage of historical data and this requires data search, data analytics and data intelligence. These are the hot areas where a lot of innovation is happening. Cloud providers offer many data warehousing and data analytics solutions that can be leveraged together with data management software, such as AWS Redshift and QuickSite. For instance, we use distributed elastic search in our software to quickly search for billions of files and find all user-related data, such as all data for a specific project, and export this data to Redshaft for further analysis. If you can’t find significant trends like inconsistencies or ransomware, why all this data? I believe we need more predictive analyzes around the data.
Venture: Will the Data Management Challenge stimulate a whole new field of startups in the next two years?
Subramaniam: Certainly. Analysts are beginning to identify data management software as a new category. In addition to the above use cases, consider funding all new types of data analytics companies such as Snowflake, Databrix, and Apache Spark. Many companies are now coming to light to address the issues of data management and data analytics.
VentureBeat: How are large cloud providers responding to problems and opportunities with unstructured data growth?
Subramaniam: They are all providing more services for storing data at different performance and price points. The Amazon Elastic File System (Amazon EFS) and Azure Files were born to address the need for file storage in the cloud. Chief CSPs are investing in partners in many areas of unstructured data management, including migration and analytics.
VentureBet’s mission is to become a digital town square for technical decision makers to gain knowledge about transformative technology and transactions. Our site provides essential information on data technology and strategies to guide you as you lead your organizations. We invite you to become a member of our community for:
- Up-to-date information on topics of interest to you
- Our newsletters
- Gated idea-leader content and discounted access to our precious events, e.g. Transformation 2021: Learn more
- Networking features and more
Become a member