Ensuring Reliable and Accessible Storage

The birth of a cloud

In the quiet confines of Jeff Bezos's home in the early 2000s, the executive team was thinking strategy. While reviewing their strengths, weaknesses, and opportunities, a transformational idea emerged. Amazon had a competitive advantage with their expertise in managing digital infrastructure. As a result of building one of the largest operational websites in the world, they had been forced to become skilled in the trade of digital architecture.

That day, the team recognized a universal challenge: countless businesses, especially those at the threshold of AI innovation, grappled with the daunting task of building and managing technology infrastructure. The skills that had helped Amazon build their e-commerce giant, were skills that were highly valued by every other company looking to grow their digital presence. To capitalize on this, Bezos and his team decided to offer Amazon’s robust digital infrastructure as service. Famous investor Bill Gurley has said this is one of the top 3 business moves of all time. This new offering allowed businesses to outsource the building of digital roads, so they could build the cars that drive upon those roads.

AWS was envisioned as the answer to a question many businesses hadn't fully articulated yet: How could they pivot from investing in the mechanics of technology to harnessing its strategic potential? With AWS, companies could tap into a scalable, secure, and sophisticated infrastructure, transforming their approach from one of operational burden to strategic advantage. The AWS retreat at the Bezos house underscored a guiding principle for businesses embarking on their AI journeys: the real value lies not in the infrastructure itself but in the innovation it enables.

For us, this story serves as a reminder as we navigate the complexities of executing an AI strategy, that our focus should remain on the problem we are solving for our customers. AWS and the other cloud service providers can handle the infrastructure, so we can innovate, create value, and transform our vision into reality.

Planning for flexibility - see the unseeable.

Let’s now take a step back and think about the big picture. Not all data, in all industries, is best suited for cloud storage. There are many heavily regulated industries and instances where sensitive data needs extra protection. In these cases an internally managed solution or hybrid solution may be required. If at all possible, try to focus efforts on collecting and storing data that is a good fit for the cloud. This will lower complexity, and speed up time to value. If you do have sensitive data, engaging a information security expert will allow you to see what secure tools a cloud provider can provide or build out the more complex managed strategy.

In addition to considering options for where the data is stored (cloud, on premises, or hybrid), we need to also decide the best storage format. Given that we aren’t fortune tellers and can often overlook important data points that may be relevant in the future, I usually recommend the first data storage location that allows for raw, unstructured formats of many different kinds. A place where you can store text and images as well as structured data that fit nicely into columns and rows.

This way you have a few advantages:

1. Cost-Effectiveness: Cloud databases like AWS S3 offer storage at $24/TB/month, making it economically feasible to warehouse vast quantities of raw data.
2. Flexibility: Premature commitment to a structure may pigeonhole your data strategy. Flexibility helps avoid the pitfalls of scattered, inaccessible databases, ensuring data scientists and analysts have access to the information.

What this means in practice is that you’ll probably want a Data Lake. This is the industry term to describe a place that allows data of all types to be stored in raw form. It also means that you may have to worry about additional layers of data storage and data transformation down the road, but with the benefit of added flexibility, this is well worth it.

Before making the suggestion that a Data Lake is for everyone, we should consider the cases in which it is not. If you have a fairly straightforward, structured dataset - you may be able to skip the Data Lake. Going right to a data warehouse with a well-defined data model may help you deliver the value you’ve laid out in your AI vision and narrative. Though this case is rare, it may be your situation, so don’t place an extra step where one isn’t needed.

Structured vs. Unstructured

What do we mean when we say structured or unstructured data? Most of us have used Microsoft Excel in some capacity. Data that fits nicely into the rows and columns is considered structured. Consider filling out a form to sign up for a website. They ask for your name, email, phone number. This data can be stored in a structured format with each row representing a user who has signed up.

Conversely, unstructured data—words, images, sensory readings—defies easy categorization, requiring specialised databases for efficient storage and retrieval. The diversity of data in AI applications often requires accommodating both structured precision and unstructured richness.

Speed and Efficiency

The next thing we need to consider is how our data scientists and analysts are going to use the data within the database. Are they going to need to extract huge chunks of data to perform model training? Do they need to quickly find single pieces of data? For AI, data scientists are definitely going to need the ability to extract large amounts of data in batches to perform their model training - which we’ll get into later in this series - but there also may be different uses for the data. Do business users want access to explore the data, or do data scientists want to sort out different relationships and extract examples from the dataset? There are two things to think about for database performance in this case.

Indexing

Imagine a library where the books are placed in random order; indexing saves us from this chaos. By creating pointers to data locations, indexes ensure rapid retrieval, directly impacting query speed and user satisfaction. Whether organising by date/time in time-series databases or utilizing unique identifiers, choosing the right indexing strategy is paramount for operational efficiency.

Partitioning

Dividing a database into manageable segments, partitioning enhances performance and simplifies data management. This strategy not only accelerates queries but also facilitates parallel processing and distributed storage, optimizing resource utilisation.

Comprehensive Risk Assessment for Data Storage

Security Measures and Risks:

Security is paramount when considering data storage options for AI projects. The risks of data breaches, unauthorized access, and data corruption must be addressed. Implementing advanced encryption for data at rest and in transit, rigorous access controls, and regular security audits can mitigate these risks. However, the choice between cloud, on-premises, and hybrid solutions will significantly influence the security strategies available. For instance, on-premises solutions may offer more direct control over security protocols but require significant investment in security infrastructure and expertise. Cloud solutions, while flexible, necessitate a thorough evaluation of the cloud provider's security measures and compliance.

Regulatory Compliance and Data Sovereignty:

Compliance with regulations such as GDPR, HIPAA, or industry-specific standards is a critical consideration. The data storage solution must not only comply with current regulations but also be adaptable to future regulatory changes. This involves assessing the geographical location of data storage in cloud solutions (data sovereignty) and ensuring that data management practices adhere to all relevant laws and standards. On-premises solutions may offer more straightforward compliance paths for certain regulations but can be resource-intensive to maintain.

Keep it simple

Above we’ve considered some of the essentials when making your decision for data storage and honestly it’s a bit overwhelming. Especially if your team lacks the guidance of a tech leader with years of experience (and battle scars) it can seem daunting to make a decision. It also doesn’t help that there are billions of dollars being poured into the market for AI development which seems to increase the universe of options by the day.

When trying to get quickly to value and validate your AI vision, it is helpful to use a few heuristics to pick a data storage option and move on:

1. Cloud-First: Consider cloud storage solutions for their scalability and ease of integration with analytics and ML services.
2. Open Source Preference: Lean towards open-source databases and frameworks, which come with extensive community support and flexibility.
3. Compatibility Check: Prioritize tools and platforms that offer native support or established connectors for your analytics and ML frameworks of choice.

Wrapping Up

Not all of us have the opportunity to hang at the Bezos residence and come up with a brand new seemingly unrelated business idea that turns into a money printing machine. What we do get as a result of this meeting is the ability to build, launch, and scale our AI products quickly and cheaply. Thanks to AWS, Microsoft, and Google we can focus on our customers and their problems. We can get to know them better than any of our competitors and use these tools to win.

When trying to get to value early, data storage can be one of the most difficult hurdles. There are so many unknowns and making a decision can be daunting. By understanding regulatory risks, your long-term plan, and the short term target, you can avoid analysis paralysis. I hope that this blog has helped you feel equipped to make a decision.

Next up we have Data Transformation. Taking our raw, ugly data and making it pretty. Getting closer to our first taste of value. See you next week!