Data Exploration & Transformation

The hardest data transformation problem in the universe

On March 7th, 2009 the countdown began. NASA was launching the Kepler Space Telescope with the goal of discovering Earth-like planets orbiting other stars, Kepler sought to answer a question as old as time itself: Are we alone in the universe?

On the surface, Kepler does not appear to be a data and analytics problem, but as you dive into the details you find that our AI Strategy Framework overlays quite well on top of the problems and solutions that the project had to grapple with. NASA and the associated teams (hundreds of scientists) were tasked with creating the hardware (space telescope) and software that would result in data being collected on hundreds of thousands of stars.

To tackle such a large problem and to facilitate cooperation between so many intelligent people, there was significant planning involved. This planning was much more involved than our suggested 6-page AI Strategic Narrative, though the purpose is the same. For data collection, the team equipped Kepler with a photometer and a 42 CCD array, capable of observing 115.6 square degrees of the sky, covering over 95 million pixels and with the ability to capture data from 10 million stars.. This is a massive amount of data, and as we’ve learned with our MVR (minimum viable robot approach), reducing the universe of available data can be a super power for AI project teams.

This is exactly what the Kepler team did, originally focussing on 180,000 stars and then reducing their scope down to 100,000. They also needed to reduce the size of the data being saved and transmitted. They did this by limiting each star to 32 pixels to focus on photometric data rather than full images. With the data collected, there was another problem to solve. With the telescope living 151,000,000 miles above earth, a continuous stream of data was not an option.

To help solve this problem, the first data transformation steps were applied. This involved on-board processing which accumulated data over 15 minutes, and then applying complex mathematics to drastically reduce the data size. This meant that the storage on board the telescope could hold data for a month before the telescope needed to pivot, point its antenna back towards earth, and send data.

With the data being sent back down to earth, the data teams could begin to dig in. They were performing what’s called Exploratory Data Analysis. Getting their hands dirty with the first readings from the stars to see what trends may be available and what issues may need to be resolved. Early on, it was clear to the teams that there were some odd values continually showing up in the raw data. Instead of being hidden messages from friendly aliens, it was instead something simpler. The mission was encountering unexpected noise from the telescope's electronics, necessitating adjustments in data processing to ensure accurate planet detection. Instead of sending a software update one-hundred and fifty-one million miles away, the team was able to adjust the data transformation pipeline and remove this noise.

From here, the team could further improve the dataset by applying algorithms within their data transformation to provide light curves which are essential for planet detection. With this cleaned up “Gold” Dataset the scientists - and later citizen scientists have been able to continue to drive astrological discoveries.

Here are just 3 of the big impacts that the Kepler dataset has had on our understanding of the universe:

• Kepler revealed that planets outnumber stars in our galaxy
• The mission discovered that 20 to 50 percent of stars could host small, possibly rocky planets within their habitable zones, hinting at the widespread potential for life.
• Kepler identified a plethora of planet types, including a common size not found in our solar system, situated between Earth and Neptune, showcasing the astonishing variety of worlds.

Through meticulous planning, innovative data strategies, and relentless pursuit of knowledge, the Kepler mission not only broadened our cosmic horizons but also underscored the critical role of data in unlocking secrets of our universe.

Exploratory Data Analysis

Using lessons from the Kepler project, let’s now learn what the data transformation layer of our hierarchy entails. After significant effort and some challenging decisions, we've secured our raw data in an accessible place, setting the stage for the crucial phase of Exploratory Data Analysis (EDA). This phase marks a pivotal moment in our journey, as the data teams are ready to play.

With the data in its raw, unstructured form, the team embarks on the manual yet critical process of EDA. This initial exploration is foundational, as it requires each team member to apply basic cleaning and structuring techniques to the data, enabling a preliminary analysis of what’s available. The primary objectives during this phase are:

1. Building a Comprehensive Map: The team's goal is to navigate the vast data landscape, identifying available information and initial trends. This mapping is instrumental in understanding the breadth and depth of the data at our disposal.
2. Identifying Early Issues: Early in the exploration, it becomes crucial to identify any potential problems with the data. This early detection is key to ensuring the integrity and reliability of subsequent analyses.
3. Making Statistics-Based Decisions: Addressing the primary issues encountered during EDA requires informed decisions. These include strategies for dealing with missing values, outliers, and datatype optimization, which are vital for preparing the data for more advanced stages of analysis and model training.

Patrick Riley of Google has written an extremely valuable guide to EDA that I always like to reference when beginning a new data analysis journey. Here are the key takeaways that Riley highlights.

1. Not all distributions are ‘normal’: Riley emphasizes the importance of looking beyond summary statistics to understand data distributions fully. Histograms, Cumulative Distribution Functions (CDFs), and Quantile-Quantile (Q-Q) plots are instrumental in revealing intricate data characteristics, such as significant outlier classes. These visual representations allow us to see the story our data is trying to tell, often uncovering insights that summary metrics alone cannot provide.
2. Don’t ignore your outliers: Outliers in your data shouldn't be hastily disregarded. As Riley points out, they can signal deeper issues within your analysis or data collection methodologies. A thoughtful examination of outliers can uncover flaws in your data or reveal unexpected truths, necessitating a nuanced approach to their management, whether by exclusion or categorization into an "Unusual" group. Understanding why data points are outliers is as crucial as identifying them.
3. Statistical significance does not equal significant impact: In the realm of large datasets, the distinction between statistical significance and practical significance is sometimes missed. Riley prompts us to consider the real-world impact of our findings, encouraging a focus on meaningful differences. This perspective ensures that our analyses drive actionable insights.

The Gold Dataset

The AI Strategy Framework promises to deliver value early and often. The Gold Dataset is that first piece of value that business stakeholders can see from your efforts. What is the Gold Dataset? “Gold’ is our term for describing a highly curated and clean dataset produced by our applied data transformations.

It is an essential precursor to model training, but also a very useful artifact for the rest of the organization. Having a set of data that is easy to understand, is easily queryable, and has been cleaned up is useful to business analysts and other decision makers in the organization. Even AI product managers can utilize this data to improve their decision making and prioritization.

What’s most important in the early days of your Gold Dataset is to make sure that there is an awareness and understanding of what it contains and provide suggestions for how the data could be used to help achieve business objectives. Often a high quality dataset becomes available and business silo’s result in low use. Lastly, it’s useful to get a low-code or no-code tool that can sit on top of the data so that non-technical users have the power to self-serve.

Storage vs. Compute: The Basics

At the core, cloud storage and cloud computing serve distinct purposes. Cloud storage focuses on saving data that can be accessed and retrieved from multiple devices over the internet. It's akin to a digital filing cabinet where you store various types of files for later use. Cloud computing, however, goes beyond storage, providing the processing power and services to run applications and perform complex computations over the internet. This distinction is crucial because it influences how businesses approach data handling and infrastructure planning​​​​.

Parallel computing

Over the years, advancements in technology have greatly influenced the way storage and compute resources are utilized. The advent of parallel computing allowed for tasks to be divided and processed simultaneously across multiple processors, significantly speeding up computations and making large-scale data processing feasible. This shift laid the groundwork for modern cloud computing platforms that offer scalable compute resources on-demand, enabling businesses to run complex data analyses and machine learning algorithms without the need for significant upfront investment in physical infrastructure​​.

Separation of storage and compute

The separation of storage and compute in cloud environments marked a pivotal change, allowing for greater flexibility and efficiency. In traditional setups, storage and compute resources were tightly coupled, making it challenging to scale one without affecting the other. Modern cloud services, however, allow these resources to be scaled independently. This means businesses can store vast amounts of data without necessarily incurring high compute costs until those resources are needed for processing. Virtual warehouses exemplify this by offering on-demand compute capacity that can process data stored separately, optimizing both cost and performance​​​​.

Implications for Business: ETL vs. ELT

Understanding the distinction between storage and compute, and their decoupling in cloud environments, is instrumental in deciding between ETL and ELT processes. ETL, which emphasizes pre-processing data before loading it into a data warehouse, can be resource-intensive, requiring significant compute power upfront. On the other hand, ELT leverages the cloud's flexible compute resources by loading data first and transforming it as needed. This approach can be more cost-effective and scalable, particularly for businesses dealing with large volumes of data that may not require immediate processing​​.

For businesses, the choice between ETL and ELT should consider not just the current needs but also the strategic direction.

In the early days of your project, and while trying to deliver value early and often, ETL might be the way to go. Especially if you’ve limited yourself to a smaller dataset and have well understood transformation rules to apply. This will get you to the ‘Gold Dataset’ quickly. Further down the road, as more data becomes available, your team may want to implement a ‘light’ cleaning of the data and place it in a ‘Bronze’ database - one step beyond your fully raw database.

Keeping it simple

Navigating data transformation options can feel overwhelming, particularly in an environment saturated with technologies all promising to revolutionize AI development. As your team moves from exploratory data analysis (EDA) towards creating a curated 'Gold Dataset', it's crucial to approach the selection of tools and processes with a few clear heuristics in mind:

1. Start with Scalability in Mind: Opt for data transformation solutions that offer scalability. This ensures that as your data grows and your AI vision expands, your chosen infrastructure can grow with you, avoiding potential bottlenecks.
2. Ensure Compatibility: Ensure that the tools you select seamlessly integrate with your planned analytics and ML frameworks. Compatibility checks can save you from future headaches by ensuring that your data transformation pipelines smoothly feed into your analytical models and ML training processes.
3. Prioritize Data Quality Over Quantity: When building your 'Gold Dataset', focus on the quality of data rather than the quantity. A well-curated, high-quality dataset will provide more value and lead to more accurate insights and predictions than a larger, less reliable one.

Wrapping Up

Data transformation is difficult - even if you’re a rocket scientist. While at this layer of the AI Strategy Framework, and your data teams begin to get their hands dirty, it can be easy to get distracted. Distracted by new data relationships, new opportunities, new insights. Don’t forget where you started and your goal. Limit complexity and maintain focus. Keep your focus on high quality data and getting the first dataset clean and ready for the next layers. This will set you apart and increase your chances of success.

Next week we’ll talk about deriving value from our Gold Dataset through data visualization and analytics. We’ll see that while our focus remains on delivering AI, sometimes good old fashioned analytics can provide massive returns. See you there!