In 1985 Freddie famously proclaimed in an interview, with trademark mischievousness: “The bigger, the better - in everything”. Who can blame him; if you’ve played to 300,000 people in Buenos Aires, it’s hard not be seduced by scale.
There’s a strong perception today that AI is all about going bigger and bigger. Big datasets equal big innovation and big benefits. Which is true, to a point - but it’s not the whole story. Sorry, Freddie.
Scientific American has put the perception down to several high-profile AI breakthroughs being driven by massive data sets. Image classification systems are underpinned by a visual database of over 10 million images, and the most advanced natural language models are so, so good because they have sucked up hundreds of billions of words during training. This type of deep learning demands phenomenal processing power; so much that EleutherAI’s recent 20 billion parameter model was roadblocked by a global shortage of hardware powerful enough to handle the tsunami of training data.
Robots and autonomous vehicles are trained through reinforcement learning; an AI agent learns how to interact with its environment via trial and error. These systems start off clueless and gradually become taskmasters through trial and error. But all this failure means consumption of huge amounts of data, driven by massive computing power.
Go big… or go small
Last year the US-based Center for Security and Emerging Technology took aim at the conventional understanding that AI-driven innovation requires ever larger volumes of data. “AI is not synonymous with big data, and there are several alternative approaches that can be used in different small data settings,” they proposed.
And while three quarters of large companies have at least one data-hungry AI initiative underway, according to Harvard Business Review, some of the most valuable data sets can be quite small. “Think kilobytes or megabytes rather than exabytes,” urged an article in 2021. “Because this data lacks the volume and velocity of big data, it’s often overlooked.”
Of course, digital solutions are always determined by what you’re trying to achieve. But there’s a growing buzz around transfer learning because it can help you when you have little data on the immediate task at hand, but lots of data on a related problem.
Think of it like this; if you learn how to ride a push bike, you can learn how to drive other two-wheeled vehicles more easily. Transfer learning is an opportunity our data science superstars and transformation troubadours at R2 Factory have been tracking and are embracing to help us do hard things.
We always have our heads up, even when we have our heads down. Which means we’re constantly inspired by small data approaches around the world. From researchers in India training a model to locate kidneys in ultrasound images using only 45 training examples, to others using it to help flag cancers based on an initial set of just 100 photos of skin lesions.
“Google’s advanced chatbot Meena emitted the equivalent of 96 metric tons of carbon dioxide in training”
While transfer learning is opening eyes to new ways of tackling hard things in digital transformation, there’s another reason why it can trump super-sizing. In a global climate emergency, we must consider our carbon footprint relative to our ultimate goals.
Luis Ceze, a Computer Science professor at the University of Washington, urges us to focus on making our models use hardware more effectively and efficiently. “This should not be ignored, given the environmental cost of global-scale AI/ML systems,” he says.
The point the professor makes is borne out by research. One recent university study estimated the equivalent of 552 metric tons of carbon dioxide were produced in training OpenAI’s GPT-3 model. That’s the same as driving 120 cars for a year. It also found Google’s advanced chatbot Meena emitted the equivalent of 96 metric tons of carbon dioxide in training - roughly the same as powering 17 homes for a year.
Be anything you want to be
Deploying big data within big digital transformation can clearly have some big impacts. Efficiency and environmental trade-offs need to be considered and mitigated across all aspects of the life cycle. It’s about how you travel as much as the final destination.
But we’ll be with you on your journey, as will many others. When we welcome you into our safe space to do hard things at R2 Factory, you’ll join a circle of like-minded, ambitious and conscientious organisations. Big challenges can’t be solved alone, so we foster collaboration for sustainable and impactful solutions.
The final words must go to Freddie: “You can be anything you want to be, just turn yourself into anything you think that you could ever be.” And this time we whole-heartedly agree. So if you’re ready to get started, so are we.