E32: [Bonus Episode - The AI Breakdown] Can OpenAI's New GPT Training Model Solve Math and AI Alignment At the Same Time?

Full Transcript

Transcript

Nathan Labenz: Hi, everyone. I wanted to share another podcast that I've been enjoying recently, The AI Breakdown. As anyone in AI knows, the pace of progress and all the new releases are relentless. I call myself an AI scout, and I work overtime to keep up. But these days, even I can't keep track of everything. The AI Breakdown helps me make sure that I don't miss anything important by curating news and analysis daily. Host Nathaniel Whittemore, aka NLW, quickly highlights the top stories of the day before going deeper on a single topic of interest. Episodes are usually 15 to 20 minutes, and he releases them every single day. Now, it's not easy to keep up with a daily release schedule and still maintain your sanity, so I really appreciate how NLW maintains a curious posture and avoids rushing to judgment. A big part of the reason I'm inclined to recommend the show is his willingness to sometimes say, I don't know. I think listeners will find The AI Breakdown to be a great complement to the long form, deep dive interviews that we create, so I encourage you to check it out.

Nathaniel Whittemore: On today's AI Breakdown, OpenAI's research team has released a new approach to training that seems to solve not only math, but offer some promise for AI alignment. Before that, on the brief, policy from Japan and Australia, and an AI camera without a lens. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Like, subscribe, and share, and learn more at breakdown.network. Welcome back to The AI Breakdown Brief, all the AI headline news you need in 5 minutes or less. Remember, as always, you can get this as a newsletter at aibreakdown.beehive.com. We kick off today with news out of Japan in what might be an important precedent. Earlier this year, Getty Images sued Stability AI. Basically, they alleged that the creator of Stable Diffusion had trained their model on some 12 million images that were owned by Getty proprietarily. You might remember when this came out, and there was this little Getty watermark that appeared to be in some of the Stable Diffusion outputs. Well, new thinking in Japan would basically nullify Getty's claim in that case. In a recent public meeting, the Japanese minister of education, culture, sports, science, and technology said that the country would not enforce copyright law when it came to training AI models. Now reporting around this is somewhat sparse, but it seems as though Japan is concerned that worries about copyright have been holding it back relative to international competitors when it comes to the development of AI, and they see this non-enforcement of copyright laws when it comes to training models as a way to get out ahead. Now could this create a global precedent? It's hard to say, but it's certainly an interesting evolution when it comes to a key area of law as relates to AI. Speaking of APAC countries and AI rules, Australia has just introduced 2 new research papers around AI as well as an 8 week consultation period designed to get feedback from citizens. Australia is no stranger to AI policy, first publishing voluntary ethics principles in 2018, but now it seems like lawmakers are wondering if tougher laws are needed. Industry and science minister Ed Husic said there is clearly in the community a concern about whether or not the technology is getting ahead of itself. Governments have got a clear role to play in recognizing the risk and putting curbs in place. As part of the consultation, Australia is saying that its interventions could range all the way from voluntary ethical principles and technical standards to bans, prohibitions, and moratoriums.

Next up, some really interesting research called GPT4Tools. We've discussed on this show how multimodal is the future of LLMs. In other words, AI models that are able to move between different modalities from images to text and back and forth and even including audio, video, etc. GPT4Tools is an approach to teaching large language models of the open source variety, things like Llama, how to use tools to develop multimodal capacities in a way that doesn't require huge amounts of computing power or data input. I use the new ChatGPT share feature as well as the XPapers plugin to provide a simplified analysis of the paper, including a bullet point overview of what it said, how the training was conducted, what the low rank adaptation technique is, and what the real world use cases of this might be, including customer service bots, content creation, education and training, accessibility tools, data analysis, research and development, and more. The TLDR for the sake of this brief is that multimodal continues to be the cutting edge when it comes to LLM research, and more than ever, people are thinking about not just how big companies that have unlimited resources and huge amounts of data can train multimodal models, but how different techniques can be used to train open source versions of those models as well.

Last of the day, people are absolutely loving this new AI camera that has no lens. Paragraphica is a different type of camera that is designed for an AI world. So instead of using a lens to capture an image of what's right in front of you, it instead uses location data as well as weather data, time data to create a prompt that then generates an AI image. So for example, a midday photo taken at Clifford Straat, Amsterdam, the weather is partly cloudy and 18 degrees. The date is Wednesday, 05/24/2023. Nearby, there is parking and a yoga studio. On the left, where he stood with the camera, and on the right, the image that was produced. Now on the viral Twitter thread where Bjorn Karmann introduced the Paragraphica, there was a little exchange that I thought was extremely reflective of the current state of the AI discourse. Tarek Khan responds to Bjorn and says, what is the purpose behind this? Do you think that this is helping anyone? Does this make our society better, or does it make it more inauthentic and synthetic? You ought to really consider this path in your motivations and more importantly, the effect it could have. Ratioing that comment was a response from PHABC, who posted a version of the midwit meme where the middle said, no, you can't be creative, have fun and build things unless they move humanity forward. And in the meanwhile, the 0.1% on the bottom say this is fun, and the 0.1% at the top say this is fun. Now while I disagree with Tarek here in the specifics, I do think it's important to have these discussions, so I'm not discouraging at all. You just gotta love a good meme. Anyways, guys, that is it for today's AI Breakdown Brief. If you are enjoying, please like, subscribe, and share, and I will be back soon for the main AI Breakdown.

Welcome back to The AI Breakdown. Today, we are talking about some exciting research out of OpenAI with progress on both math as well as AI alignment. But where we're going to start is just looking at the company's set of updates. There's been an absolute flurry over the last couple of weeks. From a product standpoint, OpenAI has been extremely busy. Probably the biggest feature is, of course, the fact that ChatGPT launched its iOS app. Surprising exactly no one, it has been number 1 basically since it launched. And while it was rolled out first just in the US, it quickly came to a dozen and then now 152 different countries. For my money, the most impressive thing about the ChatGPT iOS app is the transcription voice to text. I agree wholeheartedly with Jackson Dahl who writes, The gap between new AI transcription tech, e.g., Whisper on ChatGPT mobile and Siri, purely transcription quality not responses, is actually unbelievable. I've been trained to assume that transcription just doesn't work at all, and in fact, Siri is just an utterly bad product. I'm so excited to embrace voice technology again after years in Apple's middle ages. This has been completely my experience as well, although I will say that Logan, who runs developer relations at OpenAI, actually responded to Jackson and said, expect this will change on June 5. Lots of very competent people who have likely been working on this exact problem. Apple is a sleeping giant. Logan is, of course, referring to Apple's WWDC conference, which happens next Monday.

Speaking of Logan, here he announces another of ChatGPT's recent features, shared links. For the first time since ChatGPT launched last November, users can now share out links to the conversations they've had. Without even trying to be really intentional about it, I found myself using this feature pretty frequently. And then finally, credit where credit is due. I literally just did an episode called Are ChatGPT Plugins Overhyped? And one of the things that I pointed to as an example of just how nascent they are is that there wasn't even a search interface for them. Well, someone at OpenAI must have heard me because less than 24 hours after I published that video, here we now have a search field for plugins. So for example, if I want to find all the plugins that allow me to interact with PDFs, I can search PDF. Same for YouTube. Now while this is of course a huge improvement over the previous entire lack of search, there are still kinks to be worked out. So far as I can tell, it's literally just pulling from text either in the names or the description, which means that some categories aren't being fully represented in search. By my accounting, there are coming up on a dozen plus plugins that relate to finance in some way, but only those that actually use the word finance in their name or description show up when you search for finance. When you search money, none at all come up. But as I said, still a huge improvement and I think representative of exactly what I was saying, which is that a lot of these features are going to be rolled out over time.

Now somehow despite all of that, OpenAI has been even more busy when it comes to the regulatory and policy side of things. A couple of weeks ago, CEO Sam Altman was the star witness on the Senate's first hearing on AI regulation in the post ChatGPT world. During that hearing, Sam said that OpenAI would support a new dedicated agency for AI regulation and some sort of licensing regime that would have control over whether companies could release super powerful models. Now there was enough of a dust up and a hullabaloo about whether this amounted to an attempt at regulatory capture that OpenAI actually released a blog post talking a little bit more in depth about what they were interested in on this front. The short piece was called Governance of Superintelligence. First, they said we need some degree of coordination among the leading development efforts. Second, OpenAI argued we're likely to eventually need something like the nuclear regulatory body for superintelligence efforts, which would come with the ability for an international authority to inspect systems, require audits, test for compliance, and even place restrictions on degrees of deployment and levels of security. And third, we need more research about AI alignment and the technical capability to make a superintelligence safe. Now what we don't need, they said, is onerous regulation on open source models and models that are below a certain capability threshold. This was their way of reinforcing the point that Sam was trying to make at that hearing that he was talking about extremely advanced models. Think GPT-5 plus, not open source tinkering with Llama.

Recognizing that even outside of existential questions, there are big, thorny issues that relate to AI in the public sphere, about a week ago, OpenAI also announced ten $100,000 grants to fund experiments in and around what they call democratic inputs to AI. So these are grants for people who have answers to questions like under what conditions should AI systems condemn or criticize public figures given different opinions across groups regarding those figures? What should the default persona for an AI system actually be? How should it be represented? The point that they say is that no single individual company or even country should dictate these decisions, and so they want to fund people with interesting ideas around them. Now, this was followed up today just about a week later after that democratic inputs grant program with a new cybersecurity grant program. This is once again a $1 million initiative with the focus, they say, to, quote, boost and quantify AI powered cybersecurity capabilities and to foster high level AI and cybersecurity discourse. Our goal is to work with defenders across the globe to change the power dynamic of cybersecurity through the application of AI and the coordination of like minded individuals working for our collective safety. Some of the project ideas that their team has put forward as the type of thing they'd like to fund include identifying security issues in source code, detecting and mitigating social engineering tactics, developing or improving confidential compute on GPUs, creating honeypots and deception technology to misdirect or trap attackers, and many, many more. Now, when it comes to government interest in AI safety and AI risk, a lot of the focus is not so much on the paperclip problem, but instead on exactly this type of cybersecurity issue. So it's not surprising to see OpenAI taking a more active role here.

Now, OpenAI has also recently been pretty transparent about its forthcoming plans. HumanLoop's Raza Habib was one of around 20 developers who recently got to sit with Sam Altman, who he said discussed in extensive detail what the company's near term plans were. Raza then wrote up a few of the key takeaways that he had from that conversation. First, reinforcing why Nvidia is now a trillion dollar company, Altman said that OpenAI is heavily GPU limited at the present time. Raza said that this came up throughout the discussion and that a lot of their short term plans are, in fact, delayed or dictated by their access to GPUs. Sam said that the biggest customer complaint was about the reliability and speed of the API and that most of that was directly the result of GPU shortages. Other ways that these GPU shortages are impacting OpenAI right now include, one, that their longer 32k context window can't be rolled out to more people. Right now, ChatGPT is generally limited to an 8k token window, which of course limits the amount of data that can be fed in without being chopped up. Sam also said that their current approaches to fine tuning are extremely compute intensive, having not adopted models like LoRA or adapters, and so that's being caught up in this GPU shortage. And then one that I found really interesting was that multimodality, which was demoed as part of the GPT-4 release, can't be extended to everyone until more GPUs come online.

So if GPU access is a huge bottleneck right now, what are they focused on in the short term? In 2023, it sounded like Sam and OpenAI's priorities included, one, cheaper and faster GPT, which they said was their top priority. They said that they want to drive the, quote, cost of intelligence down as far as possible. Second, longer context windows. Again, this is limited a little bit by GPU access, but is something they want to focus on. Three, the fine tuning API we just talked about, and four, a stateful API. The way that Raza describes it is, quote, When you call the chat API today, you have to repeatedly pass through the same conversation history and pay for the same tokens again and again. In the future, there will be a version of the API that remembers the conversation history. So that's what's coming up at least on the API front in the short term.

Now a couple of other interesting things from that conversation. Apparently, a number of developers in the room were nervous about building on OpenAI's API, given that they thought that OpenAI might end up releasing products that were competitive. Sam, however, said that OpenAI was not focused on releasing products beyond ChatGPT. He said that the vision for ChatGPT is to be a super smart assistant for work, but there's going to be lots of other GPT use cases that they won't get into. Altman reinforced the idea that open source is going to be an important part of the AI policy future, and said that they were considering open sourcing GPT-3. When it comes to scaling, despite all of the Internet's assertions that the age of giant AI models is already over, Sam said that OpenAI did believe that making models larger will continue to yield performance. Now lastly, one that I found really interesting, especially in light of the discussion that I was having on the episode yesterday about whether plugins are overhyped. Sam said that while developers are interested in getting access to ChatGPT plugins via the API, he didn't think that they'd be releasing that anytime soon. Right now, Altman said in his view, plugins don't have product market fit outside of browsing. The way that Raza put it was that Sam suggested that a lot of people thought they wanted their apps to be inside ChatGPT, but what they really wanted was ChatGPT in their apps. This gets back exactly to the interface questions that I was asking in that episode. In other words, are we really going to move all of our activity to that ChatGPT interface, or are there some types of experiences that still make sense in dedicated environments? Anyways, a lot of really interesting stuff there about what the near term future of OpenAI and ChatGPT might look like.

But then just today, OpenAI released some new research that has really captured people's attention. The announcement blog post is called Improving Mathematical Reasoning with Process Supervision. Dr. Jim Fan from Nvidia says, The idea is so simple that it fits in one tweet. For challenging step by step problems, give a reward at each step instead of a single reward at the end. The way that OpenAI describes it is, We've trained a model to achieve a new state of the art in mathematical problem solving by rewarding each correct step of reasoning, i.e. process supervision, instead of simply rewarding the correct final answer, which is outcome supervision. In addition to boosting performance relative to outcome supervision, process supervision also has an important alignment benefit. It directly trains the model to produce a chain of thought that is endorsed by humans. Let's read a little bit more from the introduction. They write, In recent years, large language models have greatly improved in their ability to perform complex, step by step reasoning. However, even state of the art models still produce logical mistakes often called hallucinations. Mitigating hallucinations is a critical step towards building aligned AGI. We can train reward models to detect hallucinations either using outcome supervision, which provides feedback based on a final result, or process supervision, which provides feedback on each individual step in a chain of thought. We conducted a detailed comparison of these two methods using the MATH dataset as our testbed. We find that process supervision leads to significantly better performance even when judged by outcomes.

So here's the simple chart for how these two different methodologies performed as it related to solving math problems. The outcome supervised approach in which the reward was only for the right outcome correctly solved problems about 71% of the time, while the process supervised approach got the right answer 78% of the time. So there are a couple things to note here. One is that even if there were no alignment benefits, teaching LLMs to solve math problems more accurately is a valuable thing in its own right. But second, it's not hard to understand the potential benefits here when it comes to an AI alignment perspective. Adept on Twitter wrote a really good summary, saying, This OpenAI paper might as well have been titled Moving Away From Paperclip Maxing. They took a base GPT-4, fine tuned it on a bit of math so that it understood the language as well as the output format, then no reinforcement learning. Instead, they trained and compared two reward models: one, outcome only, and two, process and outcome. This is clearly a building block to reducing the expense of human supervision for reinforcement learning. The humans move up the value chain from supervising the model to supervising the reward model to the model. The process reward model system is so human, exactly the way teachers teach math in early grades: show your work or steps. Process matters as much as outcome. It's only applied to math right now, but I can totally see a way to move this to teaching rules and laws of human society just like we do with kids. They tested on AP Chem, Physics, etc, and found the process model outperforming the objective model. It's a step away from paperclip maximization, i.e, objective goal focusing whenever necessary.

Now briefly, this idea of paperclip maxing is one of the most oft talked about AI safety or AI risk scenarios, And basically what it's shorthand for is the idea that if an AI has an objective, one of the ways that things could go badly is if it determines that humans are in some ways the barrier to that objective. So if its goal has been programmed to be make the most paper clips, what happens if it decides that humans are getting in its way? So what Adept is pointing out is that this approach to training takes some emphasis off the end objective and also rewards along the process of how an AI gets there. OpenAI also points out that it has interpretability benefits. They write, Process supervision is also more likely to produce interpretable reasoning since it encourages the model to follow a human approved process. In contrast, outcome supervision may reward an unaligned process, and it is generally harder to scrutinize. Now as OpenAI points out, there has been a sense in the past that safer methods for AI systems can sometimes lead to reduced performance. This is sometimes known as an alignment tax, and of course alignment taxes may hinder the adoption of alignment methods. However, they say in this case, when it comes to math, their process supervision model incurs a negative alignment tax, i.e. a performance benefit over other approaches. This, they say, could increase the adoption of process supervision, which we believe would have positive alignment side effects.

So what's the problem with this? Well, as Dr. Jim Fan points out, a caveat is that the process reward model does require a lot more human labeling. For example, as part of this research, OpenAI released their human feedback dataset, which was 800,000 step level labels across 70,000 solutions to 12,000 math problems. Still, even with that caveat, it's not every day that we get something that shows both improved performance of AI models as well as better alignment. And so maybe this moves some folks' p(doom) out there just a little bit down. That's it for today's AI Breakdown. If you're enjoying the show, please like, subscribe, and share. Check out the newsletter version at aibreakdown.beehive.com, or go check out The AI Breakdown podcast. Till next time, guys. Peace.