AI has always been a top fascination for anyone working with computers; it was certainly mine from my teenage years on and during its first peak in 1991 before the advent of Internet tech then occupied us all.
When Bill Gates donated to the construction of the new computer science building at Stanford, in 1996, giving us all nice offices with windows, he too said during the opening how he was anxious missing out on all the advances in AI about to happen when starting Microsoft. (He also complained back then, in 1996, that for the $6m he donated for the building, he sold MSFT stock that a few months later had appreciated a good percentage. Well, it appreciated 100x until today, making this effectively a $600m donation!). In any case, the fascination with AI is clear and there’s been many approaches and most of them quite fruitless.
When I got the chance of working at Xerox PARC in Palo Alto (along with Reid Hoffman and others) with a Stanford professor as a teenager, I sat in on David Rumelhart’s research group meetings in the psychology department at Stanford during a summer. In hindsight still, they were the most magical research group meetings I’ve ever attended. It’s the reason I embarked on a PhD at Stanford. David invented the backpropagation algorithm that is now the basis of most all AI. People don’t mention him that much any longer because, for one, he is dead and furthermore his contribution is complete standard now everywhere.
I also worked at Siemens neural network group in 1991 in Munich, coding a new type of a radial backpropagation system from scratch starting with a C compiler. I ended up presenting it at the best conference in neural network, NIPS, in 1991 as a teenager. Back then Geoff Hinton, etc. were already sitting in the first row at these conferences and everyone was worried about getting a question from him about whether the brain actually works like this. (It is with great relief in 2023 that I now read he no longer cares and in fact the algorithms we have may well be better than the brain’s).
When I joined Stanford then in computer science, the surprise was that there was exactly no interest in neural networks by any faculty. They were all “symbolic systems” guys, with no knowledge of floating point numbers and how to multiply them with each other in matrices. I sort of had to hide my interest in neural network learning. It’s quite stunning how entire communities of intelligent people, awarded jobs at prestigious places, can be entirely wrong. There was surely interest in knowing about the power of data based learning. At Xeroc PARC in 1991, I witnessed how computational linguists went through some hours each week trying to learn about conditional probabilities in statistics. It was a hopeless effort of course. Their backgrounds were too different. I also so how some of the top leaders in research at PARC took the gutsy approach of saying to today’s now dominant form of artificial intelligence that “If that’s the solution, then I wasn’t interested in the problem.” I was shocked back then about such blissfulness as I am today finding how dead wrong that was.
I just saw a Youtube of nVidia CEO Jensen interviewing one of the technical founders of OpenAI, and he recounts on how he smartly picked neural nets over Bayesian networks and other systems as a basis for his big breakthrough now. It’s a bit funny because guess what — that was a necessary but not sufficient accomplishment made by exactly each best graduates in each year for each of the past 30 years, the exact same conclusion yet landed in history in the wrong year!
Undeniably this year is different now as of 2023, finally! What’s different? Well, Google put up a lot of R&D funding and developed a critical mass of chip and research power; and nVidia nailed it with GPUs.
And recalling Rumelhart, it is equally stunning that a psychology professor figured it all out! Backpropagation is possibly as good as it gets in learning because it assigns blame for errors in the exact fraction of the contribution made to the error. What could possibly work better. The human brain does not use it — but at its peril!
I am excited that after decades (!) I am back now supporting this area with faster chips!