Public Perception

This passage from Jane Mayer’s article in The New Yorker about Robert Mercer is fascinating:

Cambridge Analytica is not the only data-driven political project that the Mercers have backed. In 2013, at a conservative conference in Palm Beach, an oil tycoon named William Lee Hanley, who had commissioned some polls from Patrick Caddell, asked him to show the data to Mercer and Bannon, who were at the event. The data showed mounting anger toward wealthy élites, who many Americans believed had corrupted the government so that it served only their interests. There was a hunger for a populist Presidential candidate who would run against the major political parties and the ruling class. The data “showed that someone could just walk into this election and sweep it,” Caddell told me. When Mercer saw the numbers, he asked for the polling to be repeated. Caddell got the same results. “It was stunning,” he said. “The country was on the verge of an uprising against its leaders. I just fell over!”

Until Election Day in 2016, Mercer and Hanley—two of the richest men in America—paid Caddell to keep collecting polling data that enabled them to exploit the public’s resentment of élites such as themselves. Caddell’s original goal was to persuade his sponsors to back an independent candidate, but they never did. In 2014, Caddell and two partners went public with what they called the Candidate Smith project, which promoted data suggesting that the public wanted a “Mr. Smith Goes to Washington” figure—an outsider—as President. During the next year or so, Caddell’s poll numbers tilted more and more away from the establishment. Caddell’s partner Bob Perkins, an advertising executive and a former finance director of the Republican Party, told me, “By then, it was clear there wouldn’t be a third-party candidate. But we thought that a Republican who harnessed the angst had a real chance.” At one point, Caddell tested all the declared Presidential candidates, including Trump, as a possible Mr. Smith. “People didn’t think Trump had the temperament to be President,” Caddell said. “He clearly wasn’t the best Smith, but he was the only Smith. He was the only one with the resources and the name recognition.” As Bernie Sanders’s campaign showed, the populist rebellion wasn’t partisan. Caddell worried, though, that there were dark undertones in the numbers: Americans were increasingly yearning for a “strong man” to fix the country.

Caddell circulated his research to anyone who would listen, and that included people inside the Trump campaign. “Pat Caddell is like an Old Testament prophet,” Bannon said. “He’s been talking about alienation of the voters for twenty-five years, and people didn’t pay attention—but he’s a brilliant guy, and he nailed it.” The political consultant and strategist Roger Stone, who is a longtime Trump confidant, was fascinated by the research, and he forwarded a memo about it to Trump. Caddell said that he spoke with Trump about “some of the data,” but noted, “With Trump, it’s all instinct—he is not exactly a deep-dive thinker.”

Robert Mercer, too, was kept informed. Perkins said, “He just loves the numbers. Most people say, ‘Tell me what you think—don’t show me the numbers.’ But he’s, like, ‘Give me the numbers!’ ”

Agree or not, the group’s deep understanding of public perception and belief in data (“When Mercer saw the numbers, he asked for the polling to be repeated”) is incredibly impressive.

Approaches to Data and Machine learning

I've found Ray Dalio's book Principles incredibly insightful. One area was around the use of technology for decision making.

Dalio first sets the stage for the relevant principles in sharing his life story. I've pulled out a few passages and emphasized things I found particularly interesting.

From very early on, whenever I took a position in the markets, I wrote down the criteria I used to make my decision. Then, when I closed out a trade, I could reflect on how well these criteria had worked. It occurred to me that if I wrote those criteria into formulas (now more fashionably called algorithms) and then ran historical data through them, I could test how well my rules would have worked in the past. ...

We tested the systems going as far back as we could, typically more than a century, in every country for which we had data, which gave me great perspective on how the economic/market machine worked through time and how to bet on it. Doing this helped educate me and led me to refine my criteria so they were timeless and universal. Once I vetted those relationships, I could run data through the systems as it flowed at us in real time and the computer could work just as my brain worked in processing it and making decisions.

The result was Bridgewater’s original interest rate, stock, currencies, and precious metals systems, which we then combined into one system for managing our portfolio of bets. Our system was like an EKG on the economy’s vital signs; as they changed, we changed our positions. However, rather than blindly following the computer’s recommendations, I would have the computer work in parallel with my own analysis and then compare the two. When the computer’s decision was different from mine, I would examine why. Most of the time, it was because I had overlooked something. In those cases, the computer taught me. But sometimes I would think about some new criteria my system would’ve missed, so I would then teach the computer. We helped each other. It didn’t take long before the computer, with its tremendous processing power, was much more effective than me. This was great, because it was like having a chess grandmaster helping me plot my moves, except this player operated according to a set of criteria that I understood and believed were logical, so there was no reason for us to ever fundamentally disagree.

The computer was much better than my brain in “thinking” about many things at once, and it could do it more precisely, more rapidly, and less emotionally. And, because it had such a great memory, it could do a better job of compounding my knowledge and the knowledge of the people I worked with as Bridgewater grew. ...

Of course, we always had the freedom to override the system, which we did less than 2 percent of the time—mostly to take money off the table during extraordinary events that weren’t programmed, like the World Trade Center going down on 9/11. While the computer was much better than our brains in many ways, it didn’t have the imagination, understanding, and logic that we did. That’s why our brains working with the computer made such a great partnership.

...

Over the last three decades of building these systems we have incorporated many more types of rules that direct every aspect of our trading. Now, as real-time data is released, our computers parse information from over 100 million datasets and give detailed instructions to other computers in ways that make logical sense to me. If I didn’t have these systems, I’d probably be broke or dead from the stress of trying so hard. We certainly wouldn’t have done as well in the markets as we have. As you will see later, I am now developing similar systems to help us make management decisions. I believe one of the most valuable things you can do to improve your decision making is to think through your principles for making decisions, write them out in both words and computer algorithms, back-test them if possible, and use them on a real-time basis to run in parallel with your brain’s decision making.

Dalio then articulates the specific principles in "Life Principles #5: Learn How to Make Decisions Effectively."

Life Principle 5.11: Convert your principles into algorithms and have the computer make decisions alongside you.

If you can do that, you will take the power of your decision making to a whole other level. In many cases, you will be able to test how that principle would have worked in the past or in various situations that will help you refine it, and in all cases, it will allow you to compound your understanding to a degree that would otherwise be impossible. It will also take emotion out of the equation. ... By developing a partnership with your computer alter ego in which you teach each other and each do what you do best, you will be much more powerful than if you went about your decision making alone. The computer will also be your link to great collective decision making, which is far more powerful than individual decision making, and will almost certainly advance the evolution of our species.

Life Principles 5.12: Be cautious about trusting AI without having deep understanding.

I worry about the dangers of AI in cases where users accept—or, worse, act upon—the cause-effect relationships presumed in algorithms produced by machine learning without understanding them deeply.

Before I explain why, I want to clarify my terms. “Artificial intelligence” and “machine learning” are words that are thrown around casually and often used as synonyms, even though they are quite different. I categorize what is going on in the world of computer-aided decision making under three broad types: expert systems, mimicking, and data mining (these categories are mine and not the ones in common use in the technology world).

Expert systems are what we use at Bridgewater, where designers specify criteria based on their logical understandings of a set of cause-effect relationships, and then see how different scenarios would emerge under different circumstances.

But computers can also observe patterns and apply them in their decision making without having any understanding of the logic behind them. I call such an approach “mimicking.” This can be effective when the same things happen reliably over and over again and are not subject to change, such as in a game bounded by hard-and-fast rules. But in the real world things do change, so a system can easily fall out of sync with reality. 

The main thrust of machine learning in recent years has gone in the direction of data mining, in which powerful computers ingest massive amounts of data and look for patterns. While this approach is popular, it’s risky in cases when the future might be different from the past. Investment systems built on machine learning that is not accompanied by deep understanding are dangerous because when some decision rule is widely believed, it becomes widely used, which affects the price. In other words, the value of a widely known insight disappears over time. Without deep understanding, you won’t know if what happened in the past is genuinely of value and, even if it was, you will not be able to know whether or not its value has disappeared—or worse. It’s common for some decision rules to become so popular that they push the price far enough that it becomes smarter to do the opposite.

Remember that computers have no common sense. For example, a computer could easily misconstrue the fact that people wake up in the morning and then eat breakfast to indicate that waking up makes people hungry. I’d rather have fewer bets (ideally uncorrelated ones) in which I am highly confident than more bets I’m less confident in, and would consider it intolerable if I couldn’t argue the logic behind any of my decisions. A lot of people vest their blind faith in machine learning because they find it much easier than developing deep understanding. For me, that deep understanding is essential, especially for what I do.

Just after finishing Principles, I picked up Judea Pearl's book The Book of Why and found it echoed these same ideas. Judea Pearl won the Turing prize in 2011 for his contributions to artificial intelligence around probabilistic and causal reasoning. He's credited with the creation of Bayesian networks and then built on that contribution with his models of causality. A basic message of the book is that statistics focused to a fault on correlation over causality, even while proclaiming "correlation is not causation." This led to a focus on "big data" and "data science," which can...

...tell you that the people who took a medicine recovered faster than those who did not take it, but they can't tell you why. Maybe those who took the medicine did so because they could afford it and would have recovered just as fast without it. 

To address this gap, Pearl offers an "inference engine" that combines data with causal knowledge to produce answers to queries of interest:

You can find the introductory chapter of The Book of Why here on Pearl's website. (The model above is on p. 12.)

This got my attention because the approach echoed Dalio's approach of systematizing principles and feeding them to the computer. However, what's lacking in the model above and different than Dalio's approach is the return path to Knowledge, or as Dalio says: "developing a partnership with your computer alter ego in which you teach each other and each do what you do best." So I added a line to the inference engine, making it an inference and teaching engine, more of a partnership:

I'm early in my learning, exploration, and curiosity mode here. I'm curious about the extent of usage and effectiveness of this approach. Is Judea Pearl something significant and new here? Is this approach widely used? Is it effective? Can it be effective? Has something significant been overlooked here in the excitement around deep learning?

I suspect the approach is not widely used. What appears to be far more common are the "black box" approaches Dalio mentions—algorithmic trading, high frequency trading, search algorithms, recommendation algorithms. To be clear, these are valuable and revolutionary. AlphaGo was an incredible success (quickly surpassed by AlphaGo Zero). But I suspect that the wild success of these approaches is masking another equally impactful, albeit less flashy approach, of improved decision making—a partnership between man and machine. 

Second-order effects: What can exist?

I started reading Paul Allen's memoir, Idea Man, and in setting the stage for the founding of Microsoft, Allen talks about monitoring the release of Intel's 4004 chip in November 1971 and then the much more powerful Intel 8008 four months later. 

My really big ideas have all begun with a stage-setting development—in this case, the evolution of Intel's early microprocessor chips. Then I ask a few basic questions: Where is the leading edge of discovery headed? What should exist but doesn't yet? How can I help create something to meet the need, and who might be enlisted to join the cause?

This is clearly an effective approach...

Consider Reed Hastings and Netflix. The below is from A16Z podcast "Tech and Entertainment in the 'Era of Mass Customization.'" Marc Andreessen recalls how Reed Hastings had invited him to discuss streaming video in 2003-2004, and Andreessen told him, "Eh, it'll never work...there are too many technical impediments." Andreessen asked Hastings to walk through how he had thought about it, and here's what Hastings said:

On bandwidth trend. If you looked at median residential bandwidth from 1980 forward, it's completely smooth. We've gone through multiple technologies: dial-up, DSL, cable, fiber. Interestingly, the speeds at which streaming was going to become practical (1-2 Mbps) was completely knowable. Now, we weren't that smart. We knew it was sometime in the future. But—right from the beginning in 1997 we envisioned it. When I was at Stanford you take the classic Tannenbaum computer networking class, and it makes you think about networks differently. You have to calculate the bandwidth of a station wagon filled with backup tapes driving across the country. So you start to think about networks differently. So in 1997 when a friend told me about DVD, I thought, "Oh my god, that's the station wagon." That's this 5 GB packet that you could mail. (High throughput, high latency—24 hour latency but good throughput.) It's that cross-fertilization of metaphor. So we always viewed DVD by mail as a digital distribution network, and we knew that eventually [we would be delivering over the internet]. And that's why we called the company Netflix and not something like "DVD by mail." So we had a slight advantage that we didn't fall in love with our first business. We knew it was a path to something else.

On timing. We knew it was an issue. We went public in 2002, and we said eventually internet delivery would come. We didn't do much with it until 2005, when we saw YouTube. That's when we realized: it's beginning. And that's when we started that effort and launched in 2007. As Clay described, the key thing is not getting into the new business. Lots of companies do that. We knew the key was focusing on it—how this is going to grow into it's own business. But it was too young to do that. It didn't have enough content so we couldn't sell a streaming content service on its own. So we "hybridized" it with DVD. As usage and content grew, we knew we could split them apart. That went until 2010. We did our first test of the streaming-only service in Canada—a new country that wasn't used to our DVD service. We wanted to understand: with the content we had, could we build a service that had word-of-mouth and could grow. It was a rocket ship. In our first three days, we got as many subscribers as we thought we'd have in three months. It was clear you could position Netflix as streaming-only.

On "Innovator's Dilemma" dynamics (old business v new). One of the most painful moments on that journey was that DVD business was all the revenue and profit. We were hybridizing it with streaming, and we were getting more streaming attention (and executives). But DVD was getting more attention. So we realized we had to kick out DVD executives from the main management meeting. They weren't adding value in streaming discussion. We always compared ourselves to a "streaming pure play"—what would we do? [Key is to separate them: Apple did the same thing, just in reverse.]

Consider Marc Andreessen and A16Z. In the podcast, "Software programs the world," Andreessen talks about the falling costs of chips: 

Let me go to the foundations. Moore's Law has flipped. This has happened over the last 7-8 years. For many years, Moore's Law was a process of the chip industry bringing out a new chip every 1.5 years that was twice as fast as the previous chip at the same price. That continued for 40-50 years. That resulted in everything from mainframes to PCs to smartphones. About 7-8 years ago, that process topped out at 3 GHz. Some people said progress was going to stall. I think what has happened is that Moore's Law has flipped. The dynamic now is instead of increased performance is reduced cost. You now have this dynamic where every 1 to 1.5 years, the chip companies release a chip that is the same speed but half the cost. This is a massive deflationary force in the technology force, and I suspect in the economy in general. Basically, computing is becoming free. What we do in this business is we chart out the graphs and assume that we get to the end point. So what we assume is that chips are going to be free. Which means chips will be embedded in everything. And we've never lived in that sort of world. 

Consider Roelof Botha. In Venture Capitalists at Work, Roelof Botha talks about the investment process for YouTube:

On enabling technologies: bandwidth, browsers, and devices. Video had come and gone—for example, with RealPlayer—but never became a huge hit. So what changed that made YouTube possible? At Sequoia, we'd investigated related ideas back in 2004 and 2005. We were keeping an eye on broadband penetration in the US—where was the topping point at which a large enough percentage of US consumers had decent home internet connections? And what new services would that unlock? ... We also kept tabs on the state of browser technology. ... We were listening to semiconductor companies that made the components for handheld devices—devices that made it easy for consumers to capture pictures and videos. 

On consumer behavior. There was the emergence of blogging, photo-sharings services like Flickr, and review sites. People wanted to express themselves through text and pictures; the next natural step was video. 

On the value proposition and product. When I first encountered the website, I uploaded a few videos. In just a few minutes I'd posted them and e-mailed them out. People were watching videos that had been sitting on my hard drive for years. Other video sites at the time had clients that you had to download. Even with the browser-based ones, their products just weren't as good. 

On founder quality. I was lucky enough to know the founders from PayPal. I knew how good they were. And they were fantastic talent magnets. When Google acquired YouTube there were only fifty-five employees. 

And also... We have a predisposition toward the long view. If you were to hold onto the shares of every IPO company we invested in until today, you would have made significantly more money than if you were to invest with us at the venture stage alone. ... People overestimate the impact of technological shifts in the short run and underestimate them in the long run. But the long-run effects are just so spectacular. 

In each of the cases, there were fundamental underlying shifts, or clear trends indicating a tipping point. The second-order effect of a key (or in some cases, many) underlying shifts—chip speeds, chip costs, bandwidth, browser technology, handheld capabilities, consumer behavior, etc.—enabled something new and valuable.

Linger among master-thinkers

I came across this on Farnam Street, and it really resonated. Seneca said (emphasis mine):

The primary indication, to my thinking, of a well-ordered mind is a man’s ability to remain in one place and linger in his own company. Be careful, however, lest this reading of many authors and books of every sort may tend to make you discursive and unsteady. You must linger among a limited number of master-thinkers, and digest their works, if you would derive ideas which shall win firm hold in your mind. Everywhere means nowhere. When a person spends all his time in foreign travel, he ends by having many acquaintances, but no friends. And the same thing must hold true of men who seek intimate acquaintance with no single author, but visit them all in a hasty and hurried manner. Food does no good and is not assimilated into the body if it leaves the stomach as soon as it is eaten; nothing hinders a cure so much as frequent change of medicine; no wound will heal when one salve is tried after another; a plant which is often moved can never grow strong. There is nothing so efficacious that it can be helpful while it is being shifted about. And in reading of many books is distraction.

Collective Unconscious

I keep coming back to the below passage from the essay "The Child and the Shadow" in Ursula Le Guin's book, The Language of the Night: Essays on Fantasy and Science Fiction. The essay is an exploration of how fantasy uses symbols and archetypes to communicate with our unconscious: "The great fantasies, myths, and tales are indeed like dreams: they speak from the unconscious to the unconscious in the language of unconscious—symbols and archetype." In the passage below, Le Guin is introducing the reader to Carl Jung's ideas, whom she describes as "the psychologist whose ideas on art are the most meaningful to artists."

Jung's terminology is notoriously difficult, as he kept changing meanings the way a growing tree changes leaves. I will try to define a few of the key terms in an amateurish way without totally misrepresenting them. Very roughly, then, Jung saw the ego, what we usually call the self, as only a part of the Self, the part of it which we are consciously aware of. The ego "revolves around the Self as the earth around the Sun," he says. The Self is transcendent, much larger than the ego; it is not a private possession, but collective—that is, we share it with all other human beings, and perhaps with all beings. It may indeed be our link with what is called God. Now this sounds mystical, and it is, but it's also exact and practical. All Jung is saying is that we are fundamentally alike; we all have the same general tendencies and configurations in our psyche, just as we all have the same general kind of lungs and bones in our body. Human being all look roughly alike; they also think and feel alike. And they are all part of the universe.

The ego, the little private individual consciousness, knows this, and it knows that if it's not to be trapped in the hopeless silence of autism it must identify with something outside itself, beyond itself, larger than itself. If it's weak, or if it's offered nothing better, what it does is identify with the "collective consciousness." That is Jung's term for the lowest common denominator of all the little egos added together, the mass mind, which consists of such things as cults, creeds, fads, fashions, status-seeking, conventions, received beliefs, advertising, popcult, all the isms, all the ideologies, all the hollow forms of communication and "togetherness" that lack real communion or real sharing. The ego, accepting these empty forms, becomes a member of the "lonely crowd." To avoid this, to attain real community, it must turn inward, away from the crowd, to the source: it must identify with its own deeper regions, the great unexplored regions of the Self. These regions of the psyche Jung calls the "collective unconscious," and it is in them, where we all meet, that he sees the source of true community; of felt religion; of art, grace, spontaneity, and love. 

This element of turning inward to connect with humanity rather than outward is compelling. Frankly, I don't know if it's true yet, but I find it fascinating, and I'm curious to learn more. So I ordered Carl Jung's book The Undiscovered Self.