path: root/books/sociology/who-owns-the-future.md

[[!meta title="Who owns the future?"]]

* Author: Jaron Lanier
* Year: 2013
* Publisher: Simon & Schuster

## Index

* Star system versus the bell curve as network designs.
* Siren Servers: narcissism, hyperamplified risk aversion, and extreme information asymmetry.
* Siren Servers and Maxwell’s Demon.
* Disruptive innovation as the tedious scheme to shrink markets.
* Science isn't automatic.
* Nine dismal humors of futurism, and a hopeful one.
* Marx as one of the first technology writers (when discussing Luddites).
* Human obsolescence is avoidable.
* Keynes Considered as a Big Data Pioneer.
* Amazon's Mechanical Turk.
* Humanistic information economics.
* What is experience? If personal experience were missing from the universe, how would things be different?
* Gurus and New Age at the Sillicon Valley: Gurdjieff, Steve Jobs.

## Prelude

    Instagram isn’t worth a billion dollars just because those thirteen employees
    are extraordinary. Instead, its value comes from the millions of users who
    contribute to the network without being paid for it. Networks need a great
    number of people to participate in them to generate significant value. But when
    they have them, only a small number of people get paid. That has the net effect
    of centralizing wealth and limiting overall economic growth.

    [...]

    By “digital networking” I mean not only the Internet and the Web, but also
    other networks operated by outfits like financial institutions and intelligence
    agencies. In all these cases, we see the phenomenon of power and money becoming
    concentrated around the people who operate the most central computers in a
    network, undervaluing everyone else. That is the pattern we have come to
    expect, but it is not the only way things can go.

## The Price of Heaven

    Utopians presume the advent of abundance not because it will be affordable, but
    because it will be free, provided we accept surveillance.

    Starting back in the early 1980s, an initially tiny stratum of gifted
    technologists conceived new interpretations of concepts like privacy, liberty,
    and power. I was an early participant in the process and helped to formulate
    many of the ideas I am criticizing in this book. What was once a tiny
    subculture has blossomed into the dominant interpretation of computation and
    software-mediated society.

    One strain of what might be called “hacker culture” held that liberty means
    absolute privacy through the use of cryptography. I remember the thrill of
    using military-grade stealth just to argue about who should pay for a pizza at
    MIT in 1983 or so.

    On the other hand, some of my friends from that era, who consumed that pizza,
    eventually became very rich building giant cross-referenced dossiers on masses
    of people, which were put to use by financiers, advertisers, insurers, or other
    concerns nurturing fantasies of operating the world by remote control.

    It is typical of human nature to ignore hypocrisy. The greater a hypocrisy, the
    more invisible it typically becomes, but we technical folk are inclined to seek
    an airtight whole of ideas. Here is one such synthesis—of cryptography for
    techies and massive spying on others—which I continue to hear fairly often:
    Privacy for ordinary people can be forfeited in the near term because it will
    become moot anyway.

    Surveillance by the technical few on the less technical many can be tolerated
    for now because of hopes for an endgame in which everything will become
    transparent to everyone. Network entrepreneurs and cyber-activists alike seem
    to imagine that today’s elite network servers in positions of information
    supremacy will eventually become eternally benign, or just dissolve.

    Bizarrely, the endgame utopias of even the most ardent high-tech libertarians
    always seem to take socialist turns. The joys of life will be too cheap to
    meter, we imagine. So abundance will go ambient.

    This is what diverse cyber-enlightened business concerns and political groups
    all share in common, from Facebook to WikiLeaks. Eventually, they imagine,
    there will be no more secrets, no more barriers to access; all the world will
    be opened up as if the planet were transformed into a crystal ball. In the
    meantime, those true believers encrypt their servers even as they seek to
    gather the rest of the world’s information and find the best way to leverage
    it.

    It is all too easy to forget that “free” inevitably means that someone else
    will be deciding how you live.

## Just Blurt the Idea Out

    So we begin with the simple question of how to design digital networks to
    deliver more help than harm in aligning human intention to meet great
    challenges. A starting point for an answer can be summarized: “Digital
    information is really just people in disguise.”

### Aristotle frets

    Aristotle directly addressed the role of people in a hypothetical high-tech
    world: If every instrument could accomplish its own work, obeying or
    anticipating the will of others, like the statues of Daedalus, or the tripods
    of Hephaestus, which, says the poet, of their own accord entered the assembly
    of the Gods; if, in like manner, the shuttle would weave and the plectrum touch
    the lyre without a hand to guide them, chief workmen would not want servants,
    nor masters slaves.1

    At this ancient date, a number of possibilities were at least slightly visible
    to Aristotle’s imagination. One was that the human condition was in part a
    function of what machines could not do. Another was that it was possible to
    imagine, at least hypothetically, that machines could do more. The synthesis
    was also conceived: Better machines could free and elevate people, even slaves.

    If we could show Aristotle the technology of our times, I wonder what he would
    make of the problem of unemployment. Would he take Marx’s position that better
    machines create an obligation (to be carried out by political bodies) to
    provide care and dignity to people who no longer need to work? Or would
    Aristotle say, “Kick the unneeded ones out of town. The polis is only for the
    people who own the machines, or do what machines still cannot do.” Would he
    stand by idly as Athens was eventually depopulated?

    I’d like to think the best of Aristotle, and assume he would realize that both
    choices are bogus; machine autonomy is nothing but theater. Information needn’t
    be thought of as a freestanding thing, but rather as a human product. It is
    entirely legitimate to understand that people are still needed and valuable
    even when the loom can run without human muscle power. It is still running on
    human thought.

    [...]

    Note: How prescient that Aristotle chose musical instruments and looms as his
    examples for machines that might one day operate automatically! These two types
    of machines did indeed turn out to be central to the prehistory of computation.
    The Jacquard programmable loom helped inspire calculating engines, while music
    theory and notation helped further the concept of abstract computation, as when
    Mozart wrote algorithmic, nondeterministic music incorporating dice throws.
    Both developments occurred around the turn of the 19th century.

    [...]

    Aristotle seems to want to escape the burden of accommodating lesser people.
    His quote about self-operating lutes and looms could be interpreted as a
    daydream that better technology will free us to some degree from having to deal
    with one another.

    It’s not as if everyone wanted to be closer to all of humanity when cities
    first formed. Athens was a necessity first, and a luxury second. No one wants
    to accommodate the diversity of strangers. People deal with each other
    politically because the material advantages are compelling. We find relative
    safety and sustenance in numbers. Agriculture and armies happened to work
    better as those enterprises got bigger, and cities built walls.

    But in Aristotle’s words you get a taste of what a nuisance it can be to
    accommodate others. Something was lost with the advent of the polis, and we
    still dream of getting it back.

    [...]

    The reward for a Roman general, upon retiring after years of combat, was a plot
    of land he could farm for himself. To be left alone, to be able to live off the
    land with the illusion of no polis to bug you, that was the dream. The American
    West offered that dream again, and still loathes giving it up. Justice Louis
    Brandeis famously defined privacy as the “right to be left alone.”

    In every case, however, abundance without politics was an illusion that could
    only be sustained in temporary bubbles, supported by armies. The ghosts of the
    losers haunt every acre of easy abundance. The greatest beneficiaries of
    civilization use all their power to create a temporary illusion of freedom from
    politics. The rich live behind gates, not just to protect themselves, but to
    pretend to not need anyone else, if only for a moment. In Aristotle’s quote, we
    find the earliest glimmer of the hope that technological advancement could
    replace territorial conquest as a way of implementing an insulating bubble
    around a person.

    [...]

    People naturally seek the benefits of society, meaning the accommodation of
    strangers, while avoiding direct vulnerabilities to specific others as much as
    possible. This is a clichéd criticism of the online culture of the moment.
    People have thousands of “friends” and yet stare at a little screen when in the
    proximity of other people. As it was in Athens, so it is online.

## Money

    Money might have begun as a mnemonic counter for assets you couldn’t keep under
    direct observation, like wandering sheep. A stone per sheep, so the shepherd
    would be confident all had been reunited after a day at pasture. In other
    words, artifacts took on information storage duties.

    [...]

    Ancient money was information storage that represented events in the past. To
    the ears of many a financier, at this early stage “money” had not been born
    yet, only accounting. That kind of money can be called “past-oriented money.”

## Noise and luck

    Consider the problem of noise, or what is known as luck in human affairs.

    [...]

    And yet the rewards of winning and losing are vastly different. While some
    critics might have aesthetic or ethical objections to winner-take-all outcomes,
    a mathematical problem with them is that noise is amplified. Therefore, if a
    societal system depends too much on winner-take-all contests, then the acuity
    of that system will suffer. It will become less reality-based.

    When a bell curve distribution is appreciated as a bell curve instead of as a
    winner-take-all distribution, then noise, luck, and conceptual ambiguity aren’t
    amplified. It makes statistical sense to talk about average intelligence or
    high intelligence, but not to identify the single most intelligent person.

## Letting Bell Curves Be Bell Curves

    In a star system, the top players are rewarded tremendously, while almost
    everyone else—facing in our era an ever-larger, more global body of competitive
    peers—is driven toward poverty (because of competition or perhaps automation).

## Absolutism

    Being an absolutist is a certain way to become a failed technologist.

    Markets are an information technology. A technology is useless if it can’t be
    tweaked. If market technology can’t be fully automatic and needs some
    “buttons,” then there’s no use in trying to pretend otherwise. You don’t stay
    attached to poorly performing quests for perfection. You fix bugs.

## The Taste of Politics

    Despite my favorable regard for organized labor, for the purposes of this book
    I have to focus somewhat on certain failings. The problems of interest to me
    are not really with the labor movement, but with the nature of levees. What
    might be called “upper-class levees,” like exclusive investment funds, have
    been known to blur into Ponzi schemes or other criminal enterprises, and the
    same pattern exists for levees at all levels.

    Levees are more human than algorithmic, and that is not an entirely good thing.
    Whether for the rich or the middle class, levees are inevitably a little
    conspiratorial, and conspiracy naturally attracts corruption. Criminals easily
    exploited certain classic middle-class levees; the mob famously infiltrated
    unions and repurposed music royalties as a money-laundering scheme.

    Levees are a rejection of unbridled algorithm and an insertion of human will
    into the flow of capital. Inevitably, human oversight brings with it all the
    flaws of humans. And yet despite their rough and troubled nature, antenimbosian
    levees worked well enough to preserve middle classes despite the floods,
    storms, twisters, and droughts of a world contoured by finance. Without our
    system of levees, rising like a glimmering bell-curved mountain of rice
    paddies, capitalism would probably have decayed into Marx’s “attractor
    nightmare” in which markets decay into plutocracy.

## A First Pass at a Definition

    A Siren Server, as I will refer to such a thing, is an elite computer,
    or coordinated collection of computers, on a network. It is
    characterized by narcissism, hyperamplified risk aversion, and extreme
    information asymmetry. It is the winner of an all-or-nothing contest,
    and it inflicts smaller all-or-nothing contests on those who interact
    with it.

    Siren Servers gather data from the network, often without having to pay
    for it. The data is analyzed using the most powerful available
    computers, run by the very best available technical people. The results
    of the analysis are kept secret, but are used to manipulate the rest of
    the world to advantage.

    That plan will always eventually backfire, because the rest of the world
    cannot indefinitely absorb the increased risk, cost, and waste dispersed
    by a Siren Server. Homer sternly warned sailors to not succumb to the
    call of the sirens, and yet was entirely complacent about Hephaestus’s
    golden female robots. But Sirens might be even more dangerous in
    inorganic form, because it is then that we are really most looking at
    ourselves in disguise. It is not the siren who harms the sailor, but the
    sailor’s inability to think straight. So it is with us and our machines.

    Siren Servers are fated by their nature to sow illusions. They are
    cousins to another seductive literary creature, star of the famous
    thought experiment known as Maxwell’s Demon, after the great 19th
    century physicist James Clerk Maxwell. The demon is an imaginary
    creature that, if it could only exist, would be able to implement a
    perpetual motion machine and perform other supernatural tricks.

    Maxwell’s Demon might be stationed at a tiny door separating two
    chambers filled with water or air. It would only allow hot molecules to
    pass one way, and cold molecules to pass in the opposite direction.
    After a while, one side would be hot and the other cold, and you could
    let them mix again, rushing together so quickly that the stream could
    run a generator. In that way, the tiny act of discriminating between hot
    and cold would produce infinite energy, because you could repeat the
    process forever.

    The reason Maxwell’s Demon cannot exist is that it does take resources
    to perform an act of discrimination. We imagine computation is free, but
    it never is. The very act of choosing which particle is cold or hot
    itself becomes an energy drain and a source of waste heat. The principle
    is also known as “no free lunch.”

    We do our best to implement Maxwell’s Demon whenever we manipulate
    reality with our technologies, but we can never do so perfectly; we
    certainly can’t get ahead of the game, which is known as entropy. All
    the air conditioners in a city emit heat that makes the city hotter
    overall. While you can implement what seems to be a Maxwell’s Demon if
    you don’t look too far or too closely, in the big picture you always
    lose more than you gain.

    Every bit in a computer is a wannabe Maxwell’s Demon, separating the
    state of “one” from the state of “zero” for a while, at a cost. A
    computer on a network can also act like a wannabe demon if it tries to
    sort data from networked people into one or the other side of some
    imaginary door, while pretending there is no cost or risk involved. For
    instance, a Siren Server might allow only those who would be cheap to
    insure through a doorway (to become insured) in order to make a
    supernaturally ideal, low-risk insurance company. Such a scheme would
    let high-risk people pass one way, and low-risk ones pass the other way,
    in order to implement a phony perpetual motion machine out of a human
    society. However, the uninsured would not cease to exist; rather, they
    would instead add to the cost of the whole system, which includes the
    people who run the Siren Server. A short-term illusion of risk reduction
    would actually lead to increased risk in the longer term.

## Candy

    The primary business of digital networking has come to be the creation of
    ultrasecret mega-dossiers about what others are doing, and using this
    information to concentrate money and power. It doesn’t matter whether the
    concentration is called a social network, an insurance company, a derivatives
    fund, a search engine, or an online store. It’s all fundamentally the same.
    Whatever the intent might have been, the result is a wielding of digital
    technology against the future of the middle class.

    [...]

    We loved the crazy cheap easy mortgages, motivated by crazed overleveraging. We
    love the free music, enabled by crazed copying. We love cheap online prices,
    offered by what would have once seemed like national intelligence agencies.
    These newer spy services do not struggle on behalf of our security, but instead
    figure out just how little payment everyone in the chain can be made to accept.
    We are not benefiting from the benevolence of some artificial intelligence
    superbeing. We are exploiting each other off the books while those
    concentrating our information remain on the books. We love our treats but will
    eventually discover we are depleting our own value.

    That’s how we can have economic troubles despite there being so much wealth in
    the system, and during a period of increasing efficiencies. Great fortunes are
    being made on shrinking the economy instead of growing it. It’s not a result of
    some evil scheme, but a side effect of an idiotic elevation of the fantasy that
    technology is getting smart and standing on its own, without people.

## From Autocollate to Autocollude

    It seems as though online services are bringing bargains to everyone, and yet
    wealth disparity is increasing while social mobility is decreasing. If everyone
    were getting better options, wouldn’t everyone be doing better as well?

## From the Customer’s Point of View

    Wal-Mart confronted the ordinary shopper with two interesting pieces of news.
    One was that stuff they wanted to buy got cheaper, which of course was great.
    This news was delivered first, and caused cheering.

    But there was another piece of news that emerged more gradually. It has often
    been claimed that Wal-Mart plays a role in the reduction of employment
    prospects for the very people who tend to be its customers.1 Wal-Mart has
    certainly made the world more efficient in a certain sense. It moved
    manufacturing to any spot in the world that could accomplish it at the very
    lowest cost; it rewarded vendors willing to cut corners to the maximum degree.

    [...]

    All Siren Servers deliver dual messages similar to the pair pioneered by
    Wal-Mart. On the one hand, “Good news! Treats await! Information systems have
    made the world more efficient for you.”

    On the other hand, a little later: “It turns out you, your needs, and your
    expectations are not maximally efficient from the lofty point of view of our
    server. Therefore, we are reshaping the world so that in the long term, your
    prospects are being reduced.”

    The initial benefits don’t remotely balance the long-term degradations.
    Initially you made some money day trading or getting an insanely easy loan, or
    saved some money couch-surfing or by using coupons from an Internet site, but
    then came the pink slip, the eviction notice, and the halving of your savings
    when the market drooped. Or you loved getting music for free, but then realized
    that you couldn’t pursue a music career yourself because there were hardly any
    middle-class, secure jobs left in what was once the music industry. Maybe you
    loved the supercheap prices at your favorite store, but then noticed that the
    factory you might have worked for closed up for good.

## Financial Siren Servers

    The schemes were remarkably similar to Silicon Valley designs. A few of them
    took as input everything they possibly could scrape from the Internet as well
    as other, proprietary networks. As in Google’s data centers, stupendous
    correlative algorithms would crunch on the whole ’net’s data overnight, looking
    for correlations. Maybe a sudden increase in comments about mosquito bites
    would cause an automatic, instant investment in a company that sold lotions.
    Actually, that’s an artificially sensible example. The real examples made no
    sense to humans. But money was made, and fairly reliably.

    Note: It should be pointed out that if only one Siren Server is milking a
    particular fluctuation in this way, a reasonable argument could be made that a
    service is being performed, in that the fluctuation reveals inefficiency, and
    the Siren is canceling it out. However, when many Sirens milk the same
    fluctuation, they lock into a feedback system with each other and inadvertently
    conspire to milk the rest of the world to no purpose.

    [...]

    What is absolutely essential to a financial Siren Server, however, is a
    superior information position. If everyone else knew what you were doing, they
    could securitize you. If anyone could buy stock in a mathematical “sure thing”
    scheme, then the benefits of it would be copied like a shared music file, and
    spread out until it was nullified. So, in today’s world your mortgage can be
    securitized in someone else’s secretive bunker, but you can’t know about the
    bunker and securitize it. If it weren’t for that differential, the new kind of
    sure thing wouldn’t exist.

## If Life Gives You EULAs, Make Lemonade

    The information economy that we are currently building doesn’t really embrace
    capitalism, but rather a new form of feudalism.

## Your Lack of Privacy Is Someone Else’s Wealth

    Occasionally the rich embrace a new token and drive up its value. The fine art
    market is a great example. Expensive art is essentially a private form of
    currency traded among the very rich. The better an artist is at making art that
    can function this way, the more valuable the art will become. Andy Warhol is
    often associated with this trick, though Pablo Picasso and others were
    certainly playing the same game earlier. The art has to be stylistically
    distinct and available in suitable small runs. It becomes a private form of
    money, as instantly recognizable as a hundred-dollar bill.

    A related trend of our times is that troves of dossiers on the private lives
    and inner beings of ordinary people, collected over digital networks, are
    packaged into a new private form of elite money. The actual data in these
    troves need not be valid. In fact, it might be better that it is not valid, for
    actual knowledge brings liabilities.

## The Nature of Our Confusion

    Our core illusion is that we imagine big data as a substance, like a natural
    resource waiting to be mined. We use terms like data-mining routinely to
    reinforce that illusion. Indeed some data is like that. Scientific big data,
    like data about galaxy formation, weather, or flu outbreaks, can be gathered
    and mined, just like gold, provided you put in the hard work.

    But big data about people is different. It doesn’t sit there; it plays against
    you. It isn’t like a view through a microscope, but more like a view of a
    chessboard.

## The Most Elite Naïveté

    As technology advances, Siren Servers will be ever more the objects of the
    struggle for wealth and power, because they are the only links in the chain
    that will not be commoditized. If present trends continue, you’ll always be
    able to seek information supremacy, just as old-fashioned barons could struggle
    for supremacy over land or natural resources. A new energy cycle will someday
    make oil much less central to geopolitics, but the information system that
    manages that new kind of energy could easily become an impregnable castle. The
    illusory golden vase becomes more and more valuable.

### Mapping out where the conversation can go

    An endgame for civilization has been foreseen since Aristotle. As technology
    reaches heights of efficiency, civilization will have to find a way to resolve
    a peculiar puzzle: What should the role of “extra” humans be if not everyone is
    still strictly needed? Do the extra people—the ones whose roles have
    withered—starve? Or get easy lives? Who decides? How?

    The same core questions, stated in a multitude of ways, have elicited only a
    small number of answers, because only a few are possible.

    What will people be when technology becomes much more advanced? With each
    passing year our abilities to act on our ideas are increased by technological
    progress. Ideas matter more and more. The ancient conversations about where
    human purpose is headed continue today, with rising implications.

    Suppose that machines eventually gain sufficient functionality that one will be
    able to say that a lot of people have become extraneous. This might take place
    in nursing, pharmaceuticals, transportation, manufacturing, or in any other
    imaginable field of employment.

    The right question to then ask isn’t really about what should be done with the
    people who used to perform the tasks now colonized by machines. By the time one
    gets to that question, a conceptual mistake has already been made.

    Instead, it has to be pointed out that outside of the spell of bad philosophy
    human obsolescence wouldn’t in fact happen. The data that drives “automation”
    has to ultimately come from people, in the form of “big data.” Automation can
    always be understood as elaborate puppetry.

    The most crucial quality of our response to very high-functioning machines,
    artificial intelligences and the like, is how we conceive of the things that
    the machines can’t do, and whether those tasks are considered real jobs for
    people or not. We used to imagine that elite engineers would be automation’s
    only puppeteers. It turns out instead that big data coming from vast numbers of
    people is needed to make machines appear to be “automated.” Do the puppeteers
    still get paid once the whole audience has joined their ranks?

## The Technology of Ambient Cheating

    Siren Servers do what comes naturally due to the very idea of computation.
    Computation is the demarcation of a little part of the universe, called a
    computer, which is engineered to be very well understood and controllable, so
    that it closely approximates a deterministic, non-entropic process. But in
    order for a computer to run, the surrounding parts of the universe must take on
    the waste heat, the randomness. You can create a local shield against entropy,
    but your neighbors will always pay for it.

    Note: A rare experimental machine called a “reversible” computer never forgets,
    so that any computation can be run backward as well as forward. Such devices
    run cool! This is an example of how thermodynamics and computation interact.
    Reversible computers don’t radiate as much heat; forgetting radiates
    randomness, which is the same thing as heating up the neighborhood.

## The Insanity of the Local/Global Flip

    A Siren Server can become so successful—sometimes in the blink of an eye—that
    it optimizes its environment—changes it—instead of changing in order to adapt
    to the environment. A successful Siren Server no longer acts only as a player
    within a larger system. Instead it becomes a central planner. This makes it
    stupid, like a central planner in a communist regime.

## The Conservation of Free Will

    A story must have actors, not automatons. Different people become more or less
    like automatons in our Sirenic era.

    Sirenic entrepreneurs intuitively cast free will—so long as it is their own—as
    an ever more magical, elite, and “meta” quality of personhood. The entrepreneur
    hopes to “dent the universe”* or achieve some other heroic, Nietzschean
    validation. Ordinary people, however, who will be attached to the nodes of the
    network created by the hero, will become more effectively mechanical.

    [...]

    We’re setting up barriers between cases where we choose to give over some
    judgment to cloud software, as if we were predictable machines, and those where
    we elevate our judgments to pious, absolute standards.

    Making choices of where to place the barrier between ego and algorithm is
    unavoidable in the age of cloud software. Drawing the line between what we
    forfeit to calculation and what we reserve for the heroics of free will is the
    story of our time.

## Rewarding and Punishing Network Effects

    To understand how Siren Servers work, it’s useful to divide network effects
    into those that are “rewarding” and those that are “punishing.” Siren Servers
    gain dominance through rewarding network effects, but keep dominance through
    punishing network effects.

## The Closing Act

    Competition becomes mostly about who can out-meta who, and only secondarily
    about specialization.

    [...]

    Individual Siren Servers can die and yet the Siren Server pattern perseveres,
    and it is that pattern that is the real problem. The systematic decoupling of
    risk from reward in the rising information economy is the problem, not any
    particular server.

## The limits of emergence as an explanation

    But the problem with freestanding concentrations of power is that you never
    know who will inherit them. If social networking has the power to synchronize
    great crowds to dethrone a pharaoh, why might it not also coordinate lynchings
    or pogroms?

    [...]

    The core ideal of the Internet is that one trusts people, and that given an
    opportunity, people will find their way to be reasonably decent. I happily
    restate my loyalty to that ideal. It’s all we have.

    But the demonstrated capability of Facebook to effortlessly engage in mass
    social engineering proves that the Internet as it exists today is not a
    purists’ emergent system, as is so often claimed, but largely a top-down,
    directed one.

    [...]

    We pretend that an emergent meta-human being is appearing in the computing
    clouds—an artificial intelligence—but actually it is humans, the operators of
    Siren Servers, pulling the levers.

    [...]

    The nuts and bolts of artificial-intelligence research can often be more
    usefully interpreted without the concept of AI at all. For example, in 2011,
    IBM scientists unveiled a “question answering” machine that is designed to play
    the TV quiz show Jeopardy. Suppose IBM had dispensed with the theatrics, and
    declared it had done Google one better and come up with a new phrase-based
    search engine. This framing of exactly the same technology would have gained
    IBM’s team as much (deserved) recognition as the claim of an artificial
    intelligence, but it would also have educated the public about how such a
    technology might actually be used most effectively.

    AI technologies typically operate on a variation of the process described
    earlier that accomplishes translations between languages. While innovation in
    algorithms is vital, it is just as vital to feed algorithms with “big data”
    gathered from ordinary people. The supposedly artificially intelligent result
    can be understood as a mash-up of what real people did before. People have
    answered a lot of questions before, and a multitude of these answers are
    gathered up by the algorithms and regurgitated by the program. This in no way
    denigrates it or proposes it isn’t useful. It is not, however, supernatural.
    The real people from whom the initial answers were gathered deserve to be paid
    for each new answer given by the machine.

    [...]

    What all this comes down to is that the very idea of artificial intelligence
    gives us the cover to avoid accountability by pretending that machines can take
    on more and more human responsibility. This holds for things that we don’t even
    think of as artificial intelligence, like the recommendations made by Netflix
    and Pandora. Seeing movies and listening to music suggested to us by algorithms
    is relatively harmless, I suppose. But I hope that once in a while the users of
    those services resist the recommendations; our exposure to art shouldn’t be
    hemmed in by an algorithm that we merely want to believe predicts our tastes
    accurately. These algorithms do not represent emotion or meaning, only
    statistics and correlations.

    What makes this doubly confounding is that while Silicon Valley might sell
    artificial intelligence to consumers, our industry certainly wouldn’t apply the
    same automated techniques to some of its own work. Choosing design features in
    a new smartphone, say, is considered too consequential a game. Engineers don’t
    seem quite ready to believe in their smart algorithms enough to put them up
    against Apple’s late chief executive, Steve Jobs, or some other person with a
    real design sensibility.

    But the rest of us, lulled by the concept of ever-more intelligent AIs, are
    expected to trust algorithms to assess our aesthetic choices, the progress of a
    student, the credit risk of a homeowner or an institution. In doing so, we only
    end up misreading the capability of our machines and distorting our own
    capabilities as human beings. We must instead take responsibility for every
    task undertaken by a machine and double-check every conclusion offered by an
    algorithm, just as we always look both ways when crossing an intersection, even
    though the signal has been given to walk.

    When we think of computers as inert, passive tools instead of people, we are
    rewarded with a clearer, less ideological view of what is going on—with the
    machines and with ourselves. So, why, aside from the theatrical appeal to
    consumers and reporters, must engineering results so often be presented in
    Frankensteinian light?

    The answer is simply that computer scientists are human, and are as terrified
    by the human condition as anyone else. We, the technical elite, seek some way
    of thinking that gives us an answer to death, for instance. This helps explain
    the allure of a place like the Singularity University. The influential Silicon
    Valley institution preaches a story that goes like this: One day in the
    not-so-distant future, the Internet will suddenly coalesce into a
    superintelligent AI, infinitely smarter than any of us individually and all of
    us combined; it will become alive in the blink of an eye, and take over the
    world before humans even realize what’s happening.

    Some think the newly sentient Internet would then choose to kill us; others
    think it would be generous and digitize us the way Google is digitizing old
    books, so that we can live forever as algorithms inside the global brain. Yes,
    this sounds like many different science fiction movies. Yes, it sounds nutty
    when stated so bluntly. But these are ideas with tremendous currency in Silicon
    Valley; these are guiding principles, not just amusements, for many of the most
    influential technologists.

    It should go without saying that we can’t count on the appearance of a
    soul-detecting sensor that will verify that a person’s consciousness has been
    virtualized and immortalized. There is certainly no such sensor with us today
    to confirm metaphysical ideas about people. All thoughts about consciousness,
    souls, and the like are bound up equally in faith, which suggests something
    remarkable: What we are seeing is a new religion, expressed through an
    engineering culture.