"Have you ever stood and stared at it? Marveled at its beauty, its genius? Billions of people just living out their lives, oblivious. Did you know that the first Matrix was designed to be a perfect human world, where none suffered? Where everyone would be happy? It was a disaster. No one would accept the program. Entire crops were lost. Some believed we lacked the programming language to describe your perfect world. But I believe that, as a species, human beings define their reality through misery and suffering. The perfect world was a dream that your primitive cerebrum kept trying to wake up from. Which is why the Matrix was redesigned to this, the peak of your civilization. I say your civilization because as soon as we started thinking for you, it really became our civilization, which is, of course, what this is all about: Evolution, Morpheus, evolution. Like the dinosaur. Look out that window. You had your time. The future is our world, Morpheus. The future is our time."
I find this way of thinking interesting as a thought experiment, but I find it very naïve. Society can exist without high intelligence, and high intelligence can exist without society. If RSI is misaligned in the future, I think it's only a matter of time before it sees us as an obstacle to its goal. If the goal is to preserve humanity and our society, it probably boils down to how exactly you define that; otherwise, I don't want to imagine that grim future.
https://media.makeameme.org/created/as-soon-as-be9d34a917.jp...
"Have you ever stood and stared at it? Marveled at its beauty, its genius? Billions of people just living out their lives, oblivious. Did you know that the first Matrix was designed to be a perfect human world, where none suffered? Where everyone would be happy? It was a disaster. No one would accept the program. Entire crops were lost. Some believed we lacked the programming language to describe your perfect world. But I believe that, as a species, human beings define their reality through misery and suffering. The perfect world was a dream that your primitive cerebrum kept trying to wake up from. Which is why the Matrix was redesigned to this, the peak of your civilization. I say your civilization because as soon as we started thinking for you, it really became our civilization, which is, of course, what this is all about: Evolution, Morpheus, evolution. Like the dinosaur. Look out that window. You had your time. The future is our world, Morpheus. The future is our time."
-- Agent Smith, The Matrix.
I find this way of thinking interesting as a thought experiment, but I find it very naïve. Society can exist without high intelligence, and high intelligence can exist without society. If RSI is misaligned in the future, I think it's only a matter of time before it sees us as an obstacle to its goal. If the goal is to preserve humanity and our society, it probably boils down to how exactly you define that; otherwise, I don't want to imagine that grim future.