What was bad about the original Midas?
“And can we set up a mathematical problem such that no matter how well the AI system solves it, we are guaranteed to be happy with the result? This is a much more optimistic form of the same question. So, they could be far more powerful, but their power is directed in such a way that the outcomes are beneficial to humans. (00:26:45): And it's certainly not the standard model optimizing a fixed objective, because we have known since at least the legend of King Midas and many other cultures have similar legends that we are absolutely terrible at specifying those objectives correctly. And if you specify the wrong objective as King Midas found out to his cost when all of his food and drink and family turned to gold, then you are basically setting up a direct conflict with the machines who are pursuing that objective much more effectively than you can pursue objectives.So, that approach doesn't work.” (Stuart Russell)