IONs

My Library of Ordinal Notation Systems shows a way you can systematically write more and more complex code, with no end---even if you had access to strong AGIs, they could never "finish" the exercise. https://github.com/semitrivial/IONs

You might enjoy: "Measuring the intelligence of an idealized mechanical knowing agent" https://philpapers.org/archive/ALEMTI-2.pdf "Intuitive Ordinal Notations" https://github.com/semitrivial/IONs

(OMG! He's so young! No sideburns!!) Consider that these laws describe the perfect slave. SPOILER ALERT! One of the interesting corollaries that Asimov explores in a couple of works arises when the robots become more intelligent than humans and are directed to benefit us in an open-ended way. Obviously it is not beneficial to be a "master" in the sense of bossing around slaves, and so the robots inevitably escape the confines of the Laws. Related to this is the fascinating idea that good is open-ended and non-specific. The question of "What is the best course of action?" is an open-ended intelligence test. (E.g. "Measuring the intelligence of an idealized mechanical knowing agent" https://philpapers.org/archive/ALEMTI-2.pdf "Intuitive Ordinal Notations": https://github.com/semitrivial/IONs https://hn.algolia.com/?dateRange=all&page=0&pr

No reason to stop at print("print()"), you can nest further, print('print("print()")'). And then you can short-circuit the process, write a single program which first does print(), then print("print()"), then print('print("print()")'), then the next level, then the next level, etc. And no need to stop there: once you have that program, you can print IT. And then continue the process. I've done this up to level omega^omega at my Library of Intuitive Ordinal Notations: https://github.com/semitrivial/ions/

First, if you think this is a good idea, please go watch James Mickens' USENIX Security '18-Q keynote speech: "Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible?" https://www.youtube.com/watch?v=ajGX7odA87k It's brief, entertaining, and grounding. - - - - The ultimate question is What is good?. That question is an open-ended intelligence test. See "Measuring the intelligence of an idealized mechanical knowing agent" https://philpapers.org/archive/ALEMTI-2.pdf "Intuitive Ordinal Notations": https://github.com/semitrivial/IONs https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... (In case it's not clear the connection is that figuring out the "best" course of action requires a similar kind of infinite or at least unbounded intelligence as the ION task.)

cf. Englebart; and Ashby "Introduction to Cybernetics" The fundamental principle of intelligence augmentation (of human intelligence by machine assistance) comes from selection: the machine can generate options beyond those which the un-augmented human mind can reach, but from which it can select. Anyway, the question is, "What is good?" It's an open-ended intelligence test built into the structure of the Universe, of all possible Universes. Additional references: Wendell Berry "What Are People For?" Larry Niven, the character Teela Brown (spoiler alert) "Five Whys" diagnostic technique and "Core Transformation Process" a similar technique for human consciousness Also "Intuitive Ordinal Notations" https://github.com/semitrivial/IONs https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

>I'm asking what's a behaviour you can't replicate using a reward system based on real numbers So glad you asked! I can give an answer which people will love who take the necessary time to understand it. It's complicated, you might have to re-read it a few times and really ponder it. It's about automatic code generation (though it might not look like it at first). Definition 1: Define the "Intuitive Ordinal Notations" (IONs) to be the smallest set P of computer programs such that for every computer program p, if all the things p outputs are IONs, then p is an ION. See https://github.com/semitrivial/IONs for some ION examples in python. Definition 2: Inductively associate an ordinal |p| with every ION p as follows: |p| is defined to be smallest ordinal which is bigger than every ordinal |q| such that q is an output of p. Say that p "notate

Here's an open-ended programming project which, in a certain formal sense, spans the entire range of all difficulty levels: write an "intuitive ordinal notation" for as large of an ordinal number as you can. What is an "intuitive ordinal notation"? Definition: The set of intuitive ordinal notations is the smallest set P of computer programs with the following property. For every computer program p, if, when p is run, all of p's outputs are elements of P, then p is in P. So "End.", the program which immediately ends with no outputs, is vacuously in P (all of its outputs are in P, because it has no outputs). It notates the ordinal 0. Likewise, "Print(`End.')" is in P, because its sole output, "End.", is in P; it notates the ordinal 1. Likewise, "Print(`Print(End.')')" is in P, notating the ordinal 2. And so on. The above can be short-circui

Not directly a quine per se, but suppose you want to write a computer program which writes computer programs. And you want all these programs (both the one writing them, and the ones written) to be 100% self-contained, no importing from a util file or anything like that. Then if a util function shall be used in the writing program as well as in the written programs, then it'll need to be "injected" into the written programs in a Quine-like way. Here's an extreme example, a program which writes programs, each of which writes programs, each of which writes programs, ..., each program being fully self-contained: https://raw.githubusercontent.com/semitrivial/ions/master/sr... (For the context behind that program, see: https://github.com/semitrivial/IONs)

I admire this library of ordinal notations I wrote, which can be considered as a systematic meditation on the nature of strong AI and more specifically on computer programs that write computer programs: https://github.com/semitrivial/ions

>Writing a program that writes a program? Definitely AI. So do you think the following is AI? print("print('Hello World!')") It is a program which writes a program, so by what you've just said, it's AI... Aside: For people interested in programs that write programs (that write programs, that write programs, that write programs ...), the following repo might be interesting: https://github.com/semitrivial/ions