Talk:Metacompiler/Archive 2

Archive 1

Archive 2

Further discussion related to request for arbitration

When I first started this topic page, there was almost nothing on Wikipedia related to the topic or that tied together the different implementations of compiler-compilers and metacompilers, especially the Schorre lineage of compiler generating systems and the Forth metacompiler systems. What seems to be in dispute here, well, it seems there are many things in dispute, not least of which is my own expertise and background, which I am always willing to share with a willing ear. But there seems to be some contention over the terms I use and the importance I give to certain properties of metacompilers. I think there is even dispute over my use of the term Metacompiler, that such a thing doesn't exist. I created this article over 5 years ago, and I made sure I asked experts in the computing field, both in academia and industry, to vet what I had written and claimed. It's a specialized field, but one that is very important to software engineering, especially with regard to the areas of Domain Analysis and Domain-Specific Languages.

What doesn't help in spreading knowledge about metacompilers, or in gathering references for the topic, is that back in the 1970's one of the best families of metacompilers was terminated due to it being declared classified/secret for national security reasons (I have no idea why) and all further development on that line of metacompilers continued in secret, I believe at SDC (System Development Corporation), the engineering sibling of RAND, the research corporation. And thus we have no access to information about it.

Nevertheless, work on metacompilers has continued, including offshoots of that line of metacompilers that got banished to top-secret-land, and many software engineering projects all over the world use this technology to get a leg up when creating software for their particular areas and specialized needs.

For example, here is a abstract from the 1997 Proceedings of the Ninth IEEE International Conference onTools with Artificial Intelligence:

http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=632252

This paper describes both the Eu-PAGE (Eurotra PArser GEnerator) meta-compiler for the Eurotra formalism, which is a tool that automatically generates parallel natural language parsers, and the Dialogos parser, which is a parallel parser for the Greek language generated by Eu-PAGE.

The metacompiler they use plays a key role in their natural language research. It's a tool they use to automatically generate parallel natural language parsers. So you see, metacompilers are a known technology, not as widely known and appreciated as spreadsheet programs and word processors, but nonetheless a valued and important tool.

As you can see from the above example, sometimes the term is spelled "metacompiler" and some users prefer to hyphenate it as "meta-compiler". Either way is accepted, just as Meta II and TreeMeta are often hyphenated too, as Meta-II and Tree-Meta.

With regard to the Metacompiler page itself, I have tried to keep this topic simple and direct, based on my knowledge of Computer Science. And whenever this page is modified, I check with the experts I know and have them re-vet what is expressed here to make sure it is relevant and up-to-date and not erroneous.

Damon Simms (talk) 16:31, 9 October 2014 (UTC)

What is in my proposed change that is counter to the original and what still needs to be addressed

Here again Damon Simms posted the following link:

http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=632252

Which describe metacompiler that do not compile them selves. Again I am misconstruing his references that refute his claims that a metacompiler compiles it's self.

1. Not all metacompilers are self compiling is my main change to metacompilers

One of Damon Simms references was to a graphics metacompiler written in PL/1. And there are other special purpose metacompilers that do not compile them selves. There have been several other metacompilers that do not compile them selves.

1. There are other uses for the metacompiler technology.

The current topic is also talking about uses of metacompiler technology that would not be metacompilers. I think the use of derived meta analysis systems is important. They are included in the topic. But are not metacompilers by Damon Simms own definition as they do not compile them selves. We should make a section for metacompiler applications or uses. Meta analysis is a use presented in the topiic. A metalanguage as used here is a transformation description language.

The ACM CWIC document does not go very deep into the details of the languages. CWIC was classified top secrete as sensitive technology. My personal thought is that the government was interested in keeping the number of programming languages under control. The number of new languages popping up was a concern to some at the time. The government was pushing for a single standardized language to be used by the military. An edict was made that COBOL was to be used for all government accounting. It is hard to say for sure why CWIC was classified as top secret sensitive technology.

CWIC was major leap in technology in the Schorre line of metacompilers. But really only the syntax was similar. The CWIC syntax language of CWIC was more like TREEMETA producing a tree structure representation of the source and having separate unparse rules generating code. CWIC extend the unparse rules of TREEMETA using LISP 2 (an ALGOL like syntax implemention of LISP) for analyzing and generating code. Keeping the unparse expression syntax and adding variables that take on elements of an unparsed tree. This can be seen in the available CWIC document in the ACM online archives.

Damon Simms climes to wont all of same things I wont. This disagreement is over the trumped up importance of self compiling.

Compiling it's self is only important in the ease in which the compiler can be changed. That is a feature of taking a metalanguage as input. The metalanguage input is the power of the meta compilation process. Weather the metalanguage is the metacompiler's own definition or of another compiler language. It is ease of modification and maintainability that is astounding. Maybe this is only true of the Schorre line of metacompiler. I can not answer that question. The Schorre documents all explain the language as being BNF like. That is vary misleading. Those are old documents. If you study the parser terminology and understand the Schorre parser language it is apparent it is an analytically grammar. In all of the Schorre compilers and derived works there is a driving rule. The TOP rule that defines the program to be parsed. It may be a simple rule defining a program to be a sequence of declarations or statements. Or just a sequance of declarations.

program = $declarations;

It is how the meta compilers of Schorre origin work. They are a TOP down parser. I think it is wrong to class them a recursive decent parser. They are recursive in the same way C or C++ is recursive. Recursion is under control of the programmer. Recursion is natural in analyzing nested forms of language. Parenthesized arithmetic and or logical expression. Multileveled block structured languages. Later derived works are a combination of top down and bottom up.

--Steamerandy (talk) 20:05, 18 October 2014 (UTC)

Note on Forth metacompilation

Though I have no interest in getting into a tug-of-war between two people with diametrically opposed views, I will side with Damon Simms in that the Forth notion of a metacompiler is far from a fringe view in the Forth community and is in fact a generally accepted way of getting Forth up and running on a new computer or microcontroller. This is because classic Forth is largely written in itself (with the exception of a few primitives written in assembly language), leading to a bootstrapping problem of how to get Forth working in the first place. One can laboriously define high-level Forth words by either writing them all in assembly language or writing "precompiled" Forth directly into memory, but either process is extremely tedious. Once you have a working Forth system, it is not that much trouble to use it to compile Forth on another system (which often uses a different assembly language); the Forth community has referred to this process as metacompilation at least since the 1980s (which is when I first started learning about Forth). One may reasonably dispute whether this use of the word "metacompiler" is valid; in most languages, the term would be "cross-compiler". Nevertheless, there is no question that (a) Forth is being used to compile Forth, and (b) this term has been used by the entire Forth community (not just a fringe) for a long time, and thus I think it merits inclusion in the article. One does not get to redefine terms just because one doesn't like common usage. I vehemently dislike the use of the term "hacker" to mean "one who uses his/her programming skills to break into unauthorized systems" instead of "an experienced, competent, and enthusiastic programmer", but the fact is that 99% of the world uses the word to mean the former and not the latter. Therefore, I don't get to edit the former definition out of Wikipedia. If no resolution to this dispute can be found, I recommend creating a disambiguation page so that each party involved can describe their notion of what a metacompiler (or "meta compiler" if you prefer) is. — Preceding unsigned comment added by Mvanier (talk • contribs) 23:50, 5 November 2014 (UTC)

Think you for you courteous input. You hit on the reason that forth should not be called a metacompiler.

One may reasonably dispute whether this use of the word "metacompiler" is valid(in reference to Forth above); in most languages, the term would be "cross-compiler"

Two other terms describe the Forth process "self-hosting compiler" and "extensible language".

There is good reason for the non-acceptance of the Forth definition of metacompiler. It describe a process used by almost all self-hosting languages. So if allowed:

C would be a metacompiler by the forth definition of metacompiler.

FORTRAN would be a metacompiler by the forth definition of metacompiler.

BASIC would be a metacompiler by the forth definition of metacompiler.

PASCAL would be a metacompiler by the forth definition of metacompiler.

C++ would be a metacompiler by the forth definition of metacompiler.

LISP would be a metacompiler by the forth definition of metacompiler.

And so on ...

LISP being an extensible language compiles it's self in almost the same identical way that Forth does.

One has to see the problem here. The Forth concept of metacompiler is incorrect and has never been accepted in Computer Science. The Forth concept of "metacompiler" fails to distinguish metacompile as different from established Computer Science terms describing self-hosting and cross-compilation compilers. In computer science, meta is a common prefix that means "about". As an example, metadata is data that describes other data (data about data). A metalanguage is a language used to describe other languages. A language about another language. The HTML META tag is used to describe the contents of a Web page. "About" is the common Computer Science usage of the prefix meta found on Wiki meta and various web dictionary sites. Damon Simms says that the Forth site definition applies:

Meta: (greek) A prefix, meaning "one level of description higher" to fit meta as in matacompiler. Here on Wiki we have the generalized equivalent definition:

on a higher level of abstraction:

In linguistics, a grammar is considered as being expressed in a metalanguage, language that operates on a higher level in order to describe properties of the plain language (and not itself).

With that explanation applied to metacompiler (operating at a higher level on a lower level compiler) we have:

When a compiler is considered as being expressed in a metacompiler, compiler that operates on a higher level in order to describe properties(operations) of the compiler (and not itself).

The forth "metacompiler" does not operate on a higher level, but at the lowest level that can support the defining of new Forth words. Using that minimal subset it extends it's self into a full Forth implementation. I think this is close to metacompilation.

This topic falls under the Computer Science umbrella and references to recognized Computer Science references trump over personal sites or opinions. From:Sci-Tech Dictionary McGraw-Hill Dictionary of Scientific and Technical Terms, 6th edition,Copyright © 2003, published by The McGraw-Hill Companies, Inc.

A Metacompiler is a compiler that is used chiefly to construct compilers for other programming languages.

I have other Technical Terms references that say basically the same thing but are pre 1990 pub dates. And the McGraw-Hill (2006)reference above can be found on the web. The Book is available from Amazon.

To sum this up than:

A Metacompiler is a compiler that is used chiefly to construct compilers for other programming languages.

chiefly meaning: Most importantly. More than for any other reason. Above all.

It should also be noted that several metacompilers have been implemented in languages other than the metalanguage they compile. These are not self-hosted metacompilers. They do not fit the narrow Damon Simms definition:

The feature that sets a metacompiler apart from a standard compiler-compiler is that a metacompiler is written in its own language and translates itself.

addendum

With the history now in place there are 4 documented metacompiler that denounce the idea that a metacompiles compiles it's self. These were written in assembly, Lisp or PL/1.

--Steamerandy (talk) 17:14, 9 November 2014 (UTC)

Most metacompilers do compile themselves. But the fact that some do not invalidates the claim:

The feature that sets a metacompiler apart from a standard compiler-compiler is that a metacompiler is written in its own language and translates itself.

--Steamerandy (talk) 13:51, 13 November 2014 (UTC)

Can we remove "and usually mind-bending"

Can we remove "(and usually mind-bending)"? I don't see how this is mind-bending. It is just iterative software development.

Response:

From my own experience and from interviewing many others in computer and software engineering, the process of bootstrapping is the part that is mind-bending. Unlike other iterative software development which just builds successive improvements and extensions on an existing base of software, in bootstrapping you start with nothing and have the software generate itself. This is not unlike God creating God in the beginning, before there was God. Usually it takes the typical programmer a day or two to wrap their minds around the process. Unfortunately the English language has a paucity of terms to identify and label states of awareness and understanding. "Gobsmacked" is applicable in this case, but is even more informal than "mind-bending". For example, how do you load a Pascal compiler written in Pascal on a new computer that has never had a Pascal compiler on it? You can't compile the Pascal program (the compiler) on that computer until you have a Pascal compiler running on that computer that will compile Pascal. The solution is either a cross-compiler or bootstrapping. The cross-compiler, which would compile Pascal on a different computer, would have to be programmed to generate assembly or machine language for the new computer, and then you would port the machine code to the new computer and run it to then have a running Pascal compiler on the new machine. Or you can bootstrap the Pascal compiler, written in Pascal, onto the machine that doesn't yet compile Pascal, without the need for a different computer running a Pascal compiler altered to target the new machine. I have done both methods. One is straightforward and tedious; the other is quicker, more elegant, and mind-bending. 75.62.129.34 (talk) 16:40, 22 December 2013 (UTC)

The point is that generating a metacompiler by bootstrapping is not "business-as-usual" programming. Another example of mind-bending software development is the macro-processing language GPM, the General Purpose Macrogenerator. It seems straightforward, but grad students have been known to fall into GPM and lose track of time and reality. 75.62.129.34 (talk) 16:48, 22 December 2013 (UTC)

Response:

Ok so now I see the problem. "in bootstrapping you start with nothing and have the software generate itself." That is not the case! You are not starting with nothing. Well unless you have no brain. The code comes from your brain. It is in your brain. You only have to put it into the computer. Every time someone has ask me how I write a program from scratch. It's an obvious answer. Start with an empty file. Debug it til it works. --Steamerandy (talk) 06:35, 30 September 2014 (UTC)--Steamerandy (talk) 09:36, 14 November 2014 (UTC)

Sorry, you're wrong. I don't think you know what you're talking about. Bootstrapping is a long-recognized concept and process in computer science. -- Damon Simms (talk) 17:39, 6 October 2014 (UTC)

See: Bootstrapping and Bootstrapping (compilers)

See also: "Bootstrapping a simple compiler from nothing" by Edmund Grimley Evans, 2001

Damon Simms (talk) 18:27, 6 October 2014 (UTC)

I have been programming sense 1965. Written 6 commercial PASCAL compiler, An ADA compiler, A COBOL cross compiler and SLIC a meta compiler derived from CWIC. And many cross assemblers. In the late 80's I was head of language development at Kontron in Anaheim Calif. Part of a division manufacturing In Circuit Emulators and development systems. My comment was specifically addressing ""mind-bending"". And yes it would be mind-bending if you compiled an empty source file and got a compiler out of it. If I am wrong. Show me a compiler you have created by giving a compiler and empty file. Oh. wait it's a non-existent compiler you must give the empty file.

Anyway. That is never the case. You start with a specification and program it in an existing language. It is not mind boggling. — Preceding unsigned comment added by Steamerandy (talk • contribs) 20:19, 6 October 2014 (UTC) On your home page "Bootstrapping a simple compiler from nothing" is a perfact example. You started with a concept and kept building on it. Were is there proof of META II being compiled in it's self. In the META II documents it is explained it was compiled in META I. A hand compiled predecessor of META II. It's likely that most META compilers were not compiled in themselves, but in a previous version not the final version. There is a lot of pit falls in bootstrapping. The most problemistic mistake is overwriting the working version with a broken one. Still not mind boggling. --Steamerandy (talk) 20:57, 6 October 2014 (UTC)

You say you've been programming for almost 50 years. I get it. You've got a big swinging dick. So what. I've been programming for over 42 years, have a Masters degree in Computer Science, and am qualified to teach classes on this stuff. I've probably worked on as many diverse projects as you have, in as many or more different computer languages. But I'm not interested in comparing dick sizes. And I'm not interested in pissing contests.

I'm tired of ignoramuses coming on here who know a little somethin about somethin and think they know everything. Just because they weren't exposed to an idea or a concept or a process, they think it doesn't exist and everybody else is wrong. When in fact they are just plain ignorant.

I call bullshit. I've been willing to give you the benefit of the doubt, but I looked at your cc compiler compiler example further down on this page. It looks totally bogus, like you really don't know what you're talking about. It looks ok until you look a little closer and realize it's gibberish. Like you almost understand the concept, and can cut and paste sections out of old documents and programs, but there's something not quite right. That's true of most of your examples here -- on close examination they are wrong, in error, badly flawed. If you can make a working program out of your cc productions example, I will be surprised.

When you make a working version of cc, why don't you post it on the Internet and come back here and post a link to cc so we can all take a look at it, and maybe learn from it.

Until then, you show up here, ridicule my statements and my writing, and keep lording over all of us that you are some sort of master genius who knows better than the rest of us. I bootstrapped my own version of Schorre's metacompiler over 30 years ago, I improved and enhanced it over the years, and I have used it for generating other software tools, including code analysis and generation. I still use, maintain, and port that system. I maintain contacts in academia and industry who are involved in using this metacompiler technology for real-world applications. And I'm constantly learning, trying to keep up with technology and developments in software engineering.

But you see fit to come here and piss on me and piss on my expertise. Who the hell do you think you are?

Put up or shut up. Let's see your working cc system, and then I'll apologize for being a rude old person. I'm working on my own new version of the metacompiler, bare minimum, written in itself, bootstrapping itself to run on it's own minimal virtual machine. I'm doing this as proof-of-concept and also this version should port very quickly to any platform. Any platform. Also, awhile back, after re-reading Schorre's paper for the n-th time, I realized that his version of the metacompiler was originally designed to parse and compile a subset of Algol, so its virtual runtime machine is quaintly skewed, not pure. My new version is pure metacompiler, written in itself, bootstrapping itself, only designed to create bigger, better, badder versions of itself. It's been a backburnered project (some more urgent important projects need my attention first), but when it's done I will post web links to it here, hopefully there will be tutorial stuff too. Maybe finally you can understand bootstrapping, metacompilers, and other concepts discussed here which you insist on denigrating and ridiculing. -- Damon Simms (talk) 13:35, 7 October 2014 (UTC)

And let me just add, you aren't reading very closely here... that See-also example I included above, ("Bootstrapping a simple compiler from nothing" by Edmund Grimley Evans) is not my work. I just included it to show that other people have the same view of this as I do. From the changing perspective of your responses here, it looks to me like you are altering what you say to cover up your ignorance as further knowledge is provided to you.

Which makes me suspect you are just here for blowhard ego strutting and/or to argue for the sake of arguing. Either way, stop it. -- Damon Simms (talk) 13:51, 7 October 2014 (UTC)

So you know a little about META II. Is that supposed to impress. I wrote SLIC over 42 years ago. It was written from scratch in MACRO-10 Assembly. It was a full implementation of CWIC the last known Schorre metacompilers developed at SDC. SLIC was not only a full implementation of CWIC it was extended having 3 addition features. Unlike CWIC the generator produced PSEUDO code. PSEUDO code instruction could be designed to best fit the language being compiled. They provided a level of abstraction separating a target processor instruction from the generators. Actual machine instruction were programed in the MACHHOP language. MACHOP defined the instruction format as binary fields. The field size was specified and took a general expression as the value. A pseudo assembly list could be produced in the listing file. The MACHOP fields could be combined and printed binary, octal, hex or decmal radix.

 H(3): xpr1;
  (3): xpr2;
  (2): xpr3
 H(...

Above the first field is H(3): xpr1; The H specifies hex format of printing a pseudo assembly. (3) is the number of bits in the field xpr1 is teh value to place in the 3 bit field. The next 3 bit field and 2 bit field not having a radix code would be combined with the first H(3) field and printed in HEX format. A linker was part of the package. The linker was linked with an object file formater. One could also create a relocatable target system relocatable file. MACHOPs were actually functions taking special parameters that specified acceptable operand type. If you were generating code for a 16 bit word machine or needed to align to a word boundary on an 8 bit machine. an MORG 16 would be used. MORG x would assign the output location to a x boundary. Sense PSEUDO code was like a high level machine instruction of a machine designed for the language There less work retargeting a different machine. META II is a toy metacompiler. It was meant as a prof of concept. A point from which more powerful metacompiler would be developed. META II is on about same level as yacc as far as capabilities. It has semantic production inbedded in the syntax rules. It had built in lexing rules for IDs and numbers. TREMETA was a step up. Instead of having semantics in the syntax it added tree building operators and called unparse rules from the syntax rules. I am no expert on META II or TREEMETA. Have their manuals for reference. But have never used either. The first time I learned of a metacompiler was CWIC at an ACM meeting. That was before the ACM paper came out in 1970. It was mentioned that Schorre was the author of META II.

I have most of the SLIC code. Not much good aas I do not have a DEC-System-10 to run it on. And really just as easy to code in IA-86 assembly. I am doing away with the klutzy ALGOL type syntax making it look more like C. >BEGIN .END are { } instead.

SLIC was written while I was at Cerritos Collage in Norwalk Calif. It is a junior collage and at that time they didn't have any computer science classes. If you wonted to learn you took Data Processing courses. The were part of the business education department. My first course in that department was Easycoder Assembler. Had already taken a FORTRAN course as part of the MATH department offerings. I was planning to transfer to UCI but took a job with Ramada Inns in Phoenix AZ. I was to be the resident DAC-10 guru. They were do a hotel reservation system. Several of the managers had came from TWA airline reservation system. The hotels were to get front desk systems that used a TI990 mini computer. It was decided to use the accounting cobol programs to write front desk applications for the TI990 mini computers. We used SLIC to write the COBOL cross compiler. The COBOL compiler ran at 2800 linex per min 300 lines per min greater then the DEC-10 COBOL compiler. SLIC was no toy metacompiler. The COBOL compiler was used by around 30 COBOL programmers. The only bug that turned up is the record contains bug discussed elsewhere here.

Oh by the way. I wasn't called Dr. Andy at my last 3 jobs for no reason. --Steamerandy (talk) 09:36, 14 November 2014 (UTC)

Counter argument

Damon Simms says: "in bootstrapping you start with nothing and have the software generate itself."

I say that is not the case! You are not starting with nothing. Well unless you have no brain. That is the magic of programming it is a creative science/art.

See Bootstrapping#Computing or more specific Bootstrapping#Software development that is not Damon Simms work either. I would not be a very good programmer if I stopped before getting things right. Damon I am not going away. You called me out. You seam to have an ego problem and just can not admit when you are wrong. Or are you a Forth freak trying to get forth recognized as a metacompiler? See Forth meta-compilation.

See Damon again does not know the computer science terminology. Or maybe he does and is just trying to misdirect us by linking to a restrictive definition that agrees with his position.

--Steamerandy (talk) 05:47, 19 October 2014 (UTC) --Steamerandy (talk) 20:51, 18 October 2014 (UTC)