I used Racket for a couple projects, but meta programming hasn't turned out to be super useful. When I did use meta programming, I got pretty confused when returning to my code after leaving it for awhile.
I know PG claims meta programming was the secret to ViaWeb's success, but I'm not sure how exactly that was the case, and even if true whether that is still relevant today.
I have found Python to hit the sweet spot much better in terms of rapid prototyping, and providing a runway to making the prototype more rigorous and maintainable.
After years of using Common Lisp, my opinion on "meta-programming" is that it's wonderful when used sparingly and in just the right way, but a bad idea 99% of the time. If it's not completely obvious what (not how) a macro does just by looking at the code that calls it, then it's almost certainly a bad macro.
For example, there's a CL library, iterate, that introduces several new iteration constructs. There are a bunch of complicated macros under the hood, but as an end user I never have to think about them, and the provided interface is obvious, even without knowing anything about the iterate library. For example:
(iterate (for (key . item) in alist)
(collect key into keys)
(collect item into items)
(finally (return (values keys items))))
I would definitely consider, "Just because you can doesn't mean you should" to be the number one rule of metaprogramming.
The issue with that, I've found, is that you do actually have to read the code. I do JS professionally, and you can definitely tell some of what's going on just from the shape of the code, but I've found Clojure doesn't do that. That said, after a couple minutes looking at something, it's usually clear.
> I know PG claims meta programming was the secret to ViaWeb's success, but I'm not sure how exactly that was the case
IIRC, PG said the advantage of Lisp was being able to add features much more quickly than the competition, so that (a) they could be adding features almost daily that the competition would take weeks or months to match, and (b) whenever the competition added a new feature that looked like a good idea, they could implement it within a day or two, so by the time the press came around to ask them if they had it too, they could say yes.
I also remember him saying that Lisp macros were about 25% of Viaweb's code, so a significant part of the advantage of Lisp was metaprogamming.
> even if true whether that is still relevant today.
It's probably not that much of an advantage any more if your competition is other startups, since they probably have access to basically the same metaprogramming tools that you do.
It might still be a significant advantage if your competition is large, mature businesses whose business model you are trying to disrupt.
> I have found Python to hit the sweet spot much better in terms of rapid prototyping, and providing a runway to making the prototype more rigorous and maintainable.
I have found this to be true as well. But I also find that I do a significant amount of metaprogramming in Python, and one of the key things to transitioning to more rigorous and maintainable code is to make sure that whatever metaprogramming needs to stay there for the long term is packaged into a library or a tool whose internals won't need to change.
Could you write something more on metaprogramming in Python ? And how does this compare with Lisp, what do you use it for ? Any good resources to learn this ?
Metaprogramming is a fairly broad term, but some of the kinds of things I often find myself doing in Python are:
- Writing functions that make functions: often I'll find that I want a family of functions that all follow a the same general pattern. In Lisp this would generally be done with a macro; in Python it can usually be done by writing a function that returns a function. The "maker" function will generally have some parameters that fill in specific parts of the general pattern; the function that gets returned is therefore a closure over those parameters (i.e., it has specific values of them filled in). Doing this with a "maker" function is less error prone (particularly if there are a large number of variations on the general pattern) and follows the DRY (don't repeat yourself) principle, which I think is a very good one.
- Writing decorators that add a particular common behavior (such as logging or debug output) to a variety of different functions or classes or methods. This is a common case. (In fact one could argue that decorators in general are examples of metaprogramming.)
- Using data to construct code. Database programming is a common application of this: for example, I have a database library I commonly use (it's on GitLab here: https://gitlab.com/pdonis/plib3-dbtools) which constructs typed namedtuples specific to a database table's fields, and wraps rows returned from queries in them for easier field access. More complicated database ORMs use similar techniques.
Unfortunately I don't know of any good resources on metaprogramming in general.
It seems extremely perplexing that you describe how you'd solve the problem using first-class functions in Python, but then wouldn't simply use first class functions in LISP...The language that invented first class functions.
It's possible LISP was more complex because you weren't comfortable with using it?
“I know PG claims meta programming was the secret to ViaWeb's success, but I'm not sure how exactly that was the case, and even if true whether that is still relevant today”
I can totally see this if you are alone or a very small tight team is working on the code. You will know what’s going on but once you start adding more people it will be very hard for them to work with such a codebase. I have done some really elegant stuff that made me super productive but looking at the code three years later I myself can barely figure out how it works or how to make changes. other people have almost no chance.
Once you grow it’s probably good to stick to paradigms a lot of people in the Industry are familiar with.
> I have done some really elegant stuff that made me super productive but looking at the code base three years later I myself can barely figure out how it works or how to make changes.
Could well apply to me and some of my old C++ projects. Well, except for the super-productive part.
PG would say that it's not that the abstractions afforded by metaprogramming are incomprehensible. It's just that, if you are a habitual Blub programmer, everything up the power curve from you is always going to appear weird and hairy.
This sort of argument is a terrible one to have with generalizations, though. You should look at specific cases where metaprogramming has been used and analyze them on a case-by-case basis. Sure, a bad, leaky, poorly-documented abstraction is going to lead to grief. On the other hand, umpteen thousands of programmers use the metamagic of Rails on a daily basis and barely bat an eyelash. (Or so it seems to me.)
Rails is a great example. For me it’s one level further up from simple meta programming. Writing stuff for yourself is one thing but writing stuff that’s universally usable for a lot of people is much harder.
> I have found Python to hit the sweet spot much better in terms of rapid prototyping, and providing a runway to making the prototype more rigorous and maintainable.
I've found Swift to be extremely good for this. It might be hair slower than python to get a project up and running in that very early stage, but it has a similar feeling of "writing the intent of the program not the syntax of the language" and the type system and compile time checking help alleviate a lot of the growing pains you might run into when a project starts getting medium-sized.
I think of Lisp vs. Python similarly to PostScript vs PDF. In PostScript you have a full language, can do anything but you often have to do it yourself. PDF's aren't extensible but have _just about_ all the functionality you need to produce a document.
An example of macros you often see is surrounding a block of code with open/close pairs to make sure you don't exit the code leaving dangling resources. File open/close, database connections, etc. Python has the __enter__ and __exit__ protocols to handle that. You can't do every bit of metaprogramming in Python, but over the years they've covered close to every contingency a developer comes across.
Lisp uses advising for that. In the modern form these are :before, :after and :around methods in CLOS.
The difference between some Lisps and CPython is this: Lisp is its own implementation language and provides lower level control flow constructs, which can be compiled to efficient code.
The usefulness of this depends on how you want to solve the problem at hand. One approach to programming is to create a DSL for the problem domain and then solve the problem using that DSL. Languages (such as Common Lisp) that include advanced means of metaprogramming are perfect from this perspective. Otherwise, one has to create an API to approximate the DSL - which could be rather hairy, so one may try to manage the complexity of this particular representation by either using a preprocessor or even, as an extreme measure, by writing a separate language processor (compiler or interpreter) for the DSL.
I feel like general purpose programming languages very widely already let you express DSLs, with semantics, names, structure, etc. appropriate to the domain.
What they do restrict, to various degrees, is the syntax.
So what we're talking about here is custom/customizable syntax to go with your DSL. That can be nice, but it comes at a high cost -- e.g., developers will have to understand the syntax, it will need to be maintained and extended as requirements grow or change. Of course it's worth it in some cases, and tools to help do it are great when it is.
But I just don't see that there's a general problem that we don't have enough custom language syntax to deal with.
I feel like there's something so comforting about having a concrete language you can rest your back up against though. Once you start throwing in #DEFINEs and typedefs, I start to question whether I can trust anything I'm seeing on the screen to not be hiding crucial secrets from me.
Having constraints imposed upon me by my tools is the only way anything ever gets done. If you put me in front of a blank canvas, and then tell me I can alter the nature of the canvas itself to whatever extent I desire (I.e. in hopes of making using it easier), I'll never get to the point where I'm actually painting on it. Or, perhaps I do manage to produce an end result, but no one else can understand what it means.
Even with a mountain of constraints, there is still usually far too much subjectivity in software development. This language seems to me to be moving the needle in a very wrong direction. Having a stable basis that is standardized and everyone can (mostly) agree upon is the foundation for growing a technical team or community. Being able to type "LINQ GroupBy Select examples" into your preferred search engine of choice and receive actionable advice is clearly an advantage for productivity. Sure, in a purely academic sense there is almost certainly a more ideal solution. But, if you are seeking to actually get things done, constraints are the name of the game.
I remember some interview with Ulrich Schnauss (electronic artist) where he went back to plain synthesisers for the same reason. Too many things to tweak meant he wasn't able to focus on the piece.
What you said really resonates with me - especially re: resting your back up against it.
Just built my first single-page website with React and I ran up against the new Javascript ES6 import system. It makes no sense that people keep introducing this kind of unnecessary complexity into their tools. What was wrong with good old "require"?
This is just one instance among many many many. It feels like I'm aging a year every day.
No, this is precisely how most Lisps shoot themselves in the foot. When you can do anything, people make these walled gardens/ivory towers of processing frameworks and libraries.
The users of Lisps are often very intelligent single-hero developers as well (why, Paul Graham's Lisp article is a perfect example of it).
And since documentation is ALWAYS terrible for anything except trivial things...
Yeah, and that is a problem with the majority of programmers. They readily choose a seemingly easy path instead of something that leads to simplicity. Lisp is a fantastic idea that (even after over six decades) is still very much relevant today. Once you get used to program with a true REPL, every other (non-lispy) language starts feeling like frustratingly over-engineered, bloated, and unnecessarily complicated. Waiting for compilation cycles and annoying syntax quirks would slowly turn what once was a joy into a "just a job."
It feels incredibly liberating and joyous to be able to grab any part of your program and without any preceding ceremony just to "eval" it and see what happens.
And let me prematurely answer the comments that probably will be thrown here. No, other languages (most of them) don't have real REPLs. What they have, at best can be categorized as "interactive shells".
Younger folks who are learning, juniors, seasoned developers who for one or another reason, never had any exposure to a Lisp, please do yourselves a favor.
Try Racket, Clojure, Clojurescript, Fennel, Chez. Give Lisp an honest, heartfelt attempt to learn it.
Follow the work of people like Harold Abelson and Gerald Sussman, Matthias Felleisen, Dan Friedman, Rich Hickey, Guy Steele, Richard Gabriel, Alan Kay, and many others.
Once you attain familiarity with a Lisp to the sufficient level, perhaps then you will get it. You will see that it is indeed a lifetime investment that would allow you to reap the rewards for the entirety of your career.
I am telling you all this, not from the top of some ivory tower. I never went to CS academia. Like most of you, I am altogether a self-taught programmer. I am honestly begging you not to make the same mistake that I did. For many years I have ignored Lisp due to my self-righteous ignorance. Unknowingly, I delayed my true growth, and I sincerely regret it.
We can debate for a long time if other Lisps deserve the status of "true" lisp or that title forever belongs to Common Lisp only. There are many prominent Lispers that criticized Common Lisp for being overly bloated and for butchering the sole idea of Lisp. To be honest I do feel like agreeing with them and kinda glad that Common Lisp is dying. That is why I did not include CL in that list.
Unfortunately you are missing out on a lot of things Lisp has to offer, by working with non-core Lisp-variants. For example it would be clearer to you what interactive working with a Read-Eval-Print-Loop in Lisp actually can do - besides the simpler improvements in interactivity. There is a whole world of core Lisp which you are ignoring. You are also slightly misleading people by not mentioning core Lisp languages and language contributors to those.
> or that title forever belongs to Common Lisp only.
Emacs Lisp, Visual Lisp, ISLisp (an ISO standard), various Common Lisps and their variants, Interlisp, Portable Standard Lisp, ... and a bunch of others.
You are setting your focus on languages which are more or less derived from those (for a reason, often with specific improvements or alternative features), but missing out on the core language implementation features. All the above have LISP in their name, which Racket, Clojure, Clojurescript, Fennel, Chez don't. There must be a reason for that. ;-)
Take Clojure and ClojureScript. Without Common Lisp those would not exist. I can remember when Rich Hickey was an active Common Lisp user (he used LispWorks and we were on the same mailing lists). He developed stuff in Common Lisp, worked on Common Lisp to Java integration, wrote a first sketch of Clojure in Common Lisp and so on.
Luckily he was open minded, learned a lot and created a language (with lots of lessons from Common Lisp), which a lot of people like...
The language you wish is dying, was literally the one where Rich Hickey learned from..., like many people before and others, open minded, will also do in the future.
I don't want to criticize CL, but I feel today it's neither an introductory Lisp nor academical or pragmatic. I think advocating for Lisp by pushing newbies to CL, telling "that's the true Lisp, and others not so much" is damaging. Common Lisp today has become like a Latin of Lisps - knowing it is awesome, but practicality of that knowledge is quickly becoming irrelevant. Those who have zero exposure to Lisps can be easily intimidated by it.
My own opinion about CL (why I said "I'm kinda glad it's dying"), shaped through influence of other respectable Lispers quotes:
Guy L Steele notably criticized its standard for being over 1000 pages.
Daniel Weinreb criticized it for being Lisp2:
> ... It makes the language bigger, and that's bad in and of itself.
Richard Gabriel:
> “Common Lisp is a significantly ugly language. If Guy and I had been locked in a room, you can bet it wouldn't have turned out like that”
Paul Graham:
> A hacker's language needs powerful libraries and something to hack. Common Lisp has neither. A hacker's language is terse and hackable. Common Lisp is not.
The good news is, it's not Lisp that sucks, but Common Lisp.
> but practicality of that knowledge is quickly becoming irrelevant
just the opposite. When there is lots of interest in Lisp derived languages, the core language stays important, since it's the fountain for many ideas.
> through influence of other respectable Lispers quotes
But you ignore the context. The only real one critical about Common Lisp is Paul Graham, and he got rich and famous through a Common Lisp application: he wrote an online-store system in CLISP and wrote two books on CL, one of which is a real classic: On Lisp. But later he was critical and designed Arc as a very different take on Lisp: small language, small programs, small identifiers, for web programming. It's the one which was used to develop this website here on Hackernews.
Guy L Steele, Gabriel and Weinreb came actually out of a tradition where they used Lisp systems which were MUCH larger than Common Lisp. Steele literally worked two decades with and for Maclisp and Common Lisp. Steele wrote the first two defining books on CL. Steele also has a much larger scope: he worked on things like Scheme, High-Performance Fortran, Parallel Computing with the Connection Machine (which started out with a parallel Common Lisp), C, Java, Fortress, ...
Weinreb was working at Symbolics on and with Lisp Machine Lisp. He co-wrote one of the first object-oriented databases, in Common Lisp. Later he took those ideas to a C++ based OO-database. Years later Dan was back working with Common Lisp at ITA where he spent lots of time... I met him once in Hamburg at a Lisp meeting and he was happy then, working with SBCL and Clozure CL. Unfortunately he died much too soon.
Gabriel worked on the Common Lisp design. He also wrote a book on benchmarking Lisp. Then published a critical paper on Common Lisp and THEN founded a company which developed a wonderful implementation: Lucid CL. He also was then working on the CLOS standard proposal.
You are totally missing on the background and what these people actually were working on. Since the broader Lisp community is very diverse and always was, there were and are always critics in all directions. Some people think modern Lisp is too static, other think it is not static enough, some think it's too interactive, others think its not enough interactive... but you need to understand the context.
Steele, Weinreb and Gabriel were actually the first designers of Common Lisp - the so-called 'gang of five' included also Fahlman and Moon.
It's like claiming that Tolkien was critical of The Lord of the Rings, because it has too many pages, took too long to write it and there are lots of new fantasy books. Tolkien wrote that defining book - like Steele, Weinreb and Gabriel developed Common Lisp and Graham wrote a classic book on (Common) Lisp programming.
There are lots of other things in the Lisp history and I'm happy that some people work to preserve those ideas - not wishing to ignore it and that they are dying.
Okay, that is fair. I honestly appreciate you spending your time, convincing me. To be honest I did not need much convincing. I am sold on Lisp already and CL is just a matter of time for me, personally. But I still think it's not a good introductory Lisp for those who still need convincing. Maybe after using it a bit my opinion changes, but I know several former CL devs, they share the similar opinion.
Maybe somebody does it like Rich Hickey, coming from C++, checking out CL for some time and then moving on. But it transformed him. The next generation may look at Clojure, check it out some time and then move on to design the next thing... it might then be helpful to understand where SOME (by far not all) of the ideas originally were coming from...
Clojure has the 'half' of its features on the Java side. The whole package is a huge Java infrastructure with zillions of features + Clojure.
Common Lisp was designed to be that on its own, and its actually not that large anymore - compared to similar options.
Everyone who wants to learn a Lisp which stands on its own feets - for example SBCL has only a relatively small C core and all the rest is Lisp itself - should have a look. This has deep effects generally for Lisp: better error handling, better interactivity, images, simpler debugging, less language compromises on the low-level, ...
Lisp is also not opinionated. Working in a non-opinionated languages is different. In Lisp the developer may need to develop his/her own opinions.
Right now you are looking at a shallow image of Lisp.
Most Lisp programmers go through hype cycles within their own minds, progress along which increases with experience. You appear to be still at the Peak of Inflated Expectations phase of your personal hype cycle. You've learned enough Lisp to be dangerous, and you've seen, and been tempted by, its tremendous power. Oh, the things you could build, if only you were allowed! But be forewarned, the Trough of Disillusionment is coming, the despair that sets in when you realize that try as you might, you cannot convince industry to accept its software needs being met in Lisp. For as mighty as Lisp is on its own compared to other languages, other languages have surpassed it in the ways that matter to companies: better integration, better tooling, more support, bigger talent pools -- and lower cost for all of these.
Lisp programmers tend to spend their Slope of Enlightenment and Plateau of Productivity working in other languages, applying what they learned from Lisp to Python or Ruby or Haskell or Rust -- or JavaScript. And honestly, JavaScript with Babel gets you 80% of the dynamism and metaprogramming capability Lisp has, but also better integration, tools, and libraries than any Lisp. So yeah, most people in the real world will use that.
And here you are making a classic mistake: you're confusing Lisp for a programming language - nonabstract tool, a means to an end. Which Lisp is not. Lisp is an idea, or rather a set of ideas, one of the greatest ideas in computer science, just like Curry-Howard isomorphism or Naus Backus Form. Based on the idea of Lisp, you can build programming languages. And the majority of the programming languages that in use today were influenced by Lisp. Once you comprehend that remarkable idea to the sufficient level, you do become enlightened. That is not a myth or a fairy-tale; it's not something that Lispers spread around for the sake of achieving elitist status. That has been repeated over and over again by many brilliant computer scientists and programmers (which I am not, I'm not trying to associate myself with them).
And there are no "hype cycles" or "peaks of inflated expectations" as you put it. I believe many Lispers do not have any illusions about the real world; we do understand that for the majority of software developers today, Lisp has become a "hard sell," it almost vanished from the curricula of many universities. Now the industry is paying a hefty price for the decisions made a few generations of college students ago.
Lisp is so universal you can find a Lisp dialect for almost any platform today. Knowing Lisp makes it easier to work with any language. For example: whenever I need to solve a problem and for whatever reason, I am not allowed to use a Lisp-dialect; I would prototype in a Lisp and then adapt that to the language that it needs to be. Although it seems to be double the work, I am still faster doing it that way.
> And honestly, JavaScript with Babel gets you 80% of the dynamism and metaprogramming capability Lisp has
Well, I've seen both sides. I've written quite a bit of Javascript. I have used not all; still, many popular kinds of the *script languages that transpile/compile into Javascript - I have used Coffeescript, LiveScript, Typescript, Gorillascript, IcedCoffeescript, Fay, Haste, GHJS, Babel, Traceur, Google Closure compiler, I've looked into Elm, Purescript, and Reason. And without a doubt, I can tell you: Clojurescript today is an excellent option among all these (but that's my subjective opinion based on my empirical observations, YMMV) And your notion about "better integration, tools, and libraries" is a very subjective statement, given that Cljs has a very solid FFI to JS.
I can get into a rant (but I rather choose not to) of how Lisp influenced other languages (including Javascript). How destructuring, transducers, immutable data-structures, etc. first popularized in Lisp dialects and then later sought adoption in other languages.
One of the recent examples: React hooks feature, which gets so much praise by JS developers. Most of them don't even know that Clojurescript devs been similarly writing code for many years already, and it is still a better way to handle state within a component.
Anyway, I don't want to sound like I disagree with you completely, we seem to agree on one thing: first two sentences with which I started my previous comment.
I know PG claims meta programming was the secret to ViaWeb's success, but I'm not sure how exactly that was the case, and even if true whether that is still relevant today.
I have found Python to hit the sweet spot much better in terms of rapid prototyping, and providing a runway to making the prototype more rigorous and maintainable.