I would guess it is possible to go through this list if you make it your full time job for a year or so. That might be worth it depending on your priorities. Regardless, it bears mentioning.
If you can't take a single week off of work during an entire year in order to advance your career, then you're satisfied with your current one.
There is no need to direct negativity toward others for making time to learn. Carmack is in this position today because of such work ethic, not in spite of it.
I assumed someone would retort with this angle, but I hoped people would interpret my comment charitably without me littering it with a dozen caveats.
Of course work isn't the only responsibility.
Of course some people have kids, pets, parents, medical conditions, etc.
It's one week away from work. It can still be done. You weren't going to be taking care of any other responsibilities during your normal 9-5 hours, so that should have no bearing on the week you take off; treat it as any other work week, except you're working for yourself.
"You weren't going to be taking care of any other responsibilities during your normal 9-5 hours, so that should have no bearing on the week you take off;"
Have you ever been married, had kids, etc? Pretty much any time I take off of work, it involves catching up on projects, chores, or parental responsibilities during the day. It seems the vast majority of people I've spoken to tell of similar situations.
Well, I have specifically avoided having kids at this stage in my life so that I may climb out of the poverty class first. I'm one abortion deep already. I was not as fortunate as 99% of the people in this little isolated corner of the internet.
I was homeless from 16 to my early 20s. I suffer from multiple traumas, both physical and mental. I have ADHD and bipolar disorder. I've had sciatica, degenerative disc disease, gout and other issues since my teens. Most days I can barely walk. When I do have kids, I'm not going to get to be the dad I want to be, running around and playing with them. My parents are abusive drug addicts who were not present in my life, so I was raised by other extremely abusive people instead, who kicked me out into the streets for being atheist. So malnourished as a child that my friends would make fun of being able to see the bones in my face and body. I spent years living homeless on scraps of food.
I have to advance my career. Every single thing is stacked against me, trying to make me fail. I don't get the luxury of deciding if I want to take a week off to do so. It's a matter of survival, and I have very purposefully structured my life in such a way that I have a shot in hell at one day raising a child above the poverty line.
Believe me, this is not a matter of me not empathizing with others or understanding the complexities of life. It's a matter of prioritization and commitment. If I can make time to learn, anyone can.
And on top of all of that, I do in fact have to provide for my sister and her four young, expensive children, because she doesn't have anyone either. I don't get much free time, man.
I'm sorry you had to go through all of this. I think your bias is your responsibility is 100% financial so is aligned with always spending more time for your career. For most people financial/career responsibility is only a part, ignoring the rest is as impossible as you putting your career aside.
That's a good point. I do experience the boundaries, I spent the majority of the last 8 years alone and was solely focused on work and hobbies. It was a very productive time for me. Now that I'm in a longer-term relationship again, my productivity has seen a major reduction. It's exacerbated by the fact that I am extremely productivity-focused, but my significant other wants to relax in her free time.
I'm still learning where I should set boundaries on both sides of the field, and this will probably be something I continue to struggle with for some time.
> It's one week away from work. It can still be done. You weren't going to be taking care of any other responsibilities during your normal 9-5 hours
I think you didn't read the parents carefully. The comment being replied to is not about 9-5 hours, it's about "locking yourself into a hotel for week", and if you have kids, this is not exactly a healthy thing to do in the face of other responsibilities. That's the point being made here, and you are changing the goal posts.
I'm not moving goalposts. The commenter tried to assert that only someone without "any other responsibilities" gets to spend a week learning something. It doesn't have to be in a hotel room. It doesn't have to be a week. It can be two, three, four weeks. Those are all implementation details.
The negative sentiment itself is what I addressed. Everyone fortunate enough to have internet access, a computer and a functional brain/body can make time, if they really want to, to learn new things that benefit their long term goals.
> The commenter tried to assert that only someone without "any other responsibilities" gets to spend a week learning something.
That's what I mean -- you didn't read the parents carefully. The commenter (me) made that reply to this:
> Carmack did it an a week by locking himself in an hotel room
Everyone can and should spend time learning, but the little point being made was that locking yourself in a hotel room is not something most people can do due to responsibilities. Somehow you changed this into your more general statement "spending a week learning something."
Spoken like someone who casts judgement and makes assumptions before considering that different people have different priorities.
It's downright silly to claim that I have no life responsibilities because I prioritize setting aside time for self-growth and encourage others to do the same.
Open your mind a little bit and ponder why you feel compelled to have such a negative and dismissive attitude about this.
I don't find that pretentious at all. What I'm conveying is that we have the Things We Think We Want and the Things We Actually Want. The former live in our minds, the latter are actually realized through our everyday actions and choices.
> I just noticed your username, “soul of mischief“, and now it makes sense to me. Are you trolling?
It's against the rules of Hacker News to assume bad faith. I certainly am not arguing in bad faith, nor do I intend to make anyone feel bad for the sake of it.
I purposefully wrote an abrasive comment in order to stoke the minds of others and make them think inwardly. There was one time in my life where I also needed to hear those exact words. That no matter what adversity I face, I must make times for the things I really care about.
I must constantly re-evaluate what drives me and check to make sure I'm not letting life pass by without achieving at least some of my goals. I must not make excuses, and I must not make excuses for others. I must not be negative toward those who are in a better station of life.
As for my username, it is derived from a popular hip hop group, Souls of Mischief, who are part of the larger Oakland set, Hieroglyphics. https://www.youtube.com/watch?v=fXJc2NYwHjw
If one cannot disappear for a week to think or study he is putting himself at some serious life disadvantage. My 2c.
I do not mean telling family that I am disappearing for a week starting tomorrow, but that I want to disappear for a week in a couple of months and the family would cover responsibilities for me (I should be willing to reciprocate).
I agree. We travel for work and disappear for a week. I've disappeared for 4-5 days to party with friends and my wife and kids have been fine with it. Obviously the work travel I don't control much. The party part - I'd be getting some serious conversations if it was happening every month. However, if it was to improve my career prospects I don't think anyone will hit much resistance. I mean provided you use the time well and don't get into a space of watching youtube clips of Dune 2.
Realistically most people just don't have the resources for this sort of thing. It's not necessarily a matter of want but a limitation of needs. Assuming one has a family that could cover, they would still have a job with responsibilities. Not everyone can afford to risk termination or go without a paycheck for a week even if it's the most intelligent option.
Most people in the field have PTO. Some people take PTO and travel, others play video games, Carmac has locked himself in a hotel room to work on projects pretty regularly since he was younger.
Many people have a lot more flexibility than they let on. They just make excuses instead of acknowledging their own priorities and commitments or lack of actual interest in change and work.
So far, my 2 years of retirement are an experiment in discretionary time. I'm married but no children or pets in the house. So I have lots of leeway in what I do every day. Maybe I average 1-2 hours of responsible time. And there are constraints - health, energy, lack of ambition, family travel, Internet distractions etc.
But in general I can do what I want. This month, it's been relearning awk to do financial data analysis and reading the books of a British author, Diana Athill.
I'll say that my personal project satisfaction level is medium. Life satisfication has a major social component that provides meaning, though. Optimizing for both isn't necessarily easy even in retirement.
Got a source on that? I know that they were married for 25 years. That's a lot of time for people to change and for differences between their needs and wants to grow.
I can say that he is exactly where he wants to be professionally, and I don't think you're suggesting that taking a week out of a year to learn new things is too risky because your wife might leave you.
I've found that the many of the relationships I'm most proud to have in my life are with those who had a reputation for being difficult and/or insufferable.
Weird flex but ok. I think though that there is some merit in being able to communicate with a difficult or insufferable person but I find no merit can exist in choosing to take pride in regularly willingly doing so.
The value is that reputations are rarely indicative of what they appear from the surface; a lot of the time the reputation for being difficult ultimately derives from challenging polite bullshit, and in my experience _that_ correlates to deep competence often enough to be worth investigating.
Yeah, it took years for me to understand that the only people telling me things like "You always want to be right," or "Stop questioning everything," etc. as a child, were in fact themselves narcissists, abusers and control freaks who despise resistance.
I don't mean to make it sound so one-dimensional or axiomatic, but at least in my particular life, this turned out to be the case. I maintain an extremely close group of friends, the only people I find myself having problems with are typically those with some sort of personality disorder or malevolent intent. And boy, do those people have mean things to say about me. :)
If you're feeling unable to learn, have an honest conversation with your family about your career and your interests. They may be more willing to support you than you think. Just as you'd do the same for them so they could achieve something important to them.
Yes. His kid is an adult now and he is reportedly single now too so that makes it a lot easier to do this. The daily life of a family is a large time-consuming responsibility and many can’t lock themselves into a hotel for a week for that simple reason.
If he really did it in a week, why hasn't anything come out of his AGI startup yet, two years later? Maybe it wasn't as easy as he seemed to imply, despite his towering intellect.
Anyone can read an absurd amount of text in a week. To absorb and understand it all is a different matter entirely.
Seems more likely to me that Carmack simply convinced himself he'd understood it all. He does seem to think very highly of himself.
The guy did pioneer 3D gaming, improve linux networking in the process (to support his games), build vertically landing, reusable rockets years before SpaceX's and play a pivotal role in modern VR.
I would not bet against his understanding on technical topics where he's done a serious deep dive. Also it was more than a year, not just a week.
I would generally agree. Another point could be that even if you understand something, executing on something can still be hard, especially when trying to push beyond existing limits or implementing something complex.
I don't think that's true, thinking back to university, if I 'locked myself in a hotel room' and did nothing but study a given topic, I reckon you could get a decent grasp of more than one lecture course.
Of course not being able to ask someone any questions would be disadvantage (but is that even a 'rule' of this scenario? Surely at least the internet is allowed, i.e. a person to ask who's available to anyone), and it would have been a lot easier while at university than with X years' rust on the background maths etc. - but still, seven full days is a lot of time if you choose to completely dedicate it to some particular thing, we just generally don't do that, so it doesn't sound it.
Sometimes you have pretty much all the prerequisite knowledge but are still missing a crucial piece that would help you "piece it together" and make a significant advance.
Almost forgot about how Carmack is the one that will give the world AGI...Easy to take the eyes of the ball with all these OpenAI and Google distractions...
FWIW, Ilya's recommendation for Kolmogorov book in the Arc viewer is "PAGE 434 onward", which lands on the last chapter "Algorithmic Statistics" (p.425 in the book). Appendix 1 starts on p.455 ...
I will also. I heard a lecture on Kolmogorov complexity at an AI conference (perhaps AAAI IN Austin in 1984?) and it forever changed my view on complexity and patterns in data.
wget -A .pdf -r -H -D arxiv.org -nd https://punkx.org/jackdoe/30.html
A friend and me used pdfunite to join them in one pdf and then printed them in a local printing shop in good a4 paper https://www.printenbind.nl/ as a spiral book, the end result is very good (also there is space around the pages so that its easy to take notes).
Keep in mind there is no confirmation that this is the actual list.
xLSTM is really really hard to grasp. It's mind boggling how they came up with such a unique architecture. I read the paper multiple times and still not sure if I can say I understood it.
What people forget is without a proper, tangible goal, no matter how many core or interesting papers you read, you will still remember nothing. At most you will be able to make armchair ML researcher comments on an online forum.
How long does it take to learn the prerequisites to learn these if you don't already have a CS background?
Reading through the list, I can barely parse half the titles due to acronyms and terms I've never seen before. Would I be able to figure them out along the way? Or is there a body of knowledge this is assuming I should already have?
(This isn't a criticism of the list, or even the idea of presenting such a list without explaining who the target audience is and isn't. I'm genuinely wondering, if I just start at the top, is there any hope I would figure this out, or do I need to start somewhere else entirely?)
Depends on what you mean by learn... And what other relevant background one has outside of CS - programming/math/stats/systems/engineering. Assuming one has a few years of that, and one want to get to the level of being able to apply this to a problem that comes up in a realistic setting (making a product etc). Maybe extreme dedication over many months, or a few years at 20 hours a week? This is assuming that one either is very good at studying alone, or have access to people one can ask alone the way.
If one has no relevant background, then I would 5+ years of full time study.
I think I may just need to accept that the work it would take to get to this level would mean a whole life change.
It's been 20+ years since I took calculus in high school, and I haven't taken a math or science class since. I reckon that puts me at least five years behind where I'd need to be to start this course and the listed ones.
Thanks for the recommendation, and helping me assess whether it's the use I want to make of my time!
The papers have bibliographies with citations in text. The knowledge you need to read a given paper usually lies within 3 degrees of bibliographical separation.
You can also use reverse citation search on Google scholar to find newer works that cite a given work. This can uncover a review or textbook, which often makes the concept much simpler to digest. And it can be a more advanced approach which gives you an alternative view or lets you abandon the original paper altogether.
2. Back to tha basics
For math, it would help if you have college level calculus, differential equations, linear algebra, and statistics. If you don't have that, be ready to assign yourself math readings and homework. It would also really help if you had analysis, topology, abstract algebra, and category theory under the belt, just so you can see the mathematical forest for the trees... But that's not really so necessary
You can often find the author's preferred reference for a given work in the bibliography, and use that.
For computer science, it helps to be able to read algorithm pseudocodes, follow complexity and correctness proofs, and understand performance experiments and charts. Skimming Knuth's "Art of Computer Programming" should be very helpful.
You'll need some algorithmic/statistical learning theory as well, since authors often use these concepts to prove that a system can learn. Though TBH, I usually skip these proofs and focus on the performance details. I usually find a higher performance paper before I need to care about the learning proofs. Still it helps to expect them and have an idea of how they work, if only to know when you can safely ignore them.
3. Narrative Arcs
Many AI results build on keystone papers that define eras. We can build on these papers by understanding their philosophy and deeper implications, even if we don't really understand their math. There's a story that you will tell yourself, and it may lead to interesting research or business results.
For example, if you read "Attention is all you need", you set the stage for LLMs. then "LLMs are in context learners", followed by the "chain of thought", "reflexion", and "toolformer" paper will give you a great sense of how LLMs are emerging as more than just stochastic parrots, ready to be developed and deployed in a lot of ways that overcome their issues. But looking at how bad the numbers can be even with our best efforts, you also get the sense of how much work these systems will need to actually be good, and the sense that corner cutting will lead to a lot of pretty awful systems.
It's a fun time to be a computer scientist or software engineer! Lots of cool papers to read!
Wow! I'm glad there are people like you out there to be excited about it! I glazed over before I even got to the end of your comment, with the dazzled sort of awestruck glaze where I realize I will never do even a tiny fraction of the work required to know things about this topic you probably take totally for granted.
Thanks for taking the time to expose me to your enthusiasm!
The argument that this is 90% of what matters in ML seems a bit bold. AFAICT it is completely missing reinforcement learning, which has been source of some mindblowing results (from Deepmind) in the past decade. It is also missing other stuff that I as a layman found fascinating (graph neural networks, 1.5 bit networks, ..). Also, I am not sure how much the Kolmogorov complexity actually matters, it's more philosophical than practical. So beware, it is a very opinionated list.
> AFAICT it is completely missing reinforcement learning
There's no RL in there, nor diffusion. There are many other very interesting papers that I think have lots more to offer — but I just have a master's degree here. I am not a phd let alone a distinguished researcher.
>The argument that this is 90% of what matters in ML seems a bit bold. AFAICT it is completely missing reinforcement learning,
Wouldn't that account for the other 10%? To me it sounds like the quote is not too far off when you consider that he said "of what matters today". There are many other things missing from that list that might become more relevant tomorrow.
Disagree, I’ve already changed my title to “ML professional” on linked in. Aside from the odd hallucination, I’m indistinguishable from an expert to the average hiring team.
This has nothing on it about llms, where in context learning, retrieval augmention, tool using, and multi modality is really blazing ground. Ideally we'd have papers about these methods, because we need these tools to implement stuff, and they're changing how we do business.
Without in context learning, self critique and tool using, for example, it's easy to think that LLMs are stochastic parrots bound by their training distribution, instead of last mile learners that can adapt to out of distribution contexts. And business applications need RAG to overcome hallucinations. Without these concepts, I don't think people will really know 90% of what matters today, or be ready to take on what matters tomorrow.
I could be wrong here, with my focus on so many llm specific topics. Things like in context learning and RAG exist as engineering workarounds for LLM limitations. They are not definite paths to tomorrow's AI, which won't need our cajoling and hand holding. There's a possibility that we'll move past them, so more speculative and theoretical papers are better.
But I think there's also a possibility that the AI scaling is done for this AI bubble, and we're stuck with building larger scale systems out of imperfect buggy LLMs for the foreseeable future. In that case, maybe some next generation Zen of integrating LLMs into a working mind eventually emerges as a way forward.. so we should pay attention to engineering efforts around LLMs!
I created a Zotero group: https://www.zotero.org/groups/5541371/ilya_sutskever/library...
This should be useful if you use Zotero.
For pre-prints that have been published, they are replaced by the published version.
Except for a few blog posts, and the Stanford course, and a copyrighted book, you should be able to obtained the PDF through Zotero automatically.
Interesting facts: published date ranges from
1993 (the only item from the 90s, by his advisor)
to 2020, types varies from
1 course,
1 dissertation,
10 preprints, (1 is a preprint of a chapter of a book, some are presented somewhere),
11 conference papers,
1 book, to
3 blog posts.
You mean for those who will just add it into their bookmark pile. You could have started yesterday and built a NN from scratch if you were truly interested.
oh man my cognitive dissonance is crazy cause i am easily in the top 50% percentile for being able to use popular LLMs but that reading list is outside my comprehension abilities. I cannot synthesize a thing I use
The first link on the list is “the annotated transformer” which goes into a fair amount of detail on attention, it’s an annotated version of the paper that introduced scaled dot product attention
and so on.
I would guess it is possible to go through this list if you make it your full time job for a year or so. That might be worth it depending on your priorities. Regardless, it bears mentioning.