Designing Data-Intensive Applications

Liked the first edition, but there’s a lot of fluff. These days I have Claude remove the useless passages from stuff I want to read fast. Helps to make flash cards to drill for system design rounds.

Humble Bundle has spoiled me and my ebook library has grown by around a 100 books this year... Tech book recommendations: 'Secure by Design', 'Designing Data-Intensive Applications', 'Building Secure and Reliable Systems' and 'Fundamentals of Software Architecture'. For scifi: 'Murderbot Diaries' and 'The Expanse' - both are just great entertainment

Designing data intensive applications

I tend to skip language, tooling, or library specific book unless I'm working on a project or are curious about the subject (hoping to discover some valuable nuggets). So the book I tend to read cover to cover are more about methodologies, concept, philosophies, and best practices. I like DDIA especially as I was always more focused on the computing logic side of software, not much about infrastructure. It has good explanations about things that I have heard about, but never had the chance to explore more.

My personal favorites: - Patterns of Enterprise Application Architecture (Martin Fowler) - early-mid 2000s and it stays valuable to this day. It's nice to have a formalized view of concepts you know in practice. - Clean Architecture (Robert Martin) - Great application architecture concepts being formalized - Designing Data Intensive Applications (Martin Kleppmann) - Fantastic perspective on your application's data. This is probably the most recommended book I've seen on Hacker News. - SQL Performance Explained (Markus Winand) - Just a killer, concise book to make you truly understand basic DB performance, specifically with indexes. I've met so many developers (myself included before this book) who thought any index will work and then they'd just wing it. Your RDBMS has tools for finding the best optimizations in your queries and you should use them. Your indexes are also more picky than you may think, but they're also incredibly fast if you place them correctly. It's a lot easier to see once you understand.

The 3-body problem trilogy (Rememberance of the Earth's past). My first sci-fi novels ever - the Netflix show sold me and these novels compelled me on a whole new hard sci-fi rabbit hole. Presumed Innocent (preceeded by the Apple TV show of same name). Murder mystery. Loved it. Tie-in novels from videogame franchise: - Deus Ex: Black Light - Devil May Cry: before the nightmare I've also started Marvel Comics in anticipation of Fantastic Four and the whole Secret Wars storyline it will build up. Couldn't finish a single non-fiction this year, but enjoyable so far: - Zero to One - When I say no, I feel guilty - Designing data-intensive applications

When I first graduated I read a bunch of tech focused books: they’re all helpful but I think practice and learning from more senior engineers is the most effective road to mastery! You can probably get away with not reading any of these books if you have good coworkers :). That being said, these have been my favorites: - designing data intensive applications (a great way of understanding systems + the basics of SRE) - the senior engineer (I love the prototyping process he lays out) - the effective engineer (lots of good gems for approaching prioritization) - debugging (by David agans)- a great resource for a formalized debugging process if you don’t have one - on writing well (I’m halfway through this, but it has been indispensable for writing tickets + messages at work)

The main thing I learned from this book was about LSMs and it solidified my understanding of B trees. I think I'd mostly overrated tho.

Seconding. This book will get you hired.

Designing Data Intensive Applications is a great book. I read this years ago and regularly recommend it, especially when people are all over talking about some special queue or database technology. I find so many people don’t understand even how databases work and it’s crucial to application performance and structuring. Adding queues works to relieve server load but if you get swamped by messages you need to make an active decision on how to handle that - and maybe a queue isn’t the best idea if you can’t lose them.

The top book imho is The Pragmatic Programer. Here some other recomendations - Designing Data-Intensive Applications - Tidy First?: A Personal Exercise in Empirical Software Design - A Philosophy of Software Design - Refactoring UI - The Software Engineer's Guidebook - https://blog.rstankov.com/programing-books/

Designing Data Intensive Applications

There are a lot of bad books but there are many good ones too, like 'Designing Data-Intensive Applications' But when starting you're better with something like a youtube tutorial

Wow, this is interesting, very few books I've read more than once! Even fewer are non fiction, but they sure do explain a lot about me: - Complex PTSD: From Surviving to Thriving by Pete Walker helped me process a number of assaults that happened to me as a child - Designing Data Intensive Applications by Martin Kleppmann helped me understand intuitively how databases work and how to use eventual consistency to handle workloads - The Highly Sensitive Person: How To Thrive when the World Overwhelms You by Elaine Aron this helped me process my and my families neurodivergence. - Structure And Interpretation of Computer Programs by Ableson and Sussman. Great book, it's how I learned to love Lisp. I felt like I grew 10x in skill from completing all the homework. - Starting Strength: Basic Barbell Training by Mark Rippetoe. This is the book that got me into powerlifting as a regular part of my fitness. I used this to heal my back, neck, and wrist pain. Changed my life. - Color and Light: A Guide for the Realist Painter by James Gurney. The GOAT of plein air painting himself teaches so much about how light works on complex forms. I've dog-eared this book.…

Designing Data Intensive Applications is really good, because it gives you an good overview of data systems. I had the feeling I don't have to understand all the details yet I would learn a lot. Are there other books like this?

Thanks, I already know Python for basic scripts. This tutorial looks awesome to build simple web apps, thanks for sharing. I'm looking forward to studying "Designing Data-Intensive Applications", it truly looks like a gem

If you're starting from scratch and interested in python, walk through Miguel grinberg's mega tutorial from start to finish, implementing everything yourself along the way. It's very good and you'll have a thorough understanding of the basics at that point. I started working professionally building simple web apps immediately after working through that. When youre ready to go galaxy brained, "Designing Data-Intensive Applications" is the standard

The best two books I read in the last year-ish are "The Golden Notebook" (novel by Doris Lessing) and "The Soul of a New Machine" by Tracy Kidder. Best tech book has been "Designing Data-Intensive Applications".

Second on Martin Kleppman’s book - really good. I have it on my nightstand as type.

The books that taught me computer science - Operating Systems: Three Easy Pieces ( https://pages.cs.wisc.edu/~remzi/OSTEP/) - Designing Data Intensive Applications by Martin Kleppmann (https://dataintensive.net) - Crafting Interpreters by Robert Nystrom (http://craftinginterpreters.com)

Guide to Reliable Distributed Systems by Kenneth P. Birman: gave me the same effect that the people who read DDIA/Designing Data Intensive Applications claimed it gave them. I first tried reading DDIA before it since it was newer and it had more hype. But DDIA is so dense that most of the things went over my head. Then I came across Ken Birman's book and I found it easier to follow. Although it had a hundred pages more than DDIA, it doesnt cover as much ground such as it didnt mention object storages. The niche topics were also better in DDIA, such as error correction schemes was only mentioned once in Birman's book. DDIA seems to be more complete. So I'll re-read DDIA again next year. Harassment Architecture: saw it on Amazon and I could never understand the humor behind the reviews for the book. So I tried a few pages, and something about it was gripping. All I could say is that I've never read anything like it.

I like to read a wide variety of genre from fiction to human psychology. For the mind: Nonviolent Communication by Marshall Rosenberg - A great book on how to communcate more effectively with others. Thinking fast and slow by Daniel Kahneman -He is the only psychologist ever to recieve the Nobel Prize if I remember correctly. This book scientifically digests human biases and errors and how they influence our every day lives and decision making. Sometimes it a bit too technical for my taste and I like mostly the first part. (its not an easy read) I rarely had so many Aha-moments. A random walk down on wall street - For investing your income in stocks. What to do and what no to do. Simple, with facts. Sapiens by Harari -Interesting, I remember it was a great read and fun read. Connecting a lot of dots :) On programming: Designing Data-Intensive Applications by Martin Kleppmann - I am failry a junior web dev, so any material that is clear and explains well is a great source of information and inspiration. This is such a book.…

Reading DDIA right now and I second this - the best technical book I've read, still holds up well after five years.

Yes, it's very often recommended. It's surprisingly entertaining for a technical book.

One of the best books I’ve read in years. There’s very little fluff, which is exceedingly rare in a technical book.

I agree that this is a great book, but I think it fails the "impractical" test - DDIA is an immensely practical book that's basically required background reading if you work with distributed data systems.

I agree it's well written but for me that's way more on the side of required reading. I'm thinking about books you don't _need_ to read but that are just really neat or advanced discussions.

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems https://www.amazon.com/Designing-Data-Intensive-Applications... (https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321) I surprisingly really enjoyed it. Well written and it pulled back the veil on a lot of concepts that I thought were too complex for me to understand/enjoy.

That's a good read regardless of your level.

> What are you reading and why? Audiobooks are a special case for me. They help fill the time when I'm driving to work, mowing the lawn or doing dishes. If I consume my entertainment during these productive times, I find I'm less drawn to television and other mindless distractions. They also help me to be motivated to do tasks that I otherwise find dull. The better alternative is to find a friend to share these tasks with, but that is not always a practical option. While I do read for entertainment, I really enjoy non-fiction as long as it's light. If I want to explore a topic and truly learn something, I often need to find a text version. That said, I have found some great audiobooks that I have learnt from: -The Goal -Pragmatic Programmer -Designing Data-Intensive Applications -Thinking Fast and Slow -Ultralearning The best entertainment so far have been -Dune -The Expanse series -The Bobiverse series -The Expeditionary force series.…

- Low level: Code Complete 2 (yes it’s old but still very relevant) - Hight level: Designing data intensive applications

Thank you. I've just purchased it and am listening now.

I second Designing Data-Intensive Applications. Deep Learning with Python by François Chollet I think works as an audiobook as well. I am a big non-fiction audio book fan and so much depends on the voice actor. I bad read can ruin the best content while Robertson Dean made Alan Greenspan's The Age of Turbulence into an enthralling adventure story.

I'm currently listening to "Designing Data-Intensive Applications" [1] on my commute and it really does work well as an Audio Book (I can attest to the positive reviews). Highly recommended if you are dealing with any requirements in the space (scale, replication, consistency, SQL vs NoSQL, etc.) [1] https://www.audible.com/pd/Designing-Data-Intensive-Applicat... (https://www.audible.com/pd/Designing-Data-Intensive-Applications-Audiobook/B08VLGDK32)

I agree that the Pragmatic Programmer is well done in it's audio form, and I also agree that Grokking Algorithms is terrible. I am currently listening Designing Data Intensive Applications and it's phenomenally done - the author clearly worked with the narrator to adapt the content to audio format, and the narrator seems to have experience or familiarity with the subject because he pronounces the technical jargon very naturally. I hope to find other software related audiobooks as good as DDAI is.

This is great, thanks! +1 for both data intensive apps and Google SRE books. The other two I haven’t checked out yet

As luck would have it, my blog on Github pages is down. So here's the post describing the four in markdown. Tldr; Designing Data Intensive Applications, Effective Python, The Google SRE book, and High Performance Browser Networking. https://github.com/eatonphil/notes.eatonphil.com/blob/master... (https://github.com/eatonphil/notes.eatonphil.com/blob/master/posts/books-developers-should-read.html)

* Fooled By Randomness (NN Taleb): Taleb is a complicated personality, but this book gave me a heuristic for thinking about long-tails and uncertain events that I could never have derived myself from a probability textbook. * Designing Data Intensive Applications (M Kleppmann): Provided a first-principles approach for thinking about the design of modern large-scale data infrastructure. It's not just about assembling different technologies -- there are principles behind how data moves and transforms that transcend current technology, and DDIA is an articulation of those principles. After reading this, I began to notice general patterns in data infrastructure, which helped me quickly grasp how new technologies worked. (most are variations on the same principles) * Introduction to Statistical Learning (James et al) and Applied Predictive Modeling (Kuhn et al). These two books gave me a grand sweep of predictive modeling methods pre-deep learning, methods which continue to be useful and applicable to a wider variety of problem contexts than AI/Deep Learning. (neural networks aren't appropriate for huge classes of problems)…

I'm a lot into how to streamline data architectures, especially around ML solutions, so some titles related to that. I'm already reading Kleppman's book right now. Tons of very useful knowledge, although quite detailed, and a lot of details around distributed computing, consensus algorithms etc (Part II), which I'm not sure I will need and which make the book rather long. Still, surely worth pressing through. Some titles I would like to dive into in the summer: - "The Art of Immutable Architecture" by Michael Perry. Possibly also: - "Machine Learning Design Patterns" by by Valliappa Lakshmanan et al. - "Building Secure and Reliable Systems" by by Heather Adkins et al. And perhaps something lighter, more inspirational for the day to day work: - "Coders at work" by Peter Seibel

Kleppmann's book is great, it will be well worth your time. I read it cover to cover. He takes care to be consistent and explicit regarding terminology, so it has helped me put a lot of bits of "knowledge" (and confusion) about distributed systems into perspective.

I couldn't agree more about curiosity mode (I'm going to use that phrase liberally). Despite reading papers for many years, I rarely go in cold. I browse blog posts and twitter, ask myself a series of questions, then try to find/read papers to answer them. Of course this only leads to more questions, and so the journey continues. I also agree with the recommendations for "Designing Data-Intensive Applications" and "Database Internals". Though, having read the latter for a book club at $employer, I felt it served better as a sort of "index for the space" for people who already had some DB experience, rather a true introduction.

Also recommend

This is a very good list. Adrian's blog is, particularly, a treasure. I wrote this post a while ago, aiming to answer a similar question about finding paper: http://brooker.co.za/blog/2020/05/25/reading.html One key point there is that there are, in my mind, multiple 'modes' of reading, and I like to use different approaches to finding material for different modes. Those blogs you list are great for curiosity mode. Another great resource there is Twitter: following distributed systems practitioners and researchers, and seeing what they tweet about. When I read a (recent) paper I really like, I often see if the authors are active on Twitter and follow them there if they are.…

This is a great book, indeed.

I thoroughly enjoyed "Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems" from Martin Kleppmann. It's a great book that goes into pretty much all of the commonly used strategies to scaling data-intensive applications. It's not incredibly deep on any of them but it will allow you to get a great overview of the entire space. For each component, there's usually references to places where you can read and study more about them.

I can vouch for designing data intensive applications. Having read it before, I highly recommend it for all experience levels.

Designing Data-Intensive Applications is probably one of the absolute best technical books I've read. I haven't done a second read through, but I can definitely see that being helpful.

I usually read good technical books twice. Recently did that for: Designing Data-Intensive Applications, DynamoDB Book.

After seeing your comment here yesterday, I started reading your book Initiative. I am in the middle of the first exercise and have some questions. Many of the examples in your book show people connecting these separate ideas that are reasonably understandable and applicable to the general population--the girl who recognized that many people have a fear of needles and sought to design a medical to device to help, or the student who liked going to festivals and thought about aligning attendee interests with the festivals' interests and waive attendance fees for attendees by having them volunteer at charities. It sounds like students in your class came up with relatable ideas by looking at problems in their lives that they noticed.…

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems is a really good book.

"Designing Data-Intensive Applications" is excellent, but it took me a long time to get through (it really helped that we read it in the book club at work). I am very glad I read it though - there is so much good stuff in it. I wrote a summary of it (mostly so I would learn the contents better): https://henrikwarne.com/2019/07/27/book-review-designing-dat... (https://henrikwarne.com/2019/07/27/book-review-designing-data-intensive-applications/)

Great thread. 2019 was the first time in about 7-8 years I made reading for pleasure a tertiary priority ((school+career)work/sleep/exercise+socializing+(NEW!)reading) and I got through about 20 books of varying length and difficulty, so here are the next 20 in my queue: - Infinite Jest (already started, will likely finish in early Jan. I love it so far.) - Quantum Computing Since Democritus (started in November as my nonfiction read, but took a break; I think I want to finish IJ first. Scott is an excellent writer and I was really enjoying this) - Building Microservices - The Code Book - How Cars Work (I want to turn the projection of a car in my brain from a black box to a gray box this year, and I don't mean by purchasing a Cybertruck) - To Kill a Mockingbird…

I just finished Kleppman last week. It took me since August since I was mostly reading during working hours, but I highly recommend it, especially as a companion piece if you already have a lot of familiarity with database technologies.

My recommendations are not exactly what you ask but what I suspect you'll enjoy if you're asking this. (1) "Understanding Computation" by Tom Stuart. Not "fundamental" as a deep textbook, but very approachable for programmers intro into a big chunk of CS, explaining deep ideas about languages using rigorous working clean code (in Ruby, no prior knowledge needed). I especially loved the first few chapters about what it means to define a programming languange and various kinds of formal semantics. (2) Designing Data-Intensive Applications, Martin Kleppmann. This gives you a phenomenally good survey of concepts and practice of distributed systems. This is more software engineering than pure CS, but in my view you can't approach the field of distributed systems without blending both anyway.…

At the Architecture level see the very good book "Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems" by Martin Kleppmann. For actual code, you do not have one book but have to glean the knowledge from a whole bunch of them.

Designing Data-Intensive Applications[0] by Martin Kleppmann. There's a previous HN thread about it[1]. Helped me understand a bit more about databases and systems. The book is also very approachable and has the perfect blend of application and theory at a high level that anyone approaching the industry for the first time stands to gain a lot from reading it. The Architecture of Open Source Applications[2] series is a good one for leaning how to build production applications and you can read it online. The chapter on Scalable Web Architecture[3] is a must-read. [0] https://www.amazon.com/Designing-Data-Intensive-Applications... (https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321) [1] https://news.ycombinator.com/item?id=15428526 [2] http://aosabook.org/en/index.html [3] http://aosabook.org/en/distsys.html

This should honestly be the top-voted answer. The standard palette of cargo cult options are at best a weak proxy for real-world experience. But Kleppman's book is genuinely and outstandingly useful for understanding the various components involved in modern backend architectures.

As far as a technical book, Data Intensive Applications by Martin Kleppman. This book covers distributed systems as well as databases, data-warehousing, and data-processing. A lot of tools and patterns you don't see in school that describe how most of the "magic" behind high-powered software we use is implemented.

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems by Martin Kleppmann. Absolute best book on system design.

For a modern look at systems design i highly recommend "Designing Data-Intensive Applications (DDIA)" By Martin Kleppmann. Not really about structuring code, but I think stepping back and realizing your code is part of a larger system is very illuminating and influence how you write code. http://dataintensive.net/

+1. Great for getting a lay of the land on distributed systems problems. I haven't ever studied distributed systems formally though, so I wonder how this differs from that.

I've been thoroughly enjoying "Designing Data-Intensive Applications" by Martin Kleppmann. It primarily deals with the current state of storing data (databases, etc) starting with storing data on one machine and expanding to distributed architectures...but most importantly it goes over the trade-offs between the various approaches. It is at a high level because of the amount of ground it covers, but it contains a ton of references to dig in deeper if you want to know more about a specific topic.

Designing Data-Intensive Applications by Martin Kleppman. He does a great job of distilling storage systems to concepts and discussing conceptual trade offs instead of focusing on particular storage products. Plenty of footnotes to relevant research papers too.

It's not even published yet, but based on the 7 of 11 chapters available I can say the book gonna be really fundamental. Designing Data-Intensive Applications The Big Ideas Behind Reliable, Scalable, and Maintainable Systems By Martin Kleppmann http://shop.oreilly.com/product/0636920032175.do http://dataintensive.net/ The author has great sense of humor.