
CTO @taktile_org prev: @databricks @neondatabase @mongodb @tweetdeck @twitter. All views my own.
Dublin
Joined June 2009
- Tweets14.4K
- Following1.8K
- Followers1.4K
- Likes21.5K
James Broadhead retweeted
😱
I have 5 simultaneous grilling sessions going
Not sure how I got here, but see you on the other side
I blame @victorditadi , @mario_cadenas , and @AndrasKauer for this
James Broadhead retweeted
Ship enterprise apps faster with Databricks AppKit and @Replit.
AppKit is a TypeScript framework with built-in observability and seamless integration with Databricks services. The new Replit integration lets you develop data-aware apps using natural language and deploy directly to Databricks with Unity Catalog governance built in.
Less code, fewer bugs, faster production. databricks.com/blog/ship-ent…
James Broadhead retweeted
Wait, I’ve seen this UI before 👀
Really excited for this launch. The Neon/Lakebase team has worked incredibly hard to unify the backend of Databricks Lakebase and Neon Postgres, and we’ve already seen major efficiency gains along the way (including pricing reductions and more free-tier projects).
We now have two entry points for the unified backend:
- Neon Postgres → serverless Postgres for developers
- Databricks Lakebase → serverless Postgres for data & AI apps
All of this shipped in under six months since the acquisition. The velocity has been incredible and we’re just getting started 💚 More work to be done but very much looking forward to what’s next as we tap into the full Databricks ecosystem for Neon 📈📈📈
We're introducing a set of upgrades to Lakebase that simplify how teams manage operational data for Data Intelligent Applications.
What's new, including eagerly-awaited @neondatabase features:
- Serverless autoscaling with scale to zero, for dynamic workloads
- Instant database branching for faster, safer development, testing and CI
- Automated backups and point-in-time recovery for fast restore
- Expanded Postgres version support
- A new UI that simplifies everyday workflows
These updates represent a significant milestone in defining the lakebase category, a serverless database architecture that separates OLTP storage from compute. databricks.com/blog/lakebase…
James Broadhead retweeted
We're introducing a set of upgrades to Lakebase that simplify how teams manage operational data for Data Intelligent Applications.
What's new, including eagerly-awaited @neondatabase features:
- Serverless autoscaling with scale to zero, for dynamic workloads
- Instant database branching for faster, safer development, testing and CI
- Automated backups and point-in-time recovery for fast restore
- Expanded Postgres version support
- A new UI that simplifies everyday workflows
These updates represent a significant milestone in defining the lakebase category, a serverless database architecture that separates OLTP storage from compute. databricks.com/blog/lakebase…
James Broadhead retweeted
Everything we have built in the past few months was made with AI in mind.
Building a simple SDK is much different then before, you have to think of things like:
- LLMS.txt
- Context7
- Claude Code Skills
- MCP Prompts/Resources/Tools
- Use LLM-friendly patterns
Check it out!👇
James Broadhead retweeted
So proud of the team. The platform we've built is now serving Postgres for thousands of enterprise customers, powering products we all use every day.
databricks.com/company/newsr…
James Broadhead retweeted
How to connect to @neondatabase?
James Broadhead retweeted
Omg i finally understand why founding engineer and i speak opposite languages
He says “so what are we solving for this machine” and im like “oh thats an interesting question. Let’s experiment, EDA, and figure it out” and he is like ???????
James Broadhead retweeted
Need a Postgres DB for a quick demo, experiment, or random thingy?
Free Postgres - no signup, no credit card, nothing. Spins up in milliseconds, claim it later :)
👉 neon.new
James Broadhead retweeted
I'm so excited to see this hit, its legit every bit as magical as it sounds.
I spun up a project over the weekend with it. Such a clean flow, just "npx neondb --yes" and you're rolling.
It'll even add the variables to your .env file!
@neondatabase team is killing it.
Need a Postgres DB for a quick demo, experiment, or random thingy?
Free Postgres - no signup, no credit card, nothing. Spins up in milliseconds, claim it later :)
👉 neon.new
James Broadhead retweeted
Most people assume that doing good work repeatedly develops good taste. But the causation mostly runs the other way.
People with refined taste find it painful to produce subpar work in areas they care about. And the signs have always been there, since childhood.
Their aesthetic sense creates an internal standard that drives them toward excellence, not because they crave external recognition, but because anything less feels wrong to them.
Compare this to people primarily motivated by external validation: recognition, rewards, or peer approval. They will optimize for what gets noticed and what is currently popular rather than what is genuinely good. With more experience they might produce impressive-looking work that hits the right notes for their audience, but this work lacks a deeper quality and differentiation.
This explains why some people naturally gravitate toward creative excellence while others demand KPIs and objective benchmarks of what’s acceptable to the right stakeholders.
It's not just about work ethic or ambition — it's about having developed the capacity to sense the difference between excellent and mediocre work. Once you can perceive that difference by directly examining the work, settling for less becomes difficult.
In other words: the drive for good work emerges from taste rather than creating it.
James Broadhead retweeted
3/5 Clark is the first AI @databricks Apps Engineer – connecting to Databricks SQL, Mosaic models and @neondatabase, while enforcing Unity Catalog permissions
James Broadhead retweeted
Watched the original "Functional Core / Imperative Shell" talk from 2012, by @garybernhardt - I'd long known of the idea but had never been to the source.
Some notes:
- He was aware of Alistair Cockburn's "Hexagon Architecture" before the talk
- He envisioned needing to break things up into functional "islands" in sufficiently large or complex programs. Contrast this with what I am tentatively calling the "Will Wilson school", where the pre-requisite for deterministic simulation testing is writing an entirely deterministic program as one solid piece.
- Ruby is so clean and I sort of miss it (code examples were in Ruby).
James Broadhead retweeted
🐘 Today, we’re excited to announce the GA of Serverless Postgres on @gokoyeb — production-grade, fault-tolerant, and scalable Postgres databases with up to 32GB of RAM, purpose-built for large-scale and AI applications🧵
James Broadhead retweeted
10 years ago I left Heroku. Heroku has worked the past few years towards replatforming as a K8s platform. I do hope there is some fresh life breathed into Heroku.
I also don't want to rehash the many debates of what is next for Heroku, whether it was a success or failure, or how it could have been different. Heroku is still used as a gold standard for Heroku for X when it comes to developer experience. But I don't want to talk about the DX of Heroku, I want to talk a little of the story about Heroku Postgres(DoD - Department of Data as we were internally known).
I was at Heroku in a lot of product roles over the course of 5 yrs, but most of my time was with that DoD team. When I left Heroku it was a team of about 8 engineers running and managing over 1.5m Postgres databases–a one in a million problem was once a week, we engineered a system that allowed us to scale without requiring a 50 person ops team just for databases.
This will be a bit of a personal journey, but also hopefully give some insights into what the vision was and hopefully a bit of what possibilities are for Postgres services in the future.
I wasn’t originally hired to work on anything related to Postgres. As an early PM I first worked on billing, then later on core languages and launching the Python support for Heroku. It was a few months in when I found myself having conversations with many of the internal engineers about Postgres. "Why aren’t you using hstore?", "Transactional DDL to rollback transactions is absolutely huge!", "Concurrent index creation runs in the background while not holding a lock, this should always be how you add an index." Now we had some great engineers, but it was the typical engineer that interacted through ActiveRecord and didn’t want to think about the database.
As I found myself evangelizing Postgres, suddenly I was being recruited by the lead of the Postgres team to come and do marketing, I didn’t know anything about marketing and thought they were joking. A couple months later found myself doing PM and marketing for DoD.
Why did Heroku pick Postgres?
But, I’m getting a little bit ahead of myself. How did Heroku even start doing Postgres or why? Running a PaaS (platform as a service) is a lot of work, running a database is a lot of work. In some sense doing both is splitting your focus. Well, way back in the day we had all these Rails developers asking for a database and we thought how hard could it be? (It was more work than we expected). So we’re gonna run a database, the question becomes which one? Most folks didn’t have strong opinion, but one of our security/ops engineers chimed in "Postgres has always had a good record of being safe and reliable for data, I’d go with it."
And with that, we were building and launching Heroku Postgres.
The first version of Heroku Postgres, no automation, no self service provisioning, you opened a ticket and we’d correspond and ask when you wanted us to set it up for you.
Before Heroku Postgres was Postgres
The very first version before we committed to Heroku Postgres was internally known as Shen. The model was much more akin to shared hosting that was common for that time and place. Within a single instance we’d pack in a lot of Postgres databases, simply running createdb, creating a user for you, then giving you access to that DB. This worked fine for those just kicking the tires and building small apps, but despite telling them to not use it for production people continued to.
While Heroku had Postgres before Heroku Postgres it became a project and its own orchestration layer for databases as a more first class citizen around 2010. The initial codename for the project was "bifrost".
The original design
Heroku Postgres was designed to have a central FSM (finite state machine) that would orchestrate the databases. This design pattern to my knowledge came from PvH’s work and appreciation of video game system design. It felt like a novel approach to the software being built at this time. The fact that it is a more common design patten now shows what a great design it was, and how ahead of its time the level that team was building at.
It was a basic Ruby application that would go out and check the state databases and go through the needed steps when interacting with either Postgres or AWS APIs. AWS APIs back then were not what they are now and this allowed to build in necessary retries, redundancies, and quality of service.
Sometimes you’re good, sometimes you’re lucky
Sometimes you’re good, sometimes you’re lucky, sometimes it’s both. Over time we built out more reliable provisioning and monitoring of databases. In early 2011 we felt a need to continue improving this. It was the early days of EC2 and reliability wasn’t the strongest spot, instances could go offline.
Per Dan Farina "We were thinking about working on replication but skipping over archiving (by more carefully managing state between servers, e.g. by directly moving things through rsync, which was/is still pretty normed postgres stuff predating pg_basebackup) but then one of the shared databases (shen) had a near miss when a disk was lost that caused a rather horrific amount of effort and some nail biting in restoring from pgbackups."
"We then decided to take previouly rejected approach of building everything up on the archives. the first versions were s3cmd based, and were something of a prototype, upon request from PvH to ship something more raw, but more quickly. We had just got early versions into production when the major disk apocalypse hit in April 2011, though it was in something of an evaluation period and it was not yet a well-exercised & monitored program, so we crossed our fingers and, thankfully, it worked on every database, once AWS had capacity to spare."
At the end of the event it was communicated eventually that if we wanted AWS could give us all of the EBS disks back, but it could be all of them were corrupted. As someone we can’t recall adequately described it: "It’s like getting a gallon of ice cream, but it may have a turd in it".
More from @danfarina: "Were it up to me at the time, I probably would have moved onto converting it to boto (one of the most mature AWS client bindings at the time, by far) before stopping to deploy the s3cmd near-prototype on every instance, but that would have been a disaster."
Our applications were resilient because we could leverage multiple dynos and the routing layer available to our system. Databases are a little different, but how could we at scale give the features you most needed for a database:
- No data loss
- Improved availability
Number one was always a core charter for the team and we would prioritize this over features, and over uptime. Uptime mattered, features mattered, but as a data provider if we lost data we’d lose trust. Thus that quick prototype became wal-e, which then went on to power many of the future features of Heroku Postgres, and be used for many years, though now has been deemed obsolete in favor of more modern tooling such as pgBackRest. But for it’s time and place it was some good execution and some luck on timing.
Thinking about the entire experience
As Heroku sat at a central point of app deployment we actively tried to think about the experience end to end. This manifested in some of the small things we actively campaigned for and collaborated with the community members who could make these happen. A few key examples come to mind for this.
The first was DATABASE_URL. Some of this originated from the 12factor concepts, others in that having 5 environment variables to define what you were connecting too felt verbose and cumbersome. Why couldn’t Rails just use DATABASE_URL if it were defined. I don’t recall the specifics here, but suspect this was something we nudged Terence Lee to help with.
The second was around some features of Postgres. At the time Postgres was going well, but most of the core community focused on performance or enterprise-y features. We were coming at it from a different angle with an audience of Rails developers. We were intentional and engaged with some of the Postgres consultancies that employed committers, with a general theme of how can we help contribute, while also advancing Postgres based on the knowledge we have from users. A few highlights here included:
- Not just the DATABASE_URL on the Rails side, but also on the native libpq wire protocol. While we didn’t do the work here, we were spent notable time advocating and engaging around it.
- pg_stat_statements in my opinion now the most valuable Postgres extension existed before, but was effectively unusable for most applications. Funding this work was foundational to make Postgres have more usable insights natively.
- JSON/JSONB collaboration
Of note we later hired several contributors to largely focus their time on upstream Postgres itself.
Dataclips vs. the team
Throughout the history of Heroku Postgres various individuals made a series of bets. First it was PvH pushing for Wal-E to get out the door as an MVP, which was absolutely the right call in retrospect. Perhaps the one that is most exciting to me and people least associate with Heroku is dataclips. Matt Soldo, had this idea of GitHub Gists for your database. But in the early days of Heroku as a PM, like many places as PM you didn’t have an ease to mandate engineers try a thing. You had to campaign and convince for a thing.
Soldo didn’t build up enough buy in from the team, but had done plenty of awesome things that it was worth letting him run with this. We were all wrong. Dataclips was built by an external consulting company as a separate standalone Heroku app. It was only live for weeks and suddenly it was powering all of our internal dashboards. A live query you could embed into google sheets, suddenly our internal KPI rails app was replaced by dataclips and a google sheet. We didn’t need looker or tableau or other fancy BI tools and this lasted into 100m in revenue for insights into the business. To this day dataclips is one of my favorite features of Heroku, and I look forward to making an experience like that but even better, thanks Soldo for not listening to the rest of us back then.
Names matter
For a long time databases haven’t been known for being user friendly. We wanted to pull from other paradigms as we were making key database tenets available to people. We looked heavily to git for inspiration around forking/following. Database terms were common as master/slave, but we knew we could do better. Ee wanted to give the user a sense of what they could do with it. Archiving the WAL every 16 MB or 60 seconds (whichever was less) became continuous protection. Forking, was a snapshot as of some point in time. Follower, something that followers a leader node (a read-replica). I still recall an hour long analyst call with Redmonk with James and Steve O'Grady–it was mostly wind them up and let them go (for the record James didn’t love fork/follow, I think he may have come around now).
This started even earlier than my time with being intentional about Postgres vs. PostgreSQL. But I’ve covered that one before, and you can even see it in some of the other lobbying around libpq.
Peacetime vs. Wartime
Things were rolling along well, we were shipping new features. We’d added dataclips, fork/follow were in existence for a while–and then we got a note from Amazon they were launching Postgres support for RDS at the next ReInvent. I was in person at that ReInvent and I’d never seen a roaring standing applause like that at a tech conference before when the moment was announced. In private channels we heard notes that this was because of us, the excitement and demand for Postgres became too clear for them to ignore and they had to add support.
We felt vindicated in our choice of Postgres and in what we’d built. But we also knew that now we had competition, running a database as a service on another infrastructure provider how can you compete? Well from some years of experience now I can say there are definitely ways and am confident that sharp narrow focus allows you to build amazing products which can be harder to do inside a large giant corporation. It was at one of allballs (UTC 00:00:00-when the data team would do happy hour on Fridays) that we were drinking beers and discussing how now we really had to focus. PvH and Farina were discussing me personally as a leader–apparently I’m okay in peacetime when things are good but in wartime they were willing to bet on me.
Two weeks later I walked into the exec team meeting with a 2-pager assessment of what may happen, how we could compete, including 3 potential acquisition targets that could allow us to have a more differentiated offering. Within a few months we made one of those acquisitions which later became Heroku Connect. It made a lot of sense for many reasons, including Adam Gross was an angel investor in Heroku and knew it well and had helped build Salesforce Platform in early years. That wasn’t the end, but was just the beginning of how we could actively compete vs. simply being "hosted Postgres" vs. a more fully managed experience.
Metrics and Monitoring that almost was
The next vision and goal for Heroku Postgres was to continue to give better ease of use and insights into your database. Postgres itself already has a ton of awesome data inside it about how it’s been used. The catalog tables and extensions like pg_stat_statements have a wealth of information, but querying it looks like 200 lines of black magic SQL. Will L was perhaps one of the first, and the best at the team and quickly making something usable for people. The first step on our metrics journey was him making these Postgres insights trivially accessible via pg-extras.
Continuing on the journey internal foundational systems were built to focus on collection of various meta data from systems and the ability to notify and communicate. I’m blanking on some of the systems, though some were obvious–observatory (I’m not sure if still in use or not) would observe databases.
These systems started to house a lot of information that then powered pieces within your Heroku Postgres dashboard. Things like slow queries and high IO or CPU load would give good insights when you logged in. The eventual goal was to connect the dots through proactive notifications. It’s one thing to get an alert from pagerduty that things are off, login to a dashboard and fix it. But what if in the early signs of things starting to go south we emailed you that you’ve got an increasing number of sequential scans that are starting to put you at risk of IOPS saturation, and because we understand Postgres you can add an index and resolve this with a single command. We could even give you an easy button in your email to add the index. All the foundations were in place, if you’ve run on Heroku you’ve gotten the notifications about database maintenance, that was powered entirely from the underlying notification system built with this goal in mind.
Postgres can still be better for developers
But we never made it there. Some of us shuffled to different teams, some of us moved on to new challenges, and many folks came on after to continue to run and power a great database service. Some of those original engineers Daniel Farina and Will Leinweber (along with PvH) understand the design and why as well as anyone. The goal from early on was that we could do better for developers.
I'm continually asked why I came to @crunchydata... aren't I tired of working with Postgres and doing the same thing. I tell them I have unfinished business. After the success at Citus tackling scalability problems where the average customer being 40TB in data, I attracted to the idea of returning to the vision we had back at the DoD of bringing a better Postgres experience to developers. Despite the rapid growth of successful DBaaS offerings, there was still something missing - that initial idea of DoD that we still wanted to create.
Postgres is an amazing database, can handle hundreds of thousands of transactions per second often without batting an eye. Has internal data that you can easily look at to assess and improve performance. Has a rich set of datatypes, indexing, and functionality. The extension ecosystem is vast. But as a developer you don’t have time to become an expert on Postgres.
- What if we told you when a N+1 query snuck into your Rails app?
- Connections don’t have to be a limitation on Postgres when you have pgBouncer right there.
- Have excessive indexes from the early stage of building your app? What if we told you about them and with a button click you could drop them.
One of our @crunchydata customers described it better than I ever could. When working with some of our experts on some deep dives into their database they said “you should take all his Postgres expertise and just bottle it up and send it in email or slack reports to me.” I want that expertise as a product.
My product strategy isn’t to go and change the world of databases. Postgres is a great database with a community that is making it better daily. I want to help make open source Postgres better and give back to it along the way. Oh we’ve ship some cool things along the way too such as scheduler, saved queries, database insights, and recently logical replication from Postgres to Iceberg–but at the core...
My product strategy is to distribute deep Postgres expertise in a consumable form to every customer of ours in the coming years.
James Broadhead retweeted
Sailing is the only place where chaos and calm coexist so perfectly—where physics becomes art, problems become puzzles, and the horizon's embrace feels like coming home.
We set sail from Africa this week, longest trip of our lives.
dashboard.seapeopleapp.com/l…
James Broadhead retweeted
My 1st day at Cloudflare, the now CTO gave me 1 big piece of advice:
Ship small wins early. Like ASAP.
The easiest way to build trust in a new org is to ship small wins early.
Ship small and fast (otherwise you're just the new PM working on the same darn feature for 6 months).
James Broadhead retweeted
We forked the Book Inventory template by @vercel, loaded 1M titles, and swapped out the AI-powered search for pg_search on Neon
The demo is FAST. Search through 1M books in milliseconds: fyi.neon.tech/books
James Broadhead retweeted
Three questions I ask before any product launch:
Is it simple enough that users can understand it in 30 seconds?
Does it solve a problem people actually have?
Would someone miss it if it disappeared tomorrow?
James Broadhead retweeted
“Personal software” that’s a nice term 👍
I used @v0 and @neondatabase to create a simple timeclock application in less than 15 minutes!
This is the era of personal software!





























