Higher Order Company

I received an interesting question about Interaction Nets: > Asperti and Guerrini themselves had certain reservations about the tractability of bringing Interaction Nets to life, do you have a path towards solving the issues they have outlined? This is not the first time and perhaps not the last this myth is propagated. AFAIK, that's not the case. Asperti is cautiously optimistic about INs, and, seemingly, frustrated that people misinterpret his own results and beliefs. In fact, he even wrote a paper to address these very misconceptions, where he defends INs (and even cites me) arxiv.org/pdf/1701.04240 He does, though, raise some concerns about real-world performance, and memory consumption, which were sub-par on his own implementation, BOHM. HVM addresses both. HVM is roughly 100x faster and more memory efficient than BOHM, and HVM3 is now faster and more memory efficient than GHC itself. It makes some tradeoffs, just like Rust, and it is not as general, but, given the right program shapes, it does achieve really great performance and memory footprint. Honestly, I think some key important people just misinterpreted Asperti's results back then, memorized their own misinterpretation, never bothered to update, and then it kept emanating it since. I think I don't really help the case since my communications read overblown (which, I swear, isn't intentional)... Anyway, HVM3 is really great right now. It is absolutely faster than GHC's runtime in a single-core for linear/affine programs, and the presence of SUPs and DUPs allow us to chase some major speedups (like we did on SupGen) when you know what you're doing. It is also worse than GHC in many cases, specially these involving lots of non-linearized sharing of first-order datatypes IMO, HVM isn't a great replacement for GHC when it comes to running the average FP code, but it is a tool that very advanced FP developers can use to chase absolute peak performance and almost miraculous speedups

morning! today I'll start reaching out to VCs. I want to close this round ASAP, start getting things done, beat ARC-AGI by EOY or fail for good. that's all, lg

AGI apart, what can HOC can bring to the world? ✨The child of Haskell & Rust, born with CUDA superpowers.✨ That's the best way to describe what a mature Bend would be. It would: - Be significantly faster than Haskell even in a single-core (HVM3 is evidence of that), and potentially as fast as Rust. - Be able to run on GPUs (although you probably want to manually pick the target, because not everything is better on GPUs). - Feature optimal evaluation (lazy cloning, superpositions, and all the stuff I'm exploiting to build SupGen right now). That said, it will definitely be a little more annoying to use than Haskell, due to linearity and label checks, just like Rust is a little bit more annoying than Zigs due to the borrow checker. And... that's about it though. Nothing more, nothing less. It is way less special than I make it sound, probably because of the way I communicate lol. Yet, it is really cool on its own right, isn't it? So, why isn't HOC doing just that? Well, give us $100m and we will 😅 Building a compiler for a whole new paradigm, an stdlib, and an entire ecosystem around it, isn't cheap, and I don't see a way to fund that. So, yeah, the plan of using HVM to build a closed source product that serves the current expanding market, in special, Symbolic AI companies, seems like a much more reasonable business direction right now. (Of course, there's still IN hardware, but it isn't time yet.) I just realized this is a very simple way to explain everything... Hope that makes sense (:

HOC is now small research lab that is laser focused on building the first Symbolic Transformer. We had many pivots through the year, but we're finally stable in a direction, with promising results and a clear business path. It also makes the most logical sense: use HVM to build the ultimate automation tool, and then use *that* to develop Bend/Kindelia, rather than the other way around. SupGen is the first step in that direction (and I think most don't see the implications of its benchmarks yet!). I'm confident it will, in a year or two, lead to a very powerful reasoning agent that nails benchmarks like the ARC Prize. But I'm also optimistic that reasoning LLMs will eventually lead to AGI. So, the big question is: which will happen first? I don't know. But there is value in trying different approaches. HOC is approaching AGI from an absolutist lens of *efficiency*, and we're not questioning just KV caches, but matmuls, transformers and even floating points. We want to get to the very core of what makes a computer capable of reasoning, distill that algorithm into its most efficient formulation, to only then scale it. Perhaps that's too extreme, perhaps that's what's needed? In any case, we're officially in this weird race now Let's see how it plays out

DEMO TIME SupGen is a generative coding AI... except it isn't an AI. There is no model, there is no pre-training. You just give it some examples, and it gives you a program. It runs locally, in a single-core CPU. Oh, and it can also prove theorems. Here's a demo, including a brief TT intro. (The synth examples start at the 6 min mark.)

HVM's Program Synthesis - Initial Results AI startups successfully developed systems capable of *learning* arbitrary skills. Yet, as impressive as LLMs are, they eventually hit a wall: the dataset they learn from. To break that wall, models were recently augmented with "search" - i.e., the ability to explore a vast space of ideas, at inference time. These new reasoning models are now, by many, believed to be sufficient to reach AGI. That said, o1-style reasoning, in a pure theoretical sense, is inherently inefficient. Neurons related to random facts, like "who is the president of Tuvalu" or "how did dinosaurs mate", metaphorically "fire" when a model is focusing on solving a mathematical equation, wasting time and energy. If compute is, indeed, key to reaching AGI, then, looking for creative new ways to boost efficiency is surely valuable. Here's a crazy idea: what if we did it in the opposite order? That is, for a moment, we ignore the "learning" part, and focus on just building the fastest "model-free" search algorithm that we can? Surprisingly enough, this approach is relatively under-researched, and embarrassingly under-scaled. As the creators of HVM, a fast symbolic processing engine, we're in good position to try. Over the last month, we implemented an initial prototype, and compared it against the SOTA alternatives on standard benchmarks. Below are the results. In all but 3 of the 32 cases considered, single-core HVM outperformed every alternative, with up to 325x speedups (!!). Our implementation is ~50x times smaller, example-agnostic (alternatives require cherry-picked examples!), and more general (it is trivial to extend it for pattern recognition, theorem proving, genetic evolution, etc.). There are a about ~5 cases which HVM couldn't solve due to minor limitations that will be addressed soon. This is a very early implementation that is being actively improved; yet, based on these initial results, we believe we found a new fundamental approach for fast program synthesis. The next step is to extend our framework with learning, scale it with massive compute, and observe which interesting capabilities will emerge.

despite making Bend as simple as I possibly could, 99.9% of people following me still have no idea what HVM is useful for. to most, it is no different than some pseudoscience bs. I eventually came to the acceptance that I shouldn't be trying to make people use interaction nets, simply because they're quite literally an alien programming paradigm that works differently from anything that exists, and, just like FP, requires deep understanding of a whole new field of CS to extract its full potential. except that, unlike FP, there aren't several universities developing that field... that's why I decided that, rather than pushing HVM into people and expecting they would use it, we should instead just make HVM as good as we can for our own internal usage, and then build things on top of it. our products should be things we build using HVM, and not the HVM itself, nor Bend itself, directly. sadly, moving feels slow because there is just *too much work* to do. suppose, for example, we want to apply HVM to AI. to do so, we need to, first: 1. build our own Haskell (Bend); 2. build own GHC (HVM); 3. build our own AI architecture on top of all that. each of these projects would be a massive, multi-year endeavor in isolation... and we need to do all 3 before we can even start extracting any value from this paradigm. and the worst part is, at least I think, we're succeeding. this shift in philosophy is starting to pay off now. I'm extremely happy about the results on HVM-powered program synthesis. the fact we can make something that is faster than all SOTA alternatives with a 1k LOC file in HVM is absolutely significant, and is a gateway to other things we'll build on top of it. a fully "optimal symbolic AI architecture" is something we will be researching later, but our (learning-free) proof synthesis application is essentially ready to be deployed and scaled. I want to make a playground ASAP and publish it for everyone to try, just like OAI did with GPT-2 back then, so that people finally go "wait so it can do *that*!? wow I get it now". literally the only bottleneck now is the CPU cluster, which we're starting to build. there is nothing else in our way. I hope we can get it done by March. that sounds like forever to me though, wish I knew how to get it done faster 1024 Minis likely to be the initial size, 50% out of my own pocket and 50% out of HOC's cash balance. I can't wait for it

btw, the best way to describe what I'm doing is: I'm maximizing the bitter lesson. that is, if ingenious approaches failed, and what made GPT so good was, ultimately, the combination of a dumb simple algorithm with fuckton of compute; then, what happens if we take this lesson by heart, and move further in that direction? that is, if we radically remove ALL ingenuity, leaving us with the dumbest learning algorithm that works: symbolic evolution. and then we aggressively optimize its constant factor (HVM's role), so that $1m of compute becomes $2m, then $4m, and so on. and *then* we train a model on that. what would happen? probably nothing. but... who knows? i wanna find out currently I'm in the "heavily optimize the constant factor" part, and it is already ooms faster than all SOTA proof synthesis algorithms published in the literature. next step is to plug that in an architecture that can learn (perhaps even by next token prediction) and, then, only god knows what will happen. perhaps loss curves will suck and I'll just laugh and call it bogoAI. or perhaps the universe will reveal it truly has no regard to human ingenuity, the bitter lesson will prevail, and this utterly simple approach will outperform everything that exists probably not though but trying won't hurt, right

we should probably create a playground like asap

times to synth a correct algorithm given 4-8 examples

hey guys - so, today I can confirm my hypothesis worked, and the main milestone I sought has been hit. the synthesizer implemented algorithms and solved problems that even o1 couldn't. using it is somewhat magical - it *feels* intelligent, even though I know isn't. it is just a really, reaaally fast explorer of the space of well-typed programs there are maaaany things to tune and optimize now. today was the first run, but interacting with it absolutely blew my mind and removed any doubt the approach works. i think this will become clear to everyone when we launch a web demo, and it should happen really soon, like february perhaps? i also think that adjusting this to ARC-AGI *should* work and I can't wait to try. but there is a lot to adjust yes I was supposed to be taking a time off but I couldn't stop thinking about whether it would work so I ended up immersed in code again 😅 well now I know

(amazing things happening at the higher enumeration company)

Experiment complete, here are the results 🥳 tl;dr: still exponential, 10x speedup (replacing loops by superpositions) Q: What is this chart? A: The evaluation time (log scale) of the dumbest factorization algorithm (brute-force) on both Bend (HVM) and Haskell (GHC). Q: How was this implemented? A: First we defined a simple recursive bitwise multiplication, and then attempted to solve `X * Y = P`, where `P` is the product of two primes, by using either a naive loop (on GHC) or superpositions (on HVM). Q: What is the result? A: Both are exponential, but HVM is up to 10x faster. Q: Is this multi-thread vs single-thread? A: No, both run in a single core. Q: Is this exploiting λ-encodings? A: No, both use just conventional datatypes. Q: Is HVM faster than Haskell in general? A: The compiler is getting there, but not quite yet. Q: Then why HVM is 10x faster here? A: Simply due to optimal evaluation. By using superpositions instead of loops, the runtime is able to ""memoize"" intermediate expressions, returning the result faster. In some cases, this gives an exponential speedup. In general, though, it reduces the exponent. In this case, we get a 10x speedup "for free". Q: Why this matters? A: I'm interested in automated theorem proving (an "AI" that solves mathematical problems). Usually, these are implemented with recursion and pattern-matching (like this benchmark). Given the importance of scale to modern AI algorithms, a 10x speedup "baseline" turns a $1m cluster into a $10m cluster. Q: Can we expect a 10x speedup on other problems? A: I expect much larger speedups. Integer factorization is a problem with barely any "exploitable" structure (that's why we use it for crypto). For proof synthesis algorithms, I believe the HVM speedup will be even higher. In an earlier post, I got 300x speedups on the task of finding bitwise functions. Q: Can I run this on my machine? A: Of course! All the code is on HVM3's repo. It is a WIP though. That's it for now. See you 🥰

How to use HVM3's collapser to solve a functional equation! 1. Implement a function on HVM3's syntax (Bend soon) 2. Enumerate all terms of X using superpositions 3. Write your equation on the main function 4. Run it with: 'hvml run main.hvml -C' And that's it. HVM will enumerate ALL possible values of X, looking for one that satisfies your equation, and it will print the first one that does. The catch is that this happens optimally, so, computations across "attempts" will be "memorized" (in a very clever way). And the cool bit is that this was never programmed in - this is an inherent feature of interaction combinators. As an example, the file below implements subtraction by implementing addition, then inverting it (i.e., solving for X in `X + A = B`). It computes any subtraction in linear time, even though enumerating all X in a loop (in any other language) would take exponential time. github.com/HigherOrderCO/HVM… And the best thing is this works with arbitrary functional algorithms. You don't need to write any logic-based clause. Just write a Haskell-like function and HVM3 will invert it. Note that this is STILL a brute-force method - it is just a very fast brute force. It won't magically be faster than existing tools for specialized tasks - like, a proper SAT solver algorithm will obviously be faster. That said, I believe the combination of clever algorithm design with this kind of optimal search has a lot of potential.

HVM3 is turning out *really* good, in ways that I didn't even anticipate. The move to Haskell+C was the best decision I ever made. Haskell gives me the expressivity I need, C gives me the performance. Extremely refreshing to watch everything aligning after a bunch of missteps

HOC - Complete Overview, no BS (Ask questions here!) Vision: - Interaction Nets are a powerful tech Mission: - Build powerful products with Interaction Nets Reasoning: - We believe this theory can result in major breakthroughs in many areas of CS, including programming languages, runtimes, compilers, processors, algorithms (physics, graphics, AI). HOC is a company devoted to exploiting that to make products. Theory: "Interaction Combinators" - Lafont 1997 - A model of distributed computation - Like Turing Machines or λ-Calculus, but FAST: - 1. Massively-Parallel: near-ideal speedup/core - 2. β-Optimal: asymptotical algorithmic speedups Team: - Founder: Taelin, a computer nerd who codes 24/7 - Founder: Sipher, a gamer friend who does all else - +7 Devs Advisors: - Alex Van De Sande (ex-Ethereum) - Aaron Stump (professor of CS) HVM1: - First practical, real-world IC implementation. - ✓ fast-ish (comparable to Go, JS, Haskell) - ✓ β-Optimal (algorithm speedups) - ✗ M.Parallel (concurrency bugs) HVM2: - Parallelism solved with atomic interactions. - ✓ fast-ish (but 3x slower than V1) - ✗ β-Optimal (due to no lazy mode) - ✓ M.Parallel (GPU support / 16k+ threads) HVM3: - Unifies the best of both architectures (soon). - ✓ fast (Rust-like codegen: ~10x faster 1-core) - ✓ β-Optimal (like V1) - ✓ M.Parallel (like V2) Business Plan A: Solve a Big Problem™ - We believe that, beyond parallelism, HVM can be used to build new logic-based AIs. That's because its "optimality" makes some loops exponentially faster (via "work sharing"), in a way that's impossible to emulate directly on traditional VMs. - We've shown examples of 300x-speedups vs raw brute-force search, and we're currently extending this approach to evolutionary algorithms, on a cluster of M4 chips. We believe this will result in AIs that greatly outperform GPTs in logical reasoning. If that holds true, the business direction becomes immediate. - HOC's "AI" is completely orthogonal to the current LLM craze. Whether that's good or bad, depends on who you ask. Essentially, we take the "bitter lesson" and "scaling is all you need" at heart, combine that with "HVM's boost", and apply it to under-the-radar algorithms. Business Plan B: Kindelia - Kindelia is a P2P computer, like Ethereum, but with the EVM instead of the HVM. This would greatly increase the compute throughput of the network by parallelizing not just contracts, but individual operations inside contracts. Moreover, contracts would run on GPUs, not on CPUs. This project is currently paused. If Plan A fails, we consider re-focusing it. Business Plan C: Bend/HVM as a product - Bend currently lacks features that would make it compelling to use for front-end and back-end development: a stdlib, a package manager, a compiler, and other key features. It isn't far from being a solid alternative to Node.js and similar. - Selling programming languages is hard and Bend is meant to be open-source, but, if all else fails, HOC considers turning Bend into a product. This can involve the creation of specialized HVM hardware, which HOC also believes could greatly outperform CPUs and GPUs in compute/watt. Rounds: - 2021: $200k pre-seed raised - 2022: $4m seed raised @ $30m valuation - 2024: reached out to 15 VCs - refused 2 offers We're currently accepting individual investments, at a $10m cap, at $200m valuation. All extra money will be used to increase our CPU cluster. The logic is simple: since our algorithm is evolutionary, the more chips we have, the faster it will progress and the more likely it'll succeed. Resources: - We have $1.5m left (3-year runway). - Soon: 512x Mac Mini M4 (donated by the founder). Materials: - See below ↓

I'm open-sourcing HVM's Symbolic AI research It doesn't matter how many scientists and engineers you hire, how many fancy papers they write, how many citations they have, or how outstanding is their work. You're not going to land on the moon by building fancier balloons. Breakthroughs don't come from improving existing primitives to achieve an impossible goal. They come from finding entirely new primitives, that make that goal easy. Without fuel and rockets, we would never have landed on the moon. That's why I personally dismiss past Symbolic AI research. Not because I think people who worked on it aren't brilliant - quite the opposite! But because I believe there is a fundamental primitive that is required for it to work, and said primitive has never been used before. When I post about optimal evaluation and fan nodes, seems like it goes over most heads how fundamental I think they are in this context. For Symbolic AI, they're the invention of fuel, and it may be actually impossible to build strong Symbolic AIs without them. A few days ago, I posted a small example where HVM outperformed Haskell (and, thus, every existing language and runtime) in the task of finding some simple binary functions. Despite staying all day on front page of /r/haskell, nobody managed to contest this result. Given proper scale, such asymptotical speedup can be the difference between a $1k to a $100 trillion run. Or, equivalently, the difference between trying to land on the moon via hot air or rocket fuel. Most AI researchers nowadays correctly dismiss Symbolic AI because of its failure in the past, but they don't realize that all past methods had a major asymptotical slowdown that prevented them from being any good. Now, if there is, suddenly, a new primitive that completely changes the complexity of the problem, then, the (valid) reason behind this dismissal is now invalidated, and we must absolutely revisit the idea. Remember the first AI bubble was centered around Lisp, Prolog and similar, and the results were as mind-blowing as GPT-4 is today. Back then, people believed we'd soon have strong automated reasoning systems. Yet, one factor ruined it all: computational complexity. Now, what if Prolog was like "Bubble Sort", and we're close to finding the "Merge Sort" of logic programming? That's what I strongly believe optimal evaluation will give us, and why I urge that intelligent people start exploiting optimal evaluation to design new algorithms. Starting from now, I'll be open sourcing and documenting my ideas. This will be a slow, granular process over the next months or years, because I'm extremely busy with HOC and HVM. Today, I'll release that small, silly file, where we find some binary programs on HVM significantly faster than Haskell's brute-force (Omega Monad) approach. Here's the Gist: gist.github.com/VictorTaelin… The implementation is fairly simple: 1. We create a superposition of all Bits->Bits functions (ENUM). 2. We apply all functions "at the same time" to a few test cases. 3. We collapse back to a single value by matching on SUP. And that's it. This simple thing is grossly faster than it should. The main novelty (compared to my former posts) is the collapser, which now pattern-matches on SUP nodes, using a priority queue to perform a "breadth-first collapse". This requires a new primitive that allows us to "quote" a SUP node, and it is available on the `dup_labels` branch of HVM1. Without that primitive, the only way to collapse would be by using DUPs, which could duplicate work when the target label isn't present on the output. As far as I can tell, this primitive is necessary for the approach to work. Now, keep in mind this is a very simplified, "minimal" example of this kind of setup, in order to demonstrate the benefits of SUP nodes. The main thing to understand is: 1. SUP nodes allow you to place two terms on the same location. This naturally creates many "parallel universes". 2. Optimal evaluation will ensure all "parallel universes" are executed with maximal sharing; i.e., with the minimal amount of "wasted work". The combination of these two facts is extremely powerful, and can be used design all sorts of efficient symbolic algorithms. A few days ago, for example, I implemented MiniKanren (a Prolog-like, logic-based language) with SUPs instead of disjunctions, and it, once again, resulted in a significant speedup. Note there are many quirks one needs to learn to design great optimal algorithms, but that comes with experience. My main advice is to avoid cloning a variable when it isn't needed (by linearizing your programs). --- In short, SUP nodes are an intuitive, powerful primitive that, under optimal evaluation, can be used to design efficient symbolic algorithms in a way that, as far as I can tell, was never done before. I'm extremely confident clever usage of this primitive will result in novel algorithms, and I also think this is *the* missing piece to making strong Symbolic AI algorithms. Currently, I'll not working on this for the next few months because I need to focus on the next iteration of the HVM, but I can't wait to resume this research. I've had some major mind-blowing moments, and I'm sure these who explore this idea will understand what I'm talking about!

Some updates on HOC's plans: - I'm currently in the process of formalizing HVM's codebase in a higher-level language. Once that is done, we'll be able to 1. increase the security by formally verifying its behavior, 2. target *all* market GPUs without needing tons of redundant code, 3. begin working on a proper compiler and, thus, all missing single-core optimizations. - Since Symbolic AI has been separated from HOC's business plans, once the process above is done, I'll publish a series of articles on how to exploit HVM's optimal evaluation to accelerate program search and similar computations, and I'll make all my progress here public, so that everyone can work together on this problem. In a world where compute is one of the most important factors in AI capacity, I believe the complexity gains and the parallelism that HVM brings us can play an important role in the progress of AI. - Bend is actively developed and improved from release in many aspects, but ultimately, it depends on HVM for features and performance. As I've said earlier, you can expect a large bump in every axis around EOY. - HOC's Series A is still ongoing and we have some initial commitments, but we're still seeking a lead investor. Since we're not going to build a cluster, our round is smaller now, so it is most likely we'll not be able to fit everyone. We're not in a hurry either, we still have a 2yr runway with current burn rate, so, we'll take time to select the best potential partners for our long term plans. As I've said before - my ultimate goal is to make HOC the leading compute company in the world, I strongly believe there is no better way to build processors in theory, and we'll keep working to make it happen in practice.