M retweeted
1. "My roommate's gone for the weekend"
That's an address and a date in one text. Guys reply "nice, enjoy the peace" and wonder why she goes cold.
Reply "So you need company?"
Then lock in a real time. Vague guys get left on read.
SF bros will do amazing things to not talk to people
We’ve raised $1.3M from YC & Afore Capital so you never have to type or speak again.
Typing is slow. Talking into your computer is awkward. Subvocal fixes both
Our glasses give you the speed of speech without you making a sound or touching a keyboard.
It tracks your internal speech muscles with non-contact sensing and decodes silent, almost invisible mouthing into text at more than 200 words/min.
Pre-orders are now live at:
subvocal.co/preorder?utm_sou…
M retweeted
we need to build a honeypot for strivers, fake prestige, 0% acceptance rate, infinite application rounds, a 40 page essay, 7 interviews, a take- home that takes 3 months, and the final round is just another application, trick is seeding it with a few real prestige ppl as alumni, one famous researcher, one founder, one random anon with insane aura, they vouch for it and never explain anything, then a small inner circle quietly accepts ppl who are already legit, so the alumni list looks insane and the verifier looks real, strivers see the names, assume its the ultimate signal, and grind it forever, decoy verifier with a hand -curated leaderboard, let them goodhart against pure noise forever
Replying to @alz_zyd_
If you don't have the eyes of a critic you will never have the hands of an artist
It’s kind of unfortunate but I believe everyone has a short window between like 18 and 30 to become truly elite at a skill of their choice before the energy of youth and probably neuroplasticity drops below a critical threshold
Always found the "curse of dimensionality" a bit of a vague concept -- Wikipedia gives about 10 different definitions. It's actually pretty concrete in the case of function approximation though. Say your data is generated from a noiseless function:
Y= f(X), where X = (X₁,…,Xₚ)
where you make no assumptions about f.
p here is the dimension of the input X.
You're given a collection of n pairs {(Yᵢ, Xᵢ): i=1,...,n}. Now you get a new X' and you want to generate Y' = E[Y| X = X'].
Assume we can take derivatives of f and we can evaluate f at any point in our training set, but we can't directly evaluate it at the new point X'. One way to estimate Y' would be to try and approximate f(X') via a local Taylor expansion, say from the closest point in the training set X₀ relative to X', based on its first k derivatives:
fᵀ(X') ≈ Σ (f⁽ᵏ⁾(X₀) / k!) * (X' - X₀)ᵏ
The assumption that we can approximate in this way is optimistic but not totally unreasonable as many learning algorithms (e.g. linear regression, splines, local polynomial regression ...) can be interpreted as local (relative to either a mean, or a set of nearby points) low-order approximations with estimation procedures for the derivates of the Taylor series. Anyway we want to analyze how the error in the approximation scales *even if* we assume we could approximate in this way.
The Taylor remainder theorem says that the error made in this approximation is exactly:
f(X') - fᵀ(X') = (f⁽ᵏ⁺¹⁾(c) / (k+1)!) * (X' - X₀)ᵏ⁺¹
where c is some point between X₀ and X' where we evaluate the k+1-th derivative. Assume for now the first terms behave like constants w.r.t X'. The error we make then scales like O(hᵏ⁺¹) where h is the distance between X' and its closest neighbor X₀.
So how wrong are we likely to be on average? Well it depends on how large h = X' - X₀ is on average, i.e. how far a new point is likely to be from its closest point in the dataset. This is clearly dependent on how many points you have and how big the space X can occupy is.
Let's assume every coordinate X₁,…,Xₚ is chosen uniformly at random on the interval [0, M]. If M = 1, then the volume of the overall space (all possible points) is V = 1ᵖ = 1.
(M=1 may seem restrictive but we just want a notion of relative distance -- you can imagine the absolute volume will scale exponentially in M but the same scaling laws still apply).
With n datapoints uniformly distributed, if you assume each point X occupies its own cube of side length s (i.e. assume on average the points are equidistant). It's not hard to show that s = (1/n)^{1/p} and, assuming points are distributed this way, the average distance to the closest box center (another point) will be the ~√p*s. So the distance h scales like:
h ≈ √p (1/n)^{1/p}
So the error in the Taylor approximation decreases as n increases but if p is large then h decays extremely slowly in n -- this is the so-called curse of dimensionality w.r.t the dimension p. You need an exponential number of points n to achieve small approximation error because points become further and further apart as p increases.
A few things come to mind (observed by many others, I'm sure):
-- For any real-world X, if it's raw form has dimension p, the function needed to predict the associated Y probably has much lower dimension. You do not need every pixel to classify an image, etc. This is the basis of work showing how various approximating classes escape the curse of dimensionality e.g.
arxiv.org/abs/1401.7278 [which assumes the data lies on a manifold with dimension d << p]
jschmidthieber.personalweb.u… [assumes a compositional function that is sparse at each level of the composition]
-- The term f⁽ᵏ⁺¹⁾(c) I called constant earlier can also be used to tighten approximation error bounds if you assume f satisfies some smoothness conditions (bounds on its derivatives). Essentially if f has bounded higher order derivatives then a low-order approximation is probably pretty accurate. This is also assumed in the works above.
2
21
2,427
PyTorch help requested.
Say I have a simple linear NN where:
Ŷ = (W ⊙ Z)*X
W is weights, Z is a binary mask, X is input
Define loss L(Ŷ, Y). If I compute ∇L = ∂L/∂W, I should have (∇L)_i = 0 wherever Z_i = 0. Similarly if I compute the Hessian H, I should have H[i,j] = ∂²L / (∂W_i ∂W_j) = 0 wherever Z_i = 0 or Z_j = 0. So H is sparse in the square of the sparsity of Z.
Question: how to only compute only the non-zero elements of H most efficiently in PyTorch without detaching elements from the comp graph etc. Here is a minimal example here with my current naive approach of just looping over non-zero rows one by one:
codedump.xyz/py/ZrevVAm9YJpz…
I expect a solution would involve some sort of vmap or some other way of batching computations across rows/cols while ignoring zero values.
Still waiting for someone to match my freak [reply speed in asynchronous research discussions].
1
16
1,960
Sometimes when I formulate a convex problem a miniature devil appears on my shoulder asking me if my method scales with data and compute
1
12
2,036
Type of guy who’s never built anything but identifies with the collective movement of people that build things
4
56
3,007
Ever notice a recurring pattern in your own biology but repress the urge to conclude anything because you can’t explain it mechanistically?
The best thing to do is to first 𝘩𝘢𝘤𝘬, figure out what actually matters for your problem, then find a neat abstraction that explains why it works. You learn something about the hacky solution when you find a nice abstraction for it and vice versa
Replying to @yacineMTB
Abstractions are bad because they frame your ability to think. They reduce your ability to think freely. They're created by other people, people of times past. They don't know any better than you!
When you are framed, you can't be creative and hack things
12
3,742








