Games with reality

Looking backwards, research careers acquire an order they never possessed while they were happening. One problem appears to lead naturally to another, accidents disappear, wrong turns acquire the dignity of “exploration,” and a sequence of decisions made for all sorts of local and occasionally silly reasons begins to resemble a plan. Mine felt much more like a sequence of games in which reality hid something and gave us an imperfect way of looking at it; we tried to understand the rules, occasionally found a loophole, built something, discovered that we had misunderstood part of the problem, and moved on carrying a few useful ideas with us.

The subjects changed rather dramatically — geometry, cameras, inverse problems, machine learning, molecules, proteins — but one question kept returning in different disguises: how much of reality can be recovered from an incomplete observation, and how should we choose what to observe in the first place? This is therefore not my publication list; Google Scholar already performs that thankless task.The Scholar profile and the full list with papers and code. The margin of this page carries only the papers that substantiate a particular step of the story; each citation links to the paper itself. It is an unreliable account of the games we played and what I think I learned from them; wherever the story looks suspiciously coherent, hindsight is doing some editing.

Throw away the coordinates

I began with geometry more or less by accident. My brother Michael and I arrived at Ron Kimmel’s group as fresh graduate students, and the problem on the table was faces: how to recognize a person whose face keeps changing its expression.Treat an expression as a bending of the facial surface and recognize the person from what bending preserves. Bronstein et al. (2005), IJCV; Bronstein et al. (2006), PNAS; Bronstein et al. (2008), Springer. Nearly all of it was done together with my twin brother Michael and with Ron Kimmel, our doctoral advisor. A shape is usually handed to a computer as coordinates, which is convenient until the object bends or changes pose, at which point most of those numbers change although something essential about the object does not. The game was to ask what survives. For non-rigid shapes, distances measured through the surface remain meaningful when Euclidean coordinates do not, and once one starts thinking this way a representation stops being bookkeeping and becomes a claim about which changes in the world matter and which should be ignored.

I did not formulate it in those terms at the time; we were trying to make algorithms work. But geometry taught me to distrust coordinates, and more generally to distrust any representation that arrives disguised as “the data.” Find the right invariants, the right metric, the right latent object, and a problem can suddenly become almost embarrassingly simple; choose badly, and no amount of optimization will rescue it.How to compare a centaur to a horse, how to survive a change of topology, how to treat a shape as a document made of geometric words, and eventually how to let a network learn the correspondence itself: Bronstein et al. (2009), IJCV; Bronstein et al. (2010), IJCV; Bronstein et al. (2011), ACM TOG; Bronstein & Bronstein (2011), IEEE TPAMI; Litman & Bronstein (2014), IEEE TPAMI; Litany et al. (2016), CGF; Litany et al. (2017), ICCV; Halimi et al. (2019), CVPR. The later chapters owe a great deal to Emanuele Rodolà, Or Litany and a small army of students of which I’m a proud academic father. Years later, on a detour into dynamical systems with my student Yonatan Elul, I discovered that the “functional map” we had been so pleased with was, almost word for word, an operator Bernard Koopman had written down a century earlier — humbling, and useful.The detour began with electrocardiograms: sharing dynamical modes across many patients to catch an arrhythmia in a rhythm that still looks normal. Elul et al. (2021), PNAS; Elul et al. (2024), Communications Physics.

Learning to live with shadows

Geometry usually gave us the object and asked for a description of it. Imaging was less generous: there the object is hidden, and what we see is light, sound or a field that has interacted with it, passed through an instrument, been sampled by a detector and corrupted by noise. We receive a shadow and ask what cast it.My very first papers were of this kind, written with Michael Zibulevsky and Yehoshua Zeevi before the geometry began: tomography from ultrasound, and the blind separation and deconvolution of signals mixed by an unknown process. Bronstein et al. (2002), IEEE TMI; Bronstein et al. (2005), IEEE TSP; Bronstein et al. (2005), IEEE TIP; Bronstein et al. (2005), IJIST. The forward direction is usually easy to state; the inverse arrow is where life becomes interesting, because several very different objects can explain the same observation almost equally well.

In the ill-posed problems that interested me, a reconstruction therefore always mixed two kinds of information: what the experiment actually told us and what we were prepared to believe before seeing it. Smoothness, sparsity, geometry, a physical model, a learned image distribution, the allowable conformation of a protein — these are all priors, whether or not we call them that, and hiding one inside an algorithm and announcing that the result is “data-driven” does not make it disappear.Sparsity was the prior of the decade, and much of the work with Guillermo Sapiro and Raja Giryes was about making sparse and low-rank models fast enough to be useful, then about what happens when the prior is learned and the algorithm is unrolled into a network. Sprechmann et al. (2012), ICML; Sprechmann et al. (2015), IEEE TPAMI; Giryes et al. (2018), IEEE TSP; Remez et al. (2018), IEEE TIP; Senouf et al. (2019), MICCAI.

If the inverse problem is ugly, change the experiment

For a while I accepted the natural division of labor: somebody builds the instrument, and the computational people arrive afterwards to recover whatever information it has managed not to destroy. Computational imaging made that division look artificial. If the reconstruction problem is terrible, why should the measurement be treated as sacred? An aperture can be coded, illumination structured, a sensor made to mix variables deliberately, and optics can produce an image that looks worse to a human eye but is much easier for an algorithm to decode.A phase mask in the lens that encodes depth into color fringes and lets a network read it back from a single photograph; a sensor of one-bit pixels; a camera pipeline learned end to end. Haim et al. (2015), Optics Express; Haim et al. (2018), IEEE TCI; Remez et al. (2015), arXiv; Schwartz et al. (2018), IEEE TIP. The optics came from Emanuel Marom’s group.

The instrument and the algorithm then cease to be two independent objects and become one information-processing system. I still find the idea beautiful because it changes the rules of the game rather than playing the same game better. Instead of asking, given these measurements, what can I reconstruct?, one asks, which measurements should I create if I already know what I will want to reconstruct from them?The same move in medical imaging: learn where an MRI scanner should sample k-space, subject to what its gradients can physically do, jointly with the network that reconstructs the image; learn how an ultrasound probe should beamform. Weiss et al. (2021), MELBA; Alush-Aben et al. (2020), MICCAI; Vedula et al. (2019), MIDL. A lecture on this, Learning to see, is on the talks page. It looks like a small rearrangement of words, but it turns sensing into design.

Reality joins the review process

Some ideas escape from papers, and this changes the questions one is forced to ask. My work on three-dimensional sensing led me into startups and, after the acquisition of Invision, into Intel and RealSense.The lab prototypes: Giryes et al. (2008), EDERS; Rubinstein et al. (2009), ICCV Workshops. Invision, founded in 2009 with Michael, Ronny Kimmel and Sagi Ben Moshe, became part of Intel in 2012; the coded-light camera we built there shipped as Intel RealSense, and the rare full account of a mass-produced computational camera is Zabatani et al. (2019), IEEE TPAMI. When I worked on Invision I was convinced that 3D sensing would change vision and become indispensable. Our cameras were indeed manufactured in many millions of units and became commonplace in industry and in research labs. Yet today, seeing how well vision works with plain 2D images, I am no longer sure we need 3D sensing quite so much; whether one needs depth for face authentication at all is asked in Livne et al. (2020), 3DV. I learned a great deal there, although very little of it fits naturally into the standard academic account of research, and many of the things I am most proud of were never published. A paper is allowed many conveniences: it can choose its dataset, state its assumptions, reject a broken sensor, recalibrate the camera, restart the experiment, present “representative results” (the only ones that worked), and move the awkward corner cases into “future work.” A product has fewer philosophical privileges. It encounters manufacturing tolerances, temperature, power budgets, components that cost twelve cents too much, scratched optics, strange rooms, strange users, bad calibration, worse calibration, and combinations of circumstances nobody had considered dignified enough to put in the model.

A product is an unusually cruel peer reviewer, and I mean that affectionately. Robustness stopped being a layer one adds around a successful mathematical core and became part of the definition of success. Entrepreneurship taught me something related: between “the equation works,” “the prototype works,” and “this can continue to exist in the world as a product or a company” lie several rather different sciences, most of which are badly documented and at least one of which consists almost entirely of humans.After Invision came video search, IVF embryo selection and, most recently, a machine-learning hedge fund; the essay The Money-Making Black Box is what the last of these taught me about markets as adversaries. Some of the companies I helped start worked very well, some did not, and the latter were often the more efficient teachers. Reality at scale has a considerably larger adversarial test set than any benchmark.

The geometry escaped too, by a route I had not planned. Some ten years ago I found myself consulting for a company whose research was led by Asi Elad, the very person whose canonical forms had started me off a decade earlier; the paths of life seem to have a fixed point. They were guiding a catheter through a beating heart by reading voltages in a wildly distorted electrical field, and the only rigid thing in the problem was that four electrodes on the catheter tip kept their mutual distances. Learning the anatomy from those four numbers was multidimensional scaling in another disguise. It worked to the precision of a CT scan, the company was later acquired by Philips, and the pre-clinical trials on pigs turned me into a vegetarian.The longer version of this story, ants included, is in a speech I gave for Freddy Bruckstein.

When the prior learns

Machine learning entered this story, for me, as a remarkably powerful new way of representing priors: instead of writing down by hand what an image, a shape or a signal ought to look like, one could learn regularities from examples.Learned similarity and hashing across modalities, an early attempt to say why a randomly initialized deep network already preserves distances, and synthesizing examples when only a handful exist: Masci et al. (2014), IEEE TPAMI; Giryes et al. (2016), IEEE TSP; Schwartz et al. (2018), NeurIPS. Making the resulting networks small enough to run on real hardware became a research program of its own with Avi Mendelson’s group: Baskin et al. (2020), ACM TOCS; Shkolnik et al. (2020), NeurIPS. This was intoxicating, and it was also dangerous, because a learned model can solve problems that resist clean analytic description while remaining very difficult to interrogate about why it works.Freddy Bruckstein, my academic grandfather, used to say that neural networks are the second best solution to any problem. The first is, of course, to understand what you are doing.

The work on adversarial examples made that tension unusually visible. Tiny perturbations, almost meaningless to a human observer, could make a highly accurate model change its decision completely.Mariani et al. (2020), CGF; Nemcovsky et al. (2022), ACCV; Blau et al. (2022), arXiv. The middle one is a patch one can print and stick on a wall to confuse a robot’s sense of where it is. The immediate reaction was to call this a pathology of neural networks, which it partly is; the more interesting possibility was that the network had learned a somewhat different game from the one we thought we were teaching it. Which variations should matter? Which invariances are semantic and which merely habits of the training distribution? These were the old questions from geometry returning in a much messier world,Literally so, in places: hearing the shape of a graph from its Laplacian spectrum, aligning graphs the way we once aligned surfaces, and asking a model to say how uncertain it is about a vector-valued answer. Tsitsulin et al. (2018), KDD; Hermanns et al. (2022), APWeb-WAIM; Rosenberg et al. (2023), ICML. and the trade they came with — richer priors, much harder to inspect — is often worth making, provided one remembers what was traded.

Molecules make the prior physical

Then the subject changed again, and this time the accident had a name. Some six years ago I met Ailie Marx, a structural biologist, and I owe to her the awe of the world of the tiny. The last time I had looked at a cell was in high school, where it was drawn as a boringly empty bag with some stuff floating here and there. Discovering that it is packed wall to wall with molecules, each with a shape and a job, was one of the first-discovery moments of my adult life.What a cell actually looks like inside: a cross-section of Escherichia coli painted by David S. Goodsell. Illustration by David S. Goodsell, RCSB Protein Data Bank, doi:10.2210/rcsb_pdb/goodsell-gallery-028, CC BY 4.0. It was also a late one: both of my grandparents were organic chemists, yet my own chemistry had ended in my teens with Asimov’s The World of Carbon.When my first protein paper came out, one of my doctoral students remarked that I had finally moved from RealSense to real science.

Molecules also changed the emotional character of the problem. An image can be visually plausible and still be useful; a molecule has less freedom. Atoms occupy space, bonds have geometry, energies matter, symmetries matter, and proteins are both exquisitely structured and relentlessly dynamic. A beautiful hallucination can still be physically impossible.Designing polycyclic aromatic molecules to order with guided diffusion, and asking an interpretable network what it had learned about them — a class of molecules one certainly does not want to ingest — with Renana Gershoni-Poranne, who is also an amazing soprano; some of our musical adventures are on the music page. Weiss et al. (2023), Nature Computational Science; Weiss et al. (2023), J. Org. Chem.; Davidson et al. (2025), J. Chem. Phys. Generative modeling became much more interesting to me once the model was no longer imitating a distribution of examples but trying to live in a world whose rules existed long before the training set.

Structural biology is also full of wonderfully indirect measurements. X-ray crystallography, NMR, EPR, SAXS and cryo-EM do not hand us a structure; they reveal partial, noisy projections of it — a distance here, an orientation there, an image integrated through depth, an ensemble-averaged observable.The way in, with Ailie Marx, was statistical rather than generative: the Protein Data Bank read as a corpus, in which the backbone geometry turned out to remember which synonymous codon had encoded it, amino-acid pairs behaved like dominos, and a good many “alternate conformations” in crystal structures were real. Rosenberg et al. (2022), Nature Communications; Rosenberg et al. (2023), PNAS; Bronstein & Marx (2023), Scientific Reports; Rosenberg et al. (2024), Scientific Data; Rosenberg et al. (2024), bioRxiv. Then AlphaFold changed the balance of the game: suddenly we had an extraordinarily strong prior over protein structure, a machine that knows, in a statistical sense, an astonishing amount about what proteins tend to look like. But experiments know things the model does not. They know something about the molecule actually present in the tube, in a particular environment, state, mixture or motion, and occasionally they contradict the answer that looks most plausible to the model. I find that contradiction much more interesting than agreement: it is where a predictive model stops being an oracle and becomes a scientific object that can itself be tested.

Two imperfect witnesses

Much of what interests me now is a negotiation between two imperfect witnesses. One is the experiment. It is tied to physical reality, which is an enormous advantage, but it is noisy, partial and filtered through an instrument, and may constrain only a few directions in an enormous space of possible structures. The other is the model. It carries a huge amount of knowledge learned from previous structures and sequences, but it can be confidently wrong, biased by its training distribution, or blind to precisely the unusual state that made the experiment worth doing. Neither witness should dominate simply because it speaks more fluently.

This is how I think about our work on experiment-guided AlphaFold.Maddipatla et al. (2025), ICML; Maddipatla et al. (2026), Nature Biotechnology; Maddipatla et al. (2026), ICML; Maddipatla et al. (2024), NeurIPS MLSB. The Oberwolfach lecture of 2025 is the spoken version of this section. Rather than using experimental data only at the end, as a check on a predicted structure, we put the measurement inside the inference loop and ask the generative model for structures that remain plausible while explaining what was observed. So far the evidence has been NMR restraints and ensemble observables, and density maps from crystallography and cryo-EM; asking an atomic model to explain raw cryo-EM particle images through the imaging physics, rather than a map already reconstructed from them, is where we are heading rather than where we have been. The interesting problem begins when the two disagree, because then several possibilities open at once: the experiment may simply be too noisy, the model may be wrong, the assumed forward model may be wrong, or the molecule may be doing something neither side expected. Those are not merely optimization failures. Very often they are where the science begins.

Closing the loop

There is one more step, and it brings me back to computational imaging. If we have a powerful model of what the world may look like, and a differentiable description of how an experiment produces data, why optimize only the interpretation of a fixed experiment? Which illumination, which receiver, which excitation pattern, which time point, which observable? Where is the next photon most valuable? Which measurement would most sharply distinguish the hypotheses that still survive?Earlier rehearsals of the same question with less expensive toys: where to put the antennas of a radar, how long to expose each frame of a shaking camera, how to shape the pump beam of a nonlinear crystal so that it emits the entangled photons one wants, how a drone might shape its own rotor noise to hear where it is, and how to reconstruct ultrafast events from multiplexed ptychography. Weiss et al. (2021), IEEE MLSP; Dahary et al. (2021), CVPR; Rozenberg et al. (2022), Optica; Serussi et al. (2024), IROS; Wengrowicz et al. (2024), Optics Express.

This is the old coded-aperture question in a larger disguise, with a protein for a scene and a generative model for a regularizer. It is also the part of “AI for science” that interests me most: not an oracle that has read more papers, and certainly not a chatbot attached to a laboratory instrument, but computation inside the epistemic loop — helping decide what to measure, interpreting what came back, proposing what remains possible, and being forced to revise when the next measurement disagrees. The model proposes; reality disposes. Twenty-something years later, I appear to be playing much the same game with more expensive toys.

What I think the games were about

Retrospectives are dangerous because they discover themes that may not have existed at the time, so I will keep the list short. Representation matters: many problems are decided before the algorithm starts. Measurements are designed rather than given. Inference contains belief, and the honest question is which prior one is willing to use and how strongly reality is allowed to contradict it. And building things is a particularly unforgiving form of epistemology, because a theorem, a prototype, a product and a house all discover one’s mistakes in different ways.Hindsight is doing some work here.

There is also beauty, which I trust as a guide more than I should and as evidence not at all. Elegant explanations deserve attention; experiments retain veto power. Especially in biology, I often wonder who the heck designed this.

The next version of the game

Structural biology has already crossed an interesting threshold. AlphaFold is, for most practical purposes, a black box. Very few of its users can explain in any meaningful mechanistic sense why it produces one structure rather than another. Yet structural biologists have learned to work with it remarkably comfortably. It makes predictions; those predictions can be compared with experiments; they can be confirmed, contradicted, refined, or occasionally shown to be spectacularly wrong. In that rather pragmatic sense, the black box has earned a place in the scientific process because it says things about the world that the world can answer.The same pragmatism, applied to a far less forgiving domain, is the subject of The Money-Making Black Box.

But there is a historical asymmetry hidden in the way we currently use it. AlphaFold and related models have been trained largely on scientific data produced by humans, through experiments and representations designed for humans. We collected diffraction patterns, spectra, micrographs and other measurements, transformed them through elaborate reconstruction pipelines into structures and other intelligible objects, inspected them, argued about them, deposited them in databases, and only then used those accumulated products to train machines. There is no fundamental reason why this intermediate human-readable representation must remain there.

Computational imaging taught me this lesson in a much simpler setting. Once reconstruction is performed by a computer, the signal arriving at the detector does not have to look like an image. It does not have to be interpretable by a human observer at all. One can deliberately distort, multiplex or encode the optical measurement if doing so preserves the information that matters to the reconstruction algorithm. The detector is no longer making a picture for a human beholder. It is producing evidence for a machine.

Something analogous may now be possible in experimental science. As laboratories become increasingly automated, we can ask not merely how to train better models from the measurements scientists already know how to make, but what experiment a machine would choose if the measurement were intended for the machine in the first place. Perhaps the optimal experiment will produce data that no human would ever want to look at directly. That need not be a defect. It may be the point. This is one of the directions we are actively exploring now: closing the loop between learned models and physical experiments, so that the machine does not merely interpret an experiment designed beforehand but participates in deciding what should be measured next, and the answer it receives changes its next question.

The technical questions are difficult enough. The methodological and epistemological ones may be harder. Science has traditionally valued theories not only because they predict observations but because they make the world intelligible. A good theory compresses many observations into a small number of concepts that a human mind can manipulate: forces, fields, genes, energy landscapes, symmetries. What happens when the most successful predictive object is a model whose internal concepts we cannot translate into ours? Is falsifiability enough? If a black box repeatedly makes precise predictions and survives increasingly hostile experiments, in what sense is it different from a scientific theory — and in what sense is it profoundly not one? And if we eventually allow such a model not only to interpret observations but to design the experiments that test its own predictions, the question becomes sharper. We may find ourselves doing science in a language that nature and the machine share more fluently than we do.

That prospect is both exhilarating and slightly unsettling. It does not, I think, make the human scientist obsolete. But it may change what the scientist is for. Perhaps our role moves upward in the hierarchy: from manually interpreting every measurement to deciding which questions are worth asking, which distinctions matter, what constitutes an explanation, which failures are interesting, and when a machine that predicts correctly still has not told us what we wanted to know.

For most of my career I have been interested in recovering hidden reality from incomplete observations, and then in designing better observations. The next version of the game may be stranger: designing a dialogue between experiment and machine whose intermediate language we ourselves do not fully speak. That raises questions that are methodological, epistemological and perhaps eventually ontological. Must understanding belong to a human mind? Can prediction without intelligibility count as explanation? And if part of science becomes an exchange between machines and the physical world, what place is left for us? I do not know the answers. It is difficult to imagine a more interesting time to be asking the questions.