Art history is a lot like archaeology — we here in the present day get artifacts and records, but the gaps between them are enormous, and the questions that they beg loom large. Historians need to be able to investigate and interpret, to unpack the meanings and the methods of a given work of art — but even for the best, the act of reconstruction is a trying test. Can we program computers to decipher the backstory of a painting — analyzing light and shadow to guess at how a piece was made? And, even more ambitiously, can AI learn to see and tell the stories rendered in an image’s symbolic content? Recent innovations yield surprising insights and suggest a cyborg future for art scholarship, in which we teach machines to not just recognize a set of objects, but to grok their context and relationships — shining light on messages and narratives once lost to time, and deepening our study of the world of signs.
Welcome to COMPLEXITY, the official podcast of the Santa Fe Institute. I’m your host, Michael Garfield, and every other week we’ll bring you with us for far-ranging conversations with our worldwide network of rigorous researchers developing new frameworks to explain the deepest mysteries of the universe.
This week we speak with David Stork, who has held full-time and visiting faculty positions in Physics, Mathematics, Computer Science, Electrical Engineering, Statistics, Neuroscience, Psychology, and Art and Art History variously at Wellesley and Swarthmore Colleges and Clark, Boston, and Stanford Universities…as well as holding corporate positions as Chief Scientist at Ricoh Innovations and Fellow at Rambus, Inc. We talk about the what happens when computers look at art — and the implications for art history and connoisseurship.
If you value our research and communication efforts, please consider making a donation at santafe.edu/give — and/or rating and reviewing us at Apple Podcasts. You can find numerous other ways to engage with us at santafe.edu/engage. Thank you for listening!
Join our Facebook discussion group to meet like minds and talk about each episode.
Podcast theme music by Mitch Mignano.
Follow us on social media:
Twitter • YouTube • Facebook • Instagram • LinkedIn
Go deeper with these additional resources:
David’s bio at International Academy, Research, and Industry Association
David’s Google Scholar Page
David’s SFI Seminar
David’s talk at The Frick Collection, “Rigorous Technical Image Analysis of Fine Art: Toward a Computer Connoisseurship”
This is a machine-generated transcript provided by podscribe.ai — with editing help from Aaron Leventman. If you would like to volunteer to help us edit transcripts, please email michaelgarfield[at]santafe[dot]edu. Thanks and enjoy!
David Stork (0s):
I look at Vanitas painting by Hendrik Van Steenwijk where you just have these objects on the table. And, you know, there's a skull and their books and there's an oil lamp and so forth. But everyone in the Dutch golden age who would have seen that painting really wouldn't be so interested in the objects and the surface level of what's being depicted. But instead the message, the moral, the message of Vanitas. Do not concern yourself with the pleasures and diversions of this life here, but instead prepare yourself lead a sober life for the eternal life to come.
David Stork (41s):
And there are many ways in which a painting can convey that message. And AI is barely touching that class of problems.
Michael Garfield (1m 17s):
Art History is a lot like archeology. We here in the present day get artifacts and records, but the gaps between them are enormous. And the questions that they beg loom large. Historians need to be able to investigate and interpret to unpack the meanings and the methods of a given work of art. But even for the best, the act of reconstruction is a trying test. Can we program computers to decipher the backstory of a painting, analyzing light and shadow to guess at how a piece was made and even more ambitiously can AI learn to see and tell the story is rendered in an images, symbolic content. Recent innovations, yield, surprising insights, and suggest a cyborg future for art scholarship in which we teach machines to not just recognize a set of objects, but to grok their context and relationships shining light on messages and narratives.
Michael Garfield (2m 14s):
One's lost to time and deepening our study of the world of signs. Welcome to complexity, the official podcast of the Santa Fe Institute. I'm your host, Michael Garfield, and every other week, we'll bring you with us. Far-ranging conversations with our worldwide network of rigorous researchers, developing new frameworks to explain the deepest mysteries of the universe. This week we speak with David G. Stork who has held full-time and visiting faculty positions in physics, mathematics, computer science, electrical engineering, statistics, neuroscience, psychology, and art and art history variously at Wellesley and Swarthmore colleges, as well as holding corporate positions as chief scientist at Rico Innovations and fellow at Rambus, Inc.
Michael Garfield (3m 2s):
We talk about what happens when computers look at art and the implications for art history and connoisseurship. If you value our research and communication efforts, please consider making a email@example.com/podcastgive and or reading and reviewing us at Apple podcasts. You can find numerous other ways to engage with us at Santafe.edu/engage. Thank you for listening. David Stork. It's a pleasure to have you on complexity podcast.
David Stork (3m 33s):
It's a great pleasure to be here.
Michael Garfield (3m 36s):
This is a very interesting topic, and I would like to start on the ground. Let's talk about your background because you started as a physics major.
David Stork (3m 50s):
Sure. How did I come to start using computers to analyze paintings?
Michael Garfield (3m 53s):
Yeah. Like why was this an interest to you? What set you off on this path?
David Stork (3m 58s):
I come from a family steeped in the arts. My great-grandfather was court painter to crown Prince Rudolph, Joe Franz Josef, rather in Austria and his paintings hang in, museums in Austria and elsewhere. I won't go through all the arts folk in my family except to go down to my younger sister, Kathleen, who was chief calligrapher in the White House under Bill Clinton. So I was raised steeped in the arts and early on though, I got very interested in science, but didn't see much of a connection between the two and my science pure science career was doing very well.
David Stork (4m 37s):
But while a graduate student, I wrote a book and then later taught some courses on optics and perception and vision and so forth. And the connection and the application to art was really fascinating to me and very fascinating to the students. So I had that sort of background. At MIT I was indeed a physics major, but my senior thesis under Edwin Land, the founder of the Polaroid corporation was on human visual perception in Mondrian images, things that involve color perception and things like this. And then in my PhD, I worked on human visual perception, very mathematical and technical and solving wave equations in the retina and things like this.
David Stork (5m 22s):
But the connection to the visual arts had always sort of been in the background. I grew up in Chevy, chase outside Washington DC, and hence had great access to the national gallery and all the great museums in Washington. And frankly, you know, those docent tours when you're had a museum and some docent is leading around 10 people in 0.1 of those absolutely changed my life. I don't know if that woman is still alive, but American art at mid century on an American abstract expressionism, Jackson Pollack, and Mark Rothko and Arshile Gorky and so on.
David Stork (6m 6s):
And I saw so much more, there was so much going on in these artworks that I had no idea what's going on before then. And so while at MIT, I cross registered for art history courses at Wellesley College, which was a very, very good art history department. And I got a sense of the depth, the subtlety, the problems that arise in art history. And I just got fascinated and to skip over a few decades. At the year 2001, I was invited to a big conference in New York to analyze artists. David Hockney's very controversial theory that Renaissance painters secretly used optical devices during the execution of their works. Spoiler alert.
David Stork (6m 54s):
They did not certainly not that early, but my work involved a lot of technical analysis of perspective and lighting and contours and things like this. And they really showed that computers can outperform even human experts in some very restricted classes of problems. And I got very, very interested in this and bit by bit I worked with more and more conservators and curators and art historians and so forth at major museums and things like this. And I started developing these techniques and taught the first courses on that at Stanford.
David Stork (7m 34s):
And I've lectured all around the world and now it's really blossomed and I'm working now at 700 as of this morning, 731 pages on my book on the subject. So that's how I got here.
Michael Garfield (7m 49s):
So when you spoke at SFI earlier this week, I was taken by the way that your approach to reconstructing the methodologies used by painters looks like kind of a crime scene investigation. You're trying to figure out, like you said, what, what were the tools, what were the weapons? How did they build the pyramids? It's it's, it's, it's a form of archeology kind of.
David Stork (8m 14s):
Oh yeah. Art scholarship, art history. At SFI I only had a chance to talk about one small portion of my total work, which is involved with what kind of lighting did they use? Was it consistent? How can we use this, these lighting analysis tools to identify the, as we say, hands, the number of artists who work on the given, given work, we're looking at perspective for perspective and consistencies and things like this, but I've also worked on authentication, including works like Jackson Pollock by image. Now a full authentication protocol involves looking at pigments, what pigments were available, what pigments were used, the support, carbon dating and a whole host of things.
David Stork (8m 59s):
My research focuses on what you can do with the image. What does a connoisseur do when he, or she analyzes a painting or drawing? And those are the kinds of techniques that I'm trying to embody and implement in algorithms.
Michael Garfield (9m 18s):
So I'd like to invite you to talk a little bit about the work that you've done with Vermeer, Girl with a Pearl Earring. This question of, we look at a painting and you said, did they actually use a living subject? Were they looking at something and painting from life or where they painting from the imagination.
David Stork (9m 40s):
Or we use optical tools Vermeer is one of the most wonderful artists, but he also brings together some of the highest technical questions you ever get. In fact, your audience can't see it, but I'm holding up to you. This issue of heritage science, the girl in the spotlight, the technical re-examination of Ramirez girl, the Prolia, it's an entire small book, just on the technical analysis of pigments and paintings and the glazing techniques and the supports and things like this. You could spend a lifetime just on Vermeer. So yes, what I spoke about at SFI was lighting analysis. So we use model dependent and model independent methods. Model independent methods are where you don't need to make any assumptions about the three dimensional form in this case of her face. Model dependent ones you do.
David Stork (10m 33s):
So we looked at cast shadows, the reflections off her eye off the Pearl. One of the most powerful techniques, which actually comes from forensic photography is called the occluding contour algorithm, which is looking at the pattern of light on the outside of an object that's being illuminated and inferring from that pattern of lightness where the light must be coming from. And the question of where was the light in Vermeer studio, actually, that alone isn't very interesting But what is interesting and what we showed really definitively is the astounding agreement, the astounding commensurateness of the estimates from all these different sources.
David Stork (11m 22s):
So that shows certainly that there was a girl present in his studio in Delft when he made this, but it also shows how accurate he was. And there were several other artists like Georges de Latour that we've worked on, whose nocturnes, these dark nighttime scenes are renowned for their portrayal of lighting throughout the scene. We show in ways that no connoisseurs, I frankly can really elucidate the consistencies and the inconsistencies in the other artworks through these computer techniques. And it turns out humans actually are not very good at inferring lighting, seeing lighting differences.
David Stork (12m 8s):
In other talks, I show a picture of Brad Pitt and Angelina Jolie, which were on the cover of, I think it was the Star Magazine. Anyway, that was photo-shopped, we'd say tampered. And the way you can determine that is by this occluding contour algorithm, looking at the side of Brad's face and Angelina's face and inferring that the lighting is very, very different, but those fakes get into the popular discourse very easily, because we are so bad at noticing these areas, but the computer techniques are absolutely superb. And so these have many applications throughout art scholarship, and that's, that's the kind of thing I'm working on.
Michael Garfield (12m 51s):
So can you talk a little bit about how you build light field models and then how you use that to analyze paintings where you have a question about whether this was a painting that was done all at once or whether this was a painting where...
David Stork (13m 9s):
Background was painted and then the subject was painted later. Sure.
Michael Garfield (13m 13s):
Right. You can identify the painter was actually like walking around, trying to recreate a light source.
David Stork (13m 20s):
Sort of. We applied our techniques to a contemporary painter, Garth Herrick, who lives in Philadelphia, whom I've learned about when I was a beginning assistant professor at Swarthmore College many years ago. And he works from photographs. He does a lot of official portraits of senators and congressmen and mayors and such like and so forth. And it's difficult to get famous people to stand in your studio for weeks on end. So he goes out and takes a photograph of the figure, but he wants this figure to be in front of a certain background.
David Stork (14m 1s):
And that might not be generally, it's not his office. So he goes somewhere else and takes another photograph, goes back into his studio and is now painting using two, we call them reference, two photographs. And when he's painting the background, he looks at one photograph. And when he's painting the figure he paid, he looks at the other photograph. There is no guarantee that the lighting in those two photographs is commensurate from the same direction, the same distribution and so forth, or that he can detect that as I've just mentioned and then fix or make it commensurate. So we've looked at several of his paintings and ask the question, can the computer tell that the lighting in the background is different from that on the figure?
David Stork (14m 48s):
And the answer is yes. And in certain cases, and this works even in complex, we call them light fields, the pattern of light coming from many different directions. So like coming through a window from a candle reflected off the floor, whatever it might be, it can be arbitrarily complex. And that occluding contour algorithm I mentioned can be generalized to infer, not just the primary direction the light's coming, but instead the entire light field. We describe it with five numbers using a spherical basis, function, set the mathematical details I didn't go into, but basically it's a way of saying, ah, the lights has this pattern on this figure, but this other pattern on someone else and thereby tell whether they were done under the same studio conditions and so forth.
David Stork (15m 43s):
And so for Garth's paintings, we did human on my faithless arm, which he painted in two episodes or campaigns we call them, the painted the background. And then nine months later, he moved to a different, he had moved to a different studio, different time of year, different position of the window. He was painting in a mirror. And so his self portrait had somewhat different light field on him than the background. And it's difficult to see by eye. I've shown, not just the digital image, but the actual painting in my aunt to art scholars. And it's hard. You really can't tell, but the computer finds it immediately and definitively.
David Stork (16m 27s):
So that's how we use lighting analysis for that kind of question.
Michael Garfield (16m 31s):
Yeah. There's the, the cast shadow analysis also.
David Stork (16m 34s):
Yeah. The cast shadow is one of the simplest. You actually don't need computers to do the basics of cache shadow analysis. You just take a point on an occluder, like the tip of girl, the Prolia earrings nose and its corresponding cash shadow point draw a straight line. But if you have many cast shadows and you want to infer the most likely position of a light source for all that cash shadow evidence, you use technique called maximum likelihood estimation. It's a standard statistical technique, but applied in a special way for two dimensional artworks to say, ah, given all this lighting information over Georges de Latour's Christ in the Carpenter's Studio.
David Stork (17m 18s):
For instance, the most likely positioned for the illuminant is here and not here. We did that analysis in order to test David Hockney's claim that that painting was done with the light shorts quote outside the frame of the picture. It absolutely positively bet your life on it was not.
Michael Garfield (17m 40s):
So, as you alluded to earlier, there's this, this other dimension, which is the use of tools, of mirrors, lenses, projection techniques. You give an example of the Van Eyck Arnolfini painting. Oh yes. And in your talk, you presented a rather thorough run through of how you actually like recreated the entire crime scene of this thing.
David Stork (18m 12s):
It's not a crime. It is the most sublime and important paintings of the Western Canon first double full length, double portrait in the West. When the earliest portraits in oil paint, the first painting in the West to show an action, something occurring inside a domicile, inside a home. No this painting, it's not a crime. It's, it's, it's a masterpiece,
Michael Garfield (18m 35s):
But it is. I mean, I guess, you know, historically artists were not really incented to show their work, right? And so the point is that it's a crime that we don't know how it was done.
David Stork (18m 47s):
Yes, it's true. Some artists had secret techniques for especially mixing pigments that they would keep secret, but often they would announce that they had secrets. I've got a secret method like the Venetian glass workers or the prototypical example, the ways in which they heated sand and so forth and made their glassware was a big secret. And there were penalties for revealing these secrets, but they were announced as secrets. But the question about whether certain artists used optical aids is really, really quite fascinating.
David Stork (19m 28s):
And first of all, we know some did, Canaletto and Thomas Akins, the great realist painter from the Philadelphia area. But I find it really frustrating when you start hearing this, the first thing that the scholars or casual folk want to talk about is whether it's cheating or not. I find that so profoundly, intellectually lazy. The real question is first, did they do it? Yes or no. And after that, it's fine to talk about what it means, whether it would have been cheating, how it affects the transition of art and the development of art and so forth.
David Stork (20m 10s):
And so we've gone through, we've spent several years on, I don't know, maybe 20 papers by now on different aspects of this in order to answer these questions. And I think we've done it. And the scholarly community is now I think universally on the side that someone like beyond what Jan Eyck did not use any optical aids directly during the execution of his works. There's a modified fall back position. Well maybe he saw an optical image and that indirectly influenced him. I don't know how you test that. We have no documentary. There's so many other sources for this rise in the realism of art around the time of Van Eyck, including the invention of oil paints.
David Stork (20m 59s):
That it's very hard to address that modified version. But the direct case I think is completely solved.
Michael Garfield (21m 8s):
And part of solving this was you took a detour into the mirror that appears in this painting.
David Stork (21m 16s):
Well, absolutely. Yes. The convex mirror at the center of the Arnel Feeney masterpieces, most scholars would claim the most famous mirror depicted in all of art. It's right at the center. It shows the back of Arnel Feeney and his wife and the two witnesses to the wedding coming through the doorway. It has metaphorical significance because it's convicts bowing out. And so it gives a wide angle omniscient view of the entire Tableau as if God is watching. And, Oh gosh, I could talk for hours on this. But I mean, one of the most famous paintings, one of the greatest paintings is Las Meninas by Diego Velazquez and the Arnel Feeney portrait hung in the Alcazar palace in Madrid.
David Stork (22m 2s):
And so he certainly saw it and that influenced Valasquez to put in a plain mirror in the middle of the Las Meninas. But the question we address, the technical question we addressed was answering the explicit claim of David Hockney and his scientific colleague, Charles Falco, that, that convex me or turned around to be a concave mirror, could have been used during the execution of the work of the painting itself. And there are a dozen reasons why that cannot be true. One is these mirrors were not perfectly circular. They were blown glass and would have imperfections.
David Stork (22m 43s):
So even if everything else were perfect, the image that gets projected we've shown through computer ray tracing would have been far too blurry. Second of all, those convex mirrors were coated with metals to be reflective like mirrors, but then coated with tar. So it's to seal it in. So there would be no reflection whatsoever. And what I showed in my SFI talk was that the focal length would have been far, far, far too short to comport with the geometry in the actual painting. And there are other rebuttals, but, but those are the main ones. So the technique we use, we've actually done it three different ways.
David Stork (23m 23s):
The first came from Antonio Criminisi in Cambridge, England, but done to others, which is basically modeled the reflection off of a convex sphere and undo the mathematics. If you will find what radius of curvature, what bulginess of the mirror, comports with the image that we see reflected in that painting. And that gives us the radius of curvature. And it turns out the focal length of such a mirror would be one half that radius of curvature. And it cannot comport with the geometry in the painting as a whole.
David Stork (24m 4s):
And this is a real technical detail. Maybe it's too much for your audience, but you might say, Oh, well maybe Van Eyck did a piecemeal patchwork projection. He projected this part or the window, and then trace that then moved over and did some... That can't explain the evidence either because there are all these nice, long, straight lines on the floor, on the window sill, along the back wall, the ceiling, the bed, and so forth, which would have curvatures and breaks as well, if you did that. So, no.
Michael Garfield (24m 42s):
Yeah. So one of the questions that this seems to be answering is specifically the reconstruction of the timeline of the innovation of painting technology.
David Stork (24m 50s):
Yeah, that's a very, very important question. Surely technology has influenced the development of art in many ways. I mean, photography alone, that the, you get the snapshot framing in paintings in France after the invention of photography and a whole host of things. One of my favorites here at Stanford, I can almost see it from my window here is the very famous Eadweard Muybridge sequence of the running horse. The, the famous debate from Governor Stanford, whether all four legs of a horse left the ground simultaneously and paintings before then let's see Jerome and several others show them with the, the, the front legs forward and the back legs, backward flying off the ground.
David Stork (25m 43s):
And everyone thought those look very natural and dynamic. And yeah, he's got that right. After the Muybridge movie, proto movie, showing that yes, all legs are off simultaneously, but it's when the legs are underneath the belly of the horse together, the front legs are back. The back legs are forward. Once you see that the previous versions practically look silly. And so that's just one of hundreds of ways in which technology has changed how we see and thus how artists create.
Michael Garfield (26m 25s):
So I, that seems like a good place to peg into the other half of this, which is about semiotics and, and, and meaning and training machines to actually interpret art the way that a human art kind of sewer can interpret art. You have a piece on this computational identification of significant actors in paintings through symbols and attributes. And I just want to say it really is. It's a steep learning curve, but to get into this kind of appreciation, because especially in the Western Canon, it's like, it's just hundreds of white guys with beards, right.
Michael Garfield (27m 11s):
And so how do you tell them apart? And you know, it's not like we're operating from a photograph. It's a canonized figure, but there isn't a bedrock of empirical GroundTruth face to work from. So what are the problems for machine vision created by this that are not present? You know, when you've got Google photos trying to do facial recognition?
David Stork (27m 36s):
Yeah. Yeah. Well, I mean, that brings up the broader question of what problems does art pose that traditional AI is not addressing. And this is one of them,. To me, the most important, and frankly, it was sort of an epiphany. I can go back to that, but most photographs are taken to document something. Here I am on the beach. Here's the bowl of spaghetti in front of me, look at this, everybody, Oh, isn't my dog cute. And things like this. Art really goes speaking in the broadest of terms. There's a great deal of art that was created in order to convey a story, a moral or a message and have a meaning that the artist has an intent.
David Stork (28m 24s):
He or she is trying to instruct you in some way or inform you. And most of in the history of Western art, most of that has come from religious art. These are stories from the Bible. These are morals. This is Adam and Eve, you know, Abraham and Isaac and, you know, descent from the cross and things like this. All of these are screaming with meaning. And as you will recall, from my SFI talk, I look at Vanitas painting by Hendrik Van Steenwijk where you just have these objects on the table. And you know, there's a skull and their books and there's a oil lamp and so forth.
David Stork (29m 4s):
But everyone in the Dutch golden age who would have seen that painting really wouldn't be so interested in the objects, the surface level of what's being depicted. But instead the message, the moral, the message of Vaniitas. Do not concern yourself with the pleasures and diversions of this life here, but instead prepare yourself lead a sober life for the eternal life to come. And there are many ways in which a painting can convey that message. And AI is barely touching that class of problems, but there's a whole branch of computer vision called semantic image analysis, but semantic for them means, Oh, there's a person on a man on a horse, a woman walking beside and a tree in the background and a river .
David Stork (29m 59s):
To them that's semantic understanding the meaning to me, that's nowhere near what is going on in the visual arts, which is one step further. You're really getting to the intention of the artist. We call them authors in this case, whereas input photography, you don't. And so this is the classic problems that I'm working on with colleagues in Cambridge University, in England and elsewhere around the world. And the paper you're referring to was just a first stab at that. If you take for Akio's a baptism of Christ, very famous painting in the Fitzy gallery in Florence. It shows Christ standing in the middle Saint John the Baptist pouring water over his head.
David Stork (30m 47s):
But how is it that the audience recognizes who these people are as a step towards understanding the meaning of Christ, you know, is washing your sin, taking your sins and so forth. And in religious art, we have something that's really very interesting to me. I called attributes for those of us who work in pattern classification. We have a different word for apt use for the word attribute, but in religious iconography, an attribute is something that an object that indicates the identity of someone. So Saint Catherine is a wheel because she was martyred on a wheel.
David Stork (31m 31s):
Christ might have a cross or a lamb. St. John the Baptist has a cross, and there are literally thousands of saints. And some of them have very unusual attributes, including one I had to look it up. Silk gloves. There's a Saint whose attribute is silk gloves. Anyway, the religious authorities, especially in the, you know, Renaissance and later would include these as patrons would have the artists portray these in order to instruct the illiterate parishioners so that, you know, an illiterate person in a cathedral in France would look at this and say, aha, I see that cross or the shell or whatever it might be.
David Stork (32m 20s):
And know, not only the actor, the, the Saint, but also the story. And so that's the kind of mental process that goes on in an art scholars and general public's literate general public's mind. And we said, can we do this with the computer, with algorithms? And the answer is yes. In our preliminary studies, basically we have two. They're called deep neural networks. They are AI techniques for doing pattern classification and other things. And one of them is for what's called semantic segmentation, breaking the image so that you can identify where there are people and where there are not people.
David Stork (33m 4s):
And so applied to that, Verrochio painting. You would get the two main figures in the Tableau. But then we had another deep network that was trained using art images to recognize those attributes. In this case, the dove over Christ and the crucifixion cross in the hands of St. John. And then it's a very simple geometric problem to say, all right, for each identified attribute, which figure is closest, very trivial, and assign that look up in your database, ah, this attribute means this Saint or, or actor.
David Stork (33m 45s):
And so indeed the computer can say, ah, this is Christ in the middle. And this is St. John. And we've done it on a number of paintings. And we're expanding that technique. Now at one level, it's easy. I mean, if you show this to an art scholar, they say, Oh, obviously, you know, you can tell that's Christ, what's the problem. This is one of the banes of those of us working in pattern classification. We worked for decades to recognize a hand or a face or something like this that a child can do. But to me, this is one of the first steps towards this entire class of problems that AI really isn't addressing.
David Stork (34m 28s):
And that is inferring the meanings, the intention of the artist. So there's a lot of work to be done on this, but that paper that we published just last last month was exactly on, on this. And we're working, working hard on pushing it forward.
Michael Garfield (34m 46s):
Yeah. It seems related to the issue of using the application of machine vision in robotics, because there's this question of you, you may be able to train a warehouse robot to recognize or a caregiving robot to recognize when someone has a hand, that's a hand, but the difference between a hand that's reaching out to hold tenderly or a hand, that's reaching out to strike an act of violence. That seems somewhat related to a complication that you, you mentioned in this paper that there's an additional layer of context and inference required because in spite of the fact that yes, everyone has their unique attributes.
Michael Garfield (35m 33s):
And we're not just talking about Christian artwork. You mentioned the Greek and Roman Pantheon. You mentioned the word, the Hindu artwork. This is very much a universal human phenomenon,
David Stork (35m 46s):
Not to interrupt, but, or to interrupt likewise telling the professions of people. This person's holding a violin. I have an idea what he or she does, or a shovel and, and things like this. So yeah, it's much broader than Christian art. The reason we worked on Christian art is that there are a lot of examples, relatively large, small compared to the number of photographs that exists on the web, but in, in the art world, there's a lot of religious art and there's a very intentional and specific use of these attributes in order to basically convey a meaning. So that's why we chose that first, but it's a much broader class of problems.
Michael Garfield (36m 30s):
Yeah. So, so to that point, you know, with a violin, for example, someone holding a violin could be a violinist, but they could also be a luthierr. And as you mentioned, you say multiple saints share the same attributes. So for those cases, the attribute taken alone is not sufficient for accurate actor identification and a more probable probabilistic inference may be necessary based on a global criteria of all the actors and their simultaneous presence in candidate stories or episodes. So it seems as though what you're getting at is that it's about teaching computers how to read the room as it were the contextual clues.
Michael Garfield (37m 12s):
Melanie Mitchell gave a great talk on this about, say a self-driving car and counters someone walking a dog across the street, but you know, okay, you can say that's a human, that's a dog, but if you don't know the relationship between them, then you don't know how to respond.
David Stork (37m 33s):
Yup. There's a lot of sort of sub versions of this, this problem. But yeah, I mean, at one level, this is as we call it, AI complete that it takes full human level intelligence and cognition and memory and so forth to interpret. I mean, things like common sense that death is a bad thing. Where does it say that in the painting. Why is Christ had dipping down the side, such a bad thing? This is going to take a long time before we really get that kind of level of common sense. So as AI has progressed, it's it started on smaller problems that are relatively constrained, that nevertheless illustrate key classes of problem solving.
David Stork (38m 19s):
And that's what we're trying to do with art. But here's one thing that you, you do not get, this is one really fascinates me and you do not get in traditional AI. And that's the role of style as part of meaning, that if you take the same content to people standing by a riverside and you portray them in different styles, the meaning of the work will change. So for instance, in that Van Steenwijk, still life, the fact that all those objects were extraordinarily realistic, they worked extraordinarily hard. Someone like Peter Dubuque would spend days on a really small passage to get the textures right, the subtleties. Not only did that showcase their technical facility, but the goal was to say, this is real. This is right in front of these aren't saints, floating through sky with halos on their heads and clouds all around them. No, this is the world that is right in front of you. If you were to have that exact same content portrayed in a, an impression of style or pointless style or abstract style, the meaning would be certainly less than maybe even drained completely. And that to me is a really fast and artists, of course know this, they bring this aspect of style towards creating their meaning.
David Stork (39m 47s):
And just to give you one other example, Roy Lichtenstein is a very famous American pop artist. His paintings look like comic book images, and one of his most famous paintings is called brushstroke. And it's this just a, I think it's on a blue background, but it's a big yellow brush stroke made like a cartoon. So it's that it's in this cartoon style that makes it savagely critical of the previous art period, abstract expressionism, you know, Jackson Pollock and Joan Mitchell and the great abstract expressionists who made their gestures very visible and so forth.
David Stork (40m 29s):
And now in the moment, you know, in the instant and now Lichtenstein is going back and drawing it very carefully, making it in two colors, black outline yellow in the center, like a comic book. And it's that style. If he were to have the exact same thing in a realistic style, like a brush stroke, the meaning would have been entirely different. So this to me is an incredibly fascinating problem that art really far more than photographs conveys meaning from the lowest levels, the colors that you use, the brush strokes, that you've all the way up to the composition and so forth in ways that photographs really don't.
David Stork (41m 12s):
I don't mean to beat up on photographs, but to me, great, art just has the kind of levels of depth of analysis that even some of the best art photographs I don't think can match. I'm sure I'm going to get a lot of photographers sending me angry emails, but I'm going to stick actually.
Michael Garfield (41m 34s):
Actually, no, you spoke directly to a question that some of the people following our coverage of your talk on Twitter had about this, which is the circumscribed utility of this kind of forensic reconstruction technique. When it comes to, you have to make certain assumptions about the intent of a realist painter, given the period and the school. And so on. When the reality, as you just said, is that even working from life, the painters will alter a scene for any of a number of reasons. And so where does sort of part one of this conversation break on part two. What's the solution for the metal layer at which it may look like you've managed to say conclusively, that the light reflections off of this object are wrong, therefore...
David Stork (42m 34s):
Yeah, no, this brings up the kind of early criticisms I got from the art community before they really knew what I was working on. And the kinds of techniques. If you have a very shallow understanding of my work, you say, Oh, Stork thinks that a painting is right if it's in perfect perspective and it's wrong, if it's not, absolutely not. I'm not saying that whatsoever. We are using these techniques to understand and quantify these aspects. And these are not normative judgments. The Arnel Feeney portrait is not wrong or bad or worse because it's not in proper perspective.
David Stork (43m 14s):
These were the techniques at the time, we have to understand the art in its context, but when there's a claim that the chandelier is in perfect perspective, that is an objective claim that regardless of the period, and as you saw in my talk, I also did some analysis of chandelier's painted in 2004 I think they were. So we have to be very aware of the context and the questions. And frankly, the part of my work that I'm really most proud of, or is not most obvious to those who, who see it is how much time I spend corresponding and talking and meeting with art scholars to understand questions that they have.
David Stork (44m 4s):
I'm not doing my work for computer scientists really. I mean the later stuff, I think we'll push AI in new directions and I'm eager with that, but it's much more important to get a good question that is relevant to the art community solved reasonably well then to solve a problem that's for the computer scientists and do it extremely well that the art community doesn't care about. If I've worked really hard and made a absolute superb edge detection algorithm that worked superbly on the Mona Lisa, there's not an art scholar in the world who would care and they shouldn't, it doesn't matter to them. I'd much rather identify a problem often with them, almost always with them that they say, ah, yes, if you can solve this, that will help us.
David Stork (44m 53s):
So here here's just one example. I gave my talk on Vermeer at the Maritz House Museum in the Hague where the Girl With the Pearl Earring hangs and view of Delft. This is a superb small, it's a Mecca for those of us who love Vermeer. And then the chief conservator came up to me afterwards and said, I've got a problem for you. She took me back to the conservation studio and showed me paintings by Jan Van Der Heyden contemporary in the Dutch golden age of Vermeer, who did these city scapes that they're called Capri shows because he would put together buildings that weren't actually always next to each other.
David Stork (45m 34s):
He'd say, Oh, I'll put this building here or like that building there. And they used the technique for painting as bricks, thousands of teeny little bricks that look perfect that may have involved counter proofing, basically taking a metal plate, etching it in the pattern of the bricks and then printing it, pushing it against the paint to leave patterns. And if he did so maybe he would have done use the same plate on multiple paintings. You can't. And so the same exact same brick pattern would appear in different paintings. You can't detect that by eye. The amount of work of getting a painting in front of you.
David Stork (46m 15s):
And let me try, Oh, look over here. No, but this is perfect for computer search. You get a template basically of the patch and you, you check every other painting. And so we published a paper on that. I had known of Jan Van Der Heyden, but I didn't know about this question. It came 100% from the chief conservator at the Maoris house and we published two papers on it. And that's what I love doing.
Michael Garfield (46m 42s):
That's actually great. You led right into the next question that I had for you, which was about the fact that historically, you know, even realist, quote, unquote, realist painters, doing portraiture for clients we're often subtly or not so subtly incented to flatter their clients. Right. You know, we're going to cut a few inches off the waist and, you know, so you would imagine that light light field analysis would reveal, Oh, actually, you know, Henry the Eighth was probably 20 or 30 pounds heavier than he's depicted in this image. So you've got a similar kind of problem now. And I think people are thinking of this as a kind of qualitatively new problem, but your work reveals that there is a continuity here between that kind of question and that kind of approach to answering the question with deep fake forensics.
Michael Garfield (47m 37s):
And yeah, I mean, the fact is that we are perhaps now more than ever astoundingly vulnerable to what York University philosopher, Regina Rini called the epistemic backstop of recorded media, you know, that we believe a photograph and you know, so I'm, I'm curious what your thoughts are on applying these techniques, not just to art history, but to the present day and making sense of and figuring out what really, you know, it's sort of like a swallowing the spider to chase the fly kind of thing, right? It's a, it's a, it's an evolutionary arms race of counterfeiting against forensics. But how do you think that these tools can help us find the base truth of our lives in a world that's just increasingly spinning out of control?
David Stork (48m 26s):
Well, I have ambitions, but maybe that's, that's too large to, to restore Western culture to an epistemic unity that, that got lost once defects started flooding the internet. Let me step back and give you a class of problems where I'll call them semi forensic. I mean, my lighting analyses can tell whether there's great consistency in portrait lighting and other lighting, and a question that art historians address is centered on the portraits of Rembrandt. Of course, one of the greatest portraitists of all time.
David Stork (49m 8s):
We know he did many of his portraits from the figure. We know who these people were. We have the records of how much they paid when, when they sat for him and so forth. So there's no question whatsoever that so-and-so was sitting in Vermeer's studio in Amsterdam, on such and such a date. There are other portraits often of saints and so forth where it's much less clear. We basically, we know that he did it from his imagination. He wanted someone with this kind of curly nose. Maybe he found someone on the street, but often didn't. And then there are examples where we really don't know art scholars debate.
David Stork (49m 51s):
What's this a real person in Amsterdam, or is it from his imagination? The research I'd love to do, and it would involve a lot of, a lot of work and getting the data. But I suspect that the lighting would be far more consistent on portraits done from the figure, and it would be much less consistent on orchards done from his imagination. And then when we had one of uncertain origin, we could do the lighting analysis and say, aha, look how consistent it is. It's as consistent as if there were a figure there. And therefore that would give at least a probabilistic suggestion that there was indeed a figure there or not.
David Stork (50m 34s):
Now I don't call those deep fakes. I mean, Rembrandt, wasn't trying to fool us. He wasn't hiding, you know, lying to us in any way, deceiving us by making a painting from his imagination. But these techniques may allow us to go that one level deeper, sort of like a deep fake, but deep fake has a epistemic overtone that Rembrandt portraits don't at least in my mind,
Michael Garfield (51m 2s):
Here's a somewhat smaller bite sized question, which you addressed. And you brought up towards the end in the Q and A of your talk that something on the order of $1 trillion in art worldwide is believed to be mis-attributed.
David Stork (51m 20s):
Well or fake. Yeah, no, the, the problem of fakes, I could talk for many hours on this. I mean, it's a profound and deep, deep problem. The number of fakes and misattributed works is really, really large. And it might be as high as 20% even on museum walls. Now that doesn't mean they're all fakes. They're just misattributed that our scholars make honest mistakes and so forth. I've heard quotes that in the commercial market, it might be as much as 40% are not authentic. It's, it's hard to say because there are all these forces making it difficult to find them out, to find those facts out.
David Stork (52m 3s):
The largest one, at least in my mind is that there's a legal doctrine called defamation of property. If you own a painting that you're telling everyone is by Van Gogh and it's worth a hundred million dollars. And I come out and say, no, I don't think it is. You consume me, even if I'm right and so forth, and you can completely ruin my life. I don't want to get into the legal sides, but the problems of authenticating art, it's really deep and profound. And the kind of image work we're doing, I think can help in certain examples. But it's only one small part of a full regiment. You have to do iconography, look for what are called anachronisms.
David Stork (52m 46s):
There was the terrorist museums debacle two or three years ago, a museum in the South of France, which had, I think it was about 140 paintings of which 82 were determined to be fake. But one of the reasons they could tell was that they were landscapes, that included buildings that were built after the artist's death. So, so a leader for German artists went out, tried to use the same style, but inserted a building. So that kind of technique also looking at material composition, iconography there, there are a whole host of very difficult, you know, complicated ones and image is just part of it.
David Stork (53m 30s):
It's true at the time of Bernard Berenson and the early connoisseurs, the I Reign Supreme and bit by bit scientific analysis and x-rays and infrared re reflectography and all these other techniques got brought to bear, but the technologies I'm working on only address the pure image analysis part. There is another level of analysis, which is how do you put together this information, how, when you have different sources of information each with different reliabilities, you know, when, when five scholars disagree, what do you do?
David Stork (54m 9s):
Well, what are in fact I'm writing right now is a paper on taking the techniques from medical diagnosis. There are now AI systems that take objective measurements like blood pressure or chemical compositions of your blood along with subjective or more subjective measures and puts those together in a way to get the best diagnosis and you learn from large corporate, large databases of medical data in order to do this. And I think the same kind of rigorous probabilistic inference can help in putting together the different sources of information all with uncertainties in attribution and authentication of art.
Michael Garfield (54m 57s):
So that may actually be an answer to the last question I had for you, which in keeping with the oddly religious tone of this entire conversation is what do you regard as the Holy Grail of this trajectory of the evolution of this technology?
David Stork (55m 18s):
Good question. There were a number of layers. I mean, in my lifetime, I, my goal is to get these computer techniques to be part of every art history school and techniques that, that, that, that art historians and curators, and connoisseurs will use these, these methods, the way computing has revolutionized every other field, basically biology, chemistry, and physics, and so forth that, that trying to do the vast amount of chemistry without computing it's, it's unthinkable now, but in the long run, long meaning once we get human level AI, I would love to see how we can take scholars, connoisseurs knowledge and capture it in some sort of algorithms so that complex, very complex system, so that we can then say, how would Bernard Berenson have responded to this painting that we just found? And the deep, deep, far, far distant blue sky dream is for such a system to come up with a novel, not just an interpretation of a deep artwork, but a novel one, one that captures our, our attention admiration and becomes part of the scholarship. But that is so far away. I'm reluctant. I'm almost loath to mention it because the art scholars in your audience are pitting the base of their palm against their head and their they've turned off the podcast already.
David Stork (57m 3s):
So very, very far away, very far away,
Michael Garfield (57m 7s):
30 years ago, compositional style transfer seems unfeasible to many people. And so maybe it's not so unbelievable that you might be able to model and resurrect art critics.
David Stork (57m 19s):
I agree, but a better example is chess. You know, when Alan Turing proposed chess as a test for artificial intelligence, it's not a good one, but at that time, the human grandmasters thought this was silly, but now you can download this kind of software for free on your phone. And every world-class grandmaster is working with computing for chess day in, day out, playing against them, learning, exploring different routes and so forth. And in the same way, I think that's someday what we'll come to art scholars approach to some of the great masterpieces.
Michael Garfield (58m 2s):
It just kind of recursive. You kind of imagine that portraits of the saints of art history in another 200 years might be depicted with the attributes of their attribute detection algorithms sitting on their shoulder like angels.
David Stork (58m 18s):
I don't envision this for a long time, but that people will create art in anticipation of it being analyzed by machines, but the economics aren't there. People, people make art to communicate with humans. And that's great. That's left us some of the greatest experiences that we can get with our eyes.
Michael Garfield (58m 41s):
Well, I mean, certainly we already have adversarial fashion designed to spoof facial recognition algorithms. So I mean, it doesn't seem completely out of the pail.
David Stork (58m 49s):
Yeah. I don't think the artists it's true. The technologists are doing this, but the artists who want to communicate something want to affect humans and that's great.
Michael Garfield (59m 2s):
Well, David, this has been a total joy. Where would you direct people if they want to learn more, if they want to go deeper with this material?
David Stork (59m 11s):
Well, the first would be the SFI recording of my talk. I have a talk online at the Frick collection, just search the Frick. Then David G Stork. I have a website which I have not updated for several years. I've been too busy search on my name and then art analysis. In the long run everyone should buy my book, pixels and paintings, Foundations of Computer Assisted Connoisseurship from Wiley. It's not yet done, but that will be sort of the compendium of this state-of-the-art up to now and just look online. Those I think are a good source.
David Stork (59m 51s):
Excellent. Thanks so much for talking to us today. Thanks Michael. That's great joy.