The Internet of Thinking Things

Alois Ferscha,

Future in a Nutshell is an Ars Electronica-organized series of talks that is now being offered exclusively to the staff of Greiner Technology & Innovation. Over the coming months, a lineup of experts will elaborate on the most important developments and technological trends of the next decade. Leading things off was computer scientist Sepp Hochreiter on the subject of artificial intelligence. Next up is Alois Ferscha of Johannes Kepler University Linz, where he heads the Department of Pervasive Computing and the Austrian Research Studio for Pervasive Computing Applications. We recently chatted with him about the Internet of Things.

Why has the Internet of Things become so fascinating of late?

Alois Ferscha: The Internet of Things has gained both significance and acceptance. In 1999 when Kevin Ashton was the first to refer to the Internet of Things in a scientific context, there ensued the first discussion of the fact that inanimate objects could be linked up in networks. And it was precisely at this time that I implemented the first reference example of the Internet of Things—an “internet suitcase” that listed its contents on an internal website. Depositing articles of clothing, books and other objects into the suitcase lengthened the inventory; removing them shortened it. Furthermore, each item placed inside had its own website—a shirt website, for example, that kept track of where it had ever been washed, where it was purchased, and where the manufacturer’s specifications and washing instructions were online. This information could be accessed via an internet-enabled mobile phone set up with the W@P technology in use back then. Of course, this wasn’t the conception of a new generation of technology; it was merely a provocative demonstration to call attention to the fact that not only human beings—discovering the world of websites at the time—but also things could get interconnected and derive a benefit from an online presence. This gained popularity with the advent of wireless local area network technologies.

In 2000 CISCO’s first 802.11 wireless system was made available to us at JKU to test how wireless communication throughout the campus could function. Then, the Wireless Campus project that I headed became Austria’s largest WIFI network with 240 access points—a gigantic WLAN and a trailblazing effort. Even more interesting were the tiny Bluetooth- and ZigBee-based wireless systems that made local area data exchange possible. I mention this because that was a generation of the Internet of Things that I like to call the Internet of Networked Things.

What could this Internet of Networked Things do?

Alois Ferscha: That was a generation of the Internet of Things in which A could send a message to B and B could receive it. It was simply a matter of networking and data exchange. This subsequently became more substantial with work to make things aware of their immediate surroundings. We’ve finally been able to implement capabilities in things whereby they can perceive themselves, ascertain their current location on the planet, if it’s hot, cold or warm there, light or dark, accelerating or motionless, quiet or not. Suddenly, things could not only receive messages; they could also perceive their environment. This induced a second generation of technology that I refer to as the Internet of Aware Things.

Thus, the Internet of Aware Things goes one step further…

Alois Ferscha: This means that things can place the incoming messages into a semantic context—namely, what the overall situation is. This perceptual capacity has expanded. Now, things can perceive not only the situation but also the users and their activities—whether they’re walking, standing or sitting, the modes of locomotion, the interaction modalities and ultimately something like a vital condition—the emotional state and level of attention. These are things that the object can ascertain for itself—whether the user is paying attention to it at the moment. If not, than the human being can be alerted abruptly if need be, or information can be provided in ways that aren’t distracting or disruptive. In the meantime, attention has become the key resource in the configuration of modern information technology; it is one of the highest levels of a thing’s ability to perceive. This can be implanted with a variety of sensors—acceleration sensors, bio-signal sensors and eye-tracking technologies.

What’s the status of the Internet of Things today?

Alois Ferscha: Right now, we’re witnessing the advent of a new generation of the Internet of Things. A combination of artificial intelligence technologies, machine learning techniques, mathematical methods of pattern recognition, and the technology used by DeepMind in the AlphaGo program is now being embedded in things.

We’ve already built things that can recognize patterns—for instance, movement patterns of feet, which these objects can register and evaluate with minute sensors built into the soles of shoes and then, in turn, impart what they’ve learned to their environment and cause something to take place—for instance, turn something on or off, adjust a dimmer, open or close something without a person having to think about it. It happens because, like in this case of the shoes, the information is communicated to the household electronics. We’ve carried out lots of projects—wearable computing, vibrating belts, armbands and smart watches all the way to smartphone applications in which the smartphone functions as a sensor node in a highly complex sensor network—acceleration, luminance, noise level, magnetometer, gyroscope, orientation via coordinates. These normally require a lot of memory storage capacity because they have to be trained. The state of such a thing’s capabilities is actually the sum of all the learning impressions or training impressions it has ever received.

Lots of memory, a gigantic collection of data and faster processing have changed the Internet of Things…

Alois Ferscha: Obviously, the more data is available to train a thing, the better it will later be able to recognize a pattern. This is data-intensive, but also computing-intensive. Consider these DeepLearning techniques in which there are feedback loops from the various layers of a neuronal network. What’s necessary for learning to take place is a weighted adjustment from one layer to another, which has to be adapted in deeply recursive fashion. All of this takes a lot of computing power. DeepBlue, the chess program that was able to defeat reigning world champion Garry Kasparov in 1997, still had to run on a supercomputer. We’re now well on our way to a point at which this same technology can be radically reduced in size, memory capacity and computer power. Ultimately, it will be possible to integrate this technology into small things—clothes, everyday objects, furniture and self-driving vehicles. Basically, that’s the new principle of the Internet of Things, which I like to refer to as an Internet of Thinking Things.

Give us a precise definition of the Internet of Thinking Things…

Alois Ferscha: This isn’t just a matter of communicative or perceptive things; rather, increasingly, due to the artificial intelligence technologies deployed in them, these are thinking things. If learning is a category of thinking, then machine learning has already gone far along this path towards systems capable of learning, but this isn’t the only dimension. There’s the perceptive dimension, the ability to recognize something. And we also have the dimension of drawing conclusions from a wide array of facts—for sizing up a situation, for making plans, for evaluating current states and predicting future states, all the way to autonomous decision making, and ultimately autonomous acting. Bringing together all of these functions in a small space and integrating them into things—that will constitute the Internet of Thinking Things.

If things start doing the thinking for us, what, in your opinion, will be the role of human beings in the future?

Alois Ferscha: The human being will remain a human being. And technology will be there to aid people when a human’s capacity is insufficient, or one prefers the support of technical systems because perception times, switching times, or setting control times are much too short for human beings to be effective and we would be well advised to turn these tasks over to machines. Consider, for example, recognizing obstacles on the street—we know that a human being’s visual recognition time is approximately 140 milliseconds, which is an eternity in comparison to a nano-speed processor. In these same 140 milliseconds, a 3-gigahertz processor executes 420 million instructions, while we’re still occupied with identifying what we’re seeing. And it would indeed be a shame to dispense with the possibilities of this technology.

In such cases, technology is a prosthesis to support a human’s perception, motor skills and capacity to get around. No human being can run 100 meters in under nine seconds, but machines can do that easily. No one can lift seven times his own body weight above his head, but that’s a simple task for a machine. You can cite lots of examples in which a person’s kinetic or motor constitution is inadequate to a task she would like to carry out. And that’s why we make use of machines—and it’s good that we do. Basically, the human’s position isn’t being called into question here; we are always at the center. The focus of everything we do to develop technology must simply be to serve humankind.

The Internet of Things is still highly dependent on human beings—updates have to be imported and human being are occupied with the constant enhancement of the technology. What’s your take on the security of the Internet of Things?

Alois Ferscha: The subject of security plays a major role in the Internet of Things just like it does in all technological developments. Autonomous vehicles have been involved in serious accidents, though their number has been small—for example, in a case in which sensors have been optimized to recognize an open road ahead, and measurements are taken underneath a semi-trailer and the side wall of the trailer reflects the sky so that the optical system interprets the cloudy sky as an open road and the vehicle drives at full-speed into the semi-trailer and the passenger inside dies. As regrettable as this accident was, it nevertheless belongs in the category of learning phase error. It was same with the first intelligent embedded systems in the mid-‘50s. When a cardiac pacemaker is implanted, a processor measures the heartbeat, intervenes in case of dysrythmia and emits impulses to keep the person alive. Here too, there were accidents initially, and people nevertheless deployed this technology to minimize the risk of dying. All of these technologies go through a maturation process—they get better and better, also with respect to security. Security is a human need, and, in the technology marketplace, anything that constitutes a human need quickly brings forth the readiness to invest money in satisfying it. It can be anticipated that the Internet of Things will become increasingly secure.

In Industry 4.0, things communicate with each other in factories, and in smart homes, things are linked up in networks. Tell us about the standards governing this.

Alois Ferscha: Most things that are industrially successful and thus practical have come to be in accordance with standards—maybe not formal ones; maybe they’re only arranged and mutually accepted. The most reliable standard in the Internet of Things is the IP Protocol developed by Leonard Kleinrock in 1960s. It divides messages into packets, numbers them, sends the packets via physical communications channels, cables, through networks, and then reassembles the messages according to the numbers. This is an internationally established standard, with 4.3 billion PCs and 6.2 billion smartphones functioning on the basis of it.

In the hierarchy of abstraction of Web technology all the way up to semantic systems and systems capable of thinking and learning, these standards are being amassed. In light of the fact that, on the free market, every participant would like to make a lot of money, many people see an opportunity to become a big player by taking matters into their own hands and developing their own standards—which means a world of local standards which is, of course, detrimental. It’s always been this way. I don’t consider this particularly dangerous; it just means a bit of a delay. It could proceed faster if we had a commitment to a standard as reference according to which everyone could do their development work. So it’s taking a little bit longer; nevertheless, the end result will be that the things will be able to get interconnected reliably, recognize one another and think.

What do you consider the most interesting thing about the Internet of Things?

Alois Ferscha: One interesting thing—though maybe not the most interesting anymore—was the data glasses we used to use. That too attracted imitators. For instance, we got a visit from Thad Starner, who was then CEO of the Google Glass project and had reviewed our papers such as “Wearable Displays for Everyone!” This was a reference for our research, to no small extent because it best reflected the way people deal with technology. Aside from people who wear glasses to make themselves look cool, they’re worn to compensate for bad eyesight. They compensate in technological form for something that human biological capabilities or motor skills alone can’t do—or can’t do anymore.

“You acquire this technology and you can see clearly again, and when you’re tired of seeing clearly you can take the glasses off.”

This is also how I regard the Internet of Things. It’s not an eternal, inseparable marriage; this is always a union subject to the individual’s free will—whether you want to own an Internet of Things thing, and how long and intensively you want to interact with it. In this sense, the glasses are a vivid example of how we should understand the human-machine relationship in the Internet of Things. This is a matter of help—but only as long as a human being wants it. When it’s no longer desired, you can put the technology aside and you’re purely human again.

Alois Ferscha

Univ.-Prof. Dr. Alois Ferscha is Director of the Institute for Pervasive Computing at JKU Linz, Head of RSA FG Research Studios Pervasive Computing Applications and Head of the Austrian COMET K1 Competence Center for Products and Production Systems of the Future, Pro²Future.