What is Semantic I/O?

What is Semantic I/O?

By Brian Gormanly

Consider everyone’s favorite first program, "Hello World!", the classic first program written by software engineers.  The goal of the program is to express a message (the intent of the programmer), as an action carried out or an interaction with a user.  The implementation of the program is traditionally written in a specific programming language for a particular computing platform.  Today, if the programmer wants to express the same intent on multiple devices such as a smartphone and a google home smart speaker, they must rewrite or build the implementation for each.  These devices struggle to communicate with each other outside their own ecosystems.  

In this article, I want to consider a different approach.  What if, our "Hello World!" program was actually just a message, one sent over a special protocol called SIOP (Semantic Input / Output Protocol) that all the devices could understand.  Our "Hello World!" message will be described as being of a type of Text/english, which allows us to describe the intent of the message semantically.  This may sound similar to how the World Wide Web abstracts information using an internet media type, and in some ways for a basic message like ours it is.  The difference is that SIOP offers a way to abstract all software except a basic client for the protocol away from the device itself.  This means software and hardware can be paired based on the interactions they provide.  

Let's assume that we have a cloud software service that wishes to send our "Hello World!" message to a user.  This cloud service might have any number of functions, perhaps it manages home security, or aggregates the users SMS, home phone, and calendar messages.  Regardless of the cloud software ultimate purpose, one of the ways it accomplishes its use cases is by sending messages to the user.  Messages like our “Hello World!”.

"Hello World!" message sent to different devices

Let’s also assume that our user has 3 devices, a smartphone, smart speaker and braille terminal they would like to be able to use with the cloud service.  Each device has different hardware architecture, runs unique software and is produced by different vendors.  But the magic behind sending our Hello World message via a protocol that categorizes interactions as semantic I/O, is that all 3 will be able to understand the message, and will provide a tailored experience to the user based on the specific hardware they have.  How they produce this output is not important to the service, in-fact it is not even known to it.  The device by declaring it has the capability to output messages of the type "Text/English" provides all that is required to the software service to enable the connection between them.  The result of our little thought experiment is that the “Hello World!” message is spoken aloud to the user by the smart speaker, appears as a message notification on the smartphone, and manipulates the surface of the Braille terminal without the cloud service having implementation specific code for each. Conversely, the hardware retains the ability to express any message of type Text/English for any service that wishes to send a message.  Thus, it can perform this service for any interested service without any "custom" programming.  This marks the biggest difference between using Semantic I/O and another abstraction such as the World Wide Web.  On the Web, the remote software service is still ultimately responsible for the device implementation of the interaction, using Semantic I/O the service has no say in the implementation.      

Internet of Things devices

While this paradigm would be useful in many areas of computing, a prevalent problem faced in the area of Internet of Things (IoT) is separation of hardware and software.  IoT devices are commonly very small, with minimal processing and memory capabilities.  Most of the software they utilize is accessed via the internet as a cloud service, but this pairing of hardware and service is currently proprietary in nature, devices can only be paired with specific cloud services.  If IoT devices such as doorbells, cameras, stereos, TVs, smart speakers, and lights were freed from their vendor specific software requirements, we would be able to realize the benefits of software and hardware separation in this space.  If Amazon and Google adopted SIOP for communication, you could easily use your Amazon Alexa to control a Google Nest camera.  You would also have the option to use any software that could communicate with SIOP with your hardware, ensuring it is always up to date, cutting edge, and possibly even growing into new purposes beyond the original use cases.

To see why a concept like semantic I/O that offers a universal method of abstraction while not increasing hardware requirements could be so important in areas like IoT, let’s take a look at interactions, abstraction and distributed systems in a bit more detail.  Understanding how they influence the current technological landscape and why the current situation is so different from the past, is critical to understanding why we need to think about abstraction in a brand new way.

Understanding Interactions

Human Computer Interaction (HCI)

Sending a piece of text in a semantic fashion is relatively straightforward, but interactions such as these represent only a tiny fraction of the types of interactions that take place with computing devices.  Creating a protocol that is capable of encoding any type of interaction requires a thorough method of categorizing and describing them.  It must represent the original intention of the message accurately without implementation details.  

There is a verbose and well studied science called HCI (Human Computer Interaction) that examines interactions between people and computers in detail.  You are likely familiar with HCI even if you have not heard it called by a specific name before. All of the biggest "shifts" in the technological landscape have been marked by new eras in HCI.  The first era in HCI was the terminal, the second the GUI (Graphical User Interface), third touch centric mobile computing and some believe we are closing in on the forth, Augmented Reality.  Types of modern interactions with computing devices have been expanded to include the spoken word, gestures, body movement, video and audio.  HCI also includes expression of primitive data values and text.  In order to be successful as a universal means of communication, Semantic I/O must be able to effectively express all of these different types of interactions and be flexible enough to allow for future interactions we have not yet considered.

Computer - Computer Interaction (CCI)

Interactions do not always take place between a human and a computer, however. For a concept like semantic I/O to provide an effective abstraction layer, we also need to consider interactions between computers directly.  Today, we tend to think about these "Computer to Computer Interactions" or CCI in terms of operating systems or software APIs (Application Programming Interfaces).  An Operating System provides a System Call API to all application programs to access system resources, and to provide abstraction between application software and the hardware.  A RESTful or SOAP based web service allow different application services to interface with each other.  Other types of software abstraction include database connections and technologies such as the World Wide Web which use browsers to render content using a markup language.   All of these methods of CCI accomplish the goal of allowing interactions between disparate systems, but are effective because they add layers of abstraction in order to achieve their goal.

The Decline of Abstraction

The abstraction onion

Progress in computer science is usually assisted by adding more layers of abstraction, a bigger metaphorical onion.  These layers allow engineers at all levels to be more effective by concentrating on their problem and utilizing interfaces with the other layers of the onion to accomplish goals.  Things like file management, user interface, networking, disk and memory management.  If all applications had to manage these concerns individually complexity of software would be inhibited by "re-inventing" the wheel every time.  Abstraction allows new software to interact with and utilize other software without having to understand the encapsulated complexity that makes those layers work.  Engineers have continued building on this model for decades and the onion has continued to add layers.

Fig. 1: Operating System abstraction

One of the fundamental abstractions we use everyday is abstracting application software from hardware using an operating system.  It allows us to choose the software to run.  If we buy a Dell computer we can still run adobe software, just like we can if we buy a Mac.  (Adobe must build their software for all platforms they want to support if it runs native, but once they do they do not have to create a new one for all Windows PCs, or the next Mac Pro, unless there is a major architectural change in hardware or Operating System) . The operating system also provides a multitasking environment, resource security, access to hardware with drivers and a litany of other services.

In addition to the abstraction provided by an operating system, application software engineers have also ventured deep into the abstraction jungle.  They have created their own layered software architectures.  This approach allows abstraction between data persistence, object-relational mapping, services, controllers, cache, APIs, Views and more!

Fig 2: Application abstraction - Image Credit: https://blog.jooq.org/2014/09/12/why-you-should-not-implement-layered-architecture/

The trade off for all these benefits is added complexity.  They enable the user to enjoy the ability to update, replace and create software for their system as they see fit.  They also enable software to communicate and integrate external services written in other programming languages by other vendors.  All of this works well so long as the hardware you are using supports the burden of the ever growing onion.  Moore's law (the principle that the speed and capability of computers can be expected to double every two years) and the shift of software away from native and towards cloud based web systems fueled the onion to larger and larger numbers of layers.  Recently however, there has been a major reversal in this trend.  Devices are suddenly getting smaller, slower and emphasizing low heat and power consumption over raw speed and gigabytes of memory.  The amount of layers of abstraction this more limited hardware can support is getting smaller.

Distributed, Small and Purpose Built

Today, computing devices are becoming less complex and more purpose built.  Their power is not measured in raw speed and memory but as what can be accomplished as an interconnected whole.  Even though the net outcome of the distributed world is positive, the individual device is shedding their ability to support a dense onion of abstraction.  Mobile devices were the first round of this trend.  Generally these devices still have an operating system, but they have diminished multitasking abilities and application software does not have the same level of access to hardware and resources.  The traditional multitasking environment of a desktop is replaced with a foreground application and background services model.  They commonly provide access to 3rd party applications only through specific “App Store” ecosystems.  Instead of using a system call API, applications access resources through a more controlled SDK.  Mobile operating systems will commonly “kill off” a process running in the background anytime resources are scarce.

IoT devices are the latest incarnation of the trend and shed even more freedom and abstraction.  They commonly have no operating system at all and are usually shipped with software engineered to run directly on the microcontroller within.  Modern IoT devices rely heavily on cloud based web services and applications to provide additional features and UI.  Cloud security cameras such as Google's Nest and Amazon's Ring stream video to cloud based application servers where the feeds are evaluated for movement faces and sound, among other features.  These processes are memory, cpu and storage intensive, and executing them in the cloud service means that the device can be much lighter on memory and other hardware resources.  This has a significant impact by reducing the cost of the unit, heat produced and battery usage.

The modern push make hardware smaller, consume less power and create less heat means that any attempt to add abstraction in the classic sense is counter intuitive.  This is one of the main contributing factors behind the rise of the vendor specific ecosystems we see today.  If you own a Google Nest doorbell, Google home smart speaker and Google Chromecast, you can easily ask your home to display the video feed from your doorbell camera on your TV.  If you have an Amazon Alexa speaker, combined with the same Nest doorbell and Chromecast, you are out of luck, at least for issuing a verbal command.

Another byproduct of this trend is the inability of the consumer to control the software paired with their hardware.  If Google decides to cancel support for an earlier generation of Nest doorbells, consumers may find themselves running outdated software on the devices.  Eventually it would also be likely that the cloud services that display the video feed and perform the motion, sound and facial detection algorithms will no longer accept the video feed of "older" devices.  Even if the devices hardware, a video camera, an LED, a speaker and a button are all still fully functional and effective.  The device would become useless at this point and the consumer forced to replace it.

Beyond the grim reality of having otherwise working devices unsupported and 'aging out', there is also the loss of competition in the space.  A Ring doorbell is likely to only ever run algorithms produced by the company, just as a Nest is likely to only run Google software.  There is no opportunity for a 3rd party software vendor to introduce a superior algorithm.  Perhaps creating software used by consumers of both Ring and Nest and other vendors.

Semantic I/O to the Rescue

Semantic I/O offers a way to abstract without additional hardware requirements.  Additionally, unlike a web service API or an operating systems system call API, you do not write code to communicate with the specific implementation of the API.  The trade off, if you want to think about that way is that the software service loses direct control over the interaction implementation.  The service can merely say this is the type of interaction I expect and here is the message I intend to convey.  The details of how it actually happens are transparent to the service.

Fig. 3: Example SIOP service architecture

SIOP Software Service

Let’s describe in a bit more detail the relationship between the software services and the hardware clients.  The software service can run on a traditional server and associated software stack.  The example architecture in fig. 3 shows a traditional n-tier application, only some common software layers have been included for brevity.  It can be a cloud based or local service so long as it is accessible on a network.  The service can be written in modern programming languages, use current build methods, employ continuous integration and current testing models.  They can run on the same assortment of application servers in use today.  

The most important change for the software service is the API layer.  Instead of employing a RESTful, SOAP/WSDL or some other web service layer, communication is instead done by sending SIOP messages.  SIOP messages can both be sent and received in the same manner as other application level protocols on an exposed port.  There is no need to have a specific URL structure to support SIOP messages as with web services, instead the service has a manifest of interactions that it supports.  These interactions are paired with hardware clients that can provide implementations for the interactions.  Any hardware device that declares it provides an implementation for a specific type of service interaction may be paired.  Multiple devices may also pair to a single interaction point.  Once paired the service uses the device’s Id number in messages received and the type of the interaction of the message to ‘route’ the message to the appropriate code function.  For applications that which to support multiple interactions of the same type, there is also a ‘context’ feature of the protocol that can be used to differentiate.  Alternatively, the service function receiving the message may also chose business logic to apply based on the specific content within the messages.  Functions do not have to only listen for messages from known, paired devices, they may accept a message from any device provided the software understands the type of intent conveyed.

Fig 4: Example SIOP client architecture

SIOP Hardware Client

The hardware client communicates with the SIOP based service over the internet or other network.  It also declares the interaction types the device can support and provides the implementation specific code that translates the intent for the hardware.  The SIOP hardware client software is the only required software for the hardware component.  Other layers may exist below the SIOP client as hardware resources allow.  An example of a generic microcontroller architecture stack may look like is shown in fig. 4.  Only the area in green is relevant, the lower level layers can change in just about any way.  We can compare it in some ways to a Java Virtual Machine (JVM), there are major differences.  Just as the JVM acts as an abstraction layer that allows the same Bytecode to run on hardware of completely different types, the SIOP hardware client allows SIOP messages to be received and implemented on hardware of completely different types.  Just as we saw earlier with the World Wide Web, the difference is that SIOP messages do not contain implementation specific details as does Bytecode.  An example is showing a Hello World message on a screen verse the message being spoken aloud.  The Byte run by the JVM would be specific to the type of output, whereas the SIOP message is not.  It simply sends the intent to express the message and relies on the hardware to implement as it sees fit.

The SIOP hardware client also provides a manifest file that declares the supported interaction types the client is capable of supporting.  The total set of available interaction types is defined within the SIOP specification.  Messages sent to the client that are not within the set of supported types will be ignored.  The SIOP client is then responsible for having implementation logic only for supported types of interactions.  Software services will read this manifest to see which interactions the hardware may be paired with.

Many things change…

As you have probably derived by this point, using semantic I/O to express intent is a big departure from the accepted computing paradigms used today.  The relationship between software and hardware changes in many ways.  SIOP clients can multitask in a sense without an operating system for example.  This is possible because all the cpu and memory resources needed for the service are managed by the server(s) running the service(s).  The only responsibility of the SIOP client is to manage SIOP communications and interactions with the user.  Since the user can only be interacting with one service at any point in time, SIOP clients are in fact capable of multi-tasking from a users point of view.  Multiple messages can be sent to multiple services which individually spend compute time and respond asynchronously to the SIOP client.

Another major change is that software becomes service focused and the concept of “Applications” diminishes.  An application includes things like a Graphical User Interface (GUI) and interaction management with the user via touchscreens, keyboards, microphone, mice and such.  SIOP services do not need any of these capabilities since they are managed by the hardware client.  This allows a more pure “service centered” software to be built.  It would also promote reuse of existing services and a more distributed approach to development since implementation specific details have been removed and services can talk to other services also using SIOP messages.  Since SIOP messages have no implementation specific details of their own, any SIOP service that contains a manifest stating it supports a particular interaction type may be used by any other service wishing to interact with it.

I will have an upcoming post to better detail these and more more “paradigm shifts” I expect from using semantic I/O as abstraction in a coming post.  For now, I just want to introduce the 5 general principles I see at this point.  Implementing and using a system using semantic I/O abstraction implies that the system will abide by these principles.  While it would possible to try to engineer a system that abstracts semantically with I/O and does not abide by these principles, it would be counter-intuitive and a “up-hill battle” to say the least.  The 5 principles are:

  1. Cloud based
  2. Semantic I/O Abstraction
  3. Software Centric
  4. Service Focused
  5. Loosely Coupled


There is much work to be done to build a working model to test these assertions.  Part 1 of this experiment was breaking out the concepts at a high level and understanding how they fit together.  This post and more that are following shortly are the results of the work of part 1, but we are a long way off from a working product.   First and foremost the SIOP protocol and the ontology of interactions that enable it must be defined.  That is the work that I am currently beginning.  Everything discussed so far is conjecture at this point and by no means do I believe that the reality of the implementation will follow this “thought experiment” word for word.  At this point feedback of all types is the most important component.

Which brings me to the final thought of this first post on semantic I/O.  Please share this post and most importantly please provide any feedback you have.  I have setup a forum and I invite you to use it to be part of the discussion.  You may also email me directly or any other way you are comfortable with.  Having a diverse input is going to be crucial to the success of the effort to get the implementation right (or as much as possible) the first time.