White Paper

v1.0 March 1998

v1.3 June 1999

Joseph 'Jofish' Kaye

The Counter Intelligence/Kitchen Sync White Paper is a look at the future of kitchen and domestic technology. It serves as an introduction to designs, fundamental technologies and future developments in this field.


Primary Stage Devices: No Communication
Secondary Stage Devices: Self-Identity
Third Stage Devices: Memory
Past Projects
PC Dinners / Microchef
Mr. Java
Current Projects
Counter Intelligence
Future Projects
Cool I/O
Everything Bit
Other Projects
Scenario: Chocolate Cake
Theory and Concepts
Cloud of Bits
Credits & Acknowledgements



This paper presents a vision of a future. In the last two decades, a great deal of time and research has been spent on "Living Room of the Future" projects. These usually meant big screen televisions, quadraphonic sound and strategically placed sofas. The problem with the living room is that it's a passive environment: the user sits and absorbs, be it reading the newspaper or watching television. Compare this to the fundamentally interactive kitchen.
  Let us step back one hundred years into a kitchen of the 1890s. It was barely recognisable as today's hygenic, (hopefully) aesthetically pleasing center of the home. Refrigeration was at best performed by lumps of iced delivered intermitantly; electricity absent, and most heat provided by coal or wood stoves. It was the domain of the servants, or perhaps the wife of the household, hidden at the back of house. Think of the progress of the last one hundred years; then think of the progress of the next one hundred years.
  The kitchen is an intrinsically dynamic environment. Raw materials enter, are processed and leave. The user of the kitchen invariably contributes to that processing, whether it be putting a pre-frozen lasagna in the microwave or meticulously leafing through The Joy of Cooking for a four-course dinner. As the two-way conversation of the Internet is to the one-sided oratory of television, Kitchen Sync is to the Living Room of the Future.
  Kitchen Sync, the vision behind Counter Intelligence, is a digitally connected, self-aware kitchen which has knowledge and memory of its activities. It is a multifaceted system, consisting of both intelligent individual elements and, fundamentally, connectivity between those elements. In practice, we are building individual components, while maintaining design philosophies that encourage inter-unit communication. Within the Personal Information Architecture group, we are also building an infrastructure, Hive, specifically designed to enable communication between such elements.
  We present here a basic overview of the three stages of devices which comprise Kitchen Sync. References to individual projects will be made clear as we go into more depth on the theories behind their operation.

Primary Stage Devices: Self-Identity & No Communication

  At the lowest level of intelligent elements are devices which are only aware of themselves and do not communicate to the outside. One of the first realizations in working with Things That Think is that this is simply not interesting. With intelligence comes the need to communicate, to move data in and out.

Secondary Stage Devices: Self-Identity & Communication

The first element of Kitchen Sync was a microwave, named PC Dinners, and later renamed MicroChef. It embodied some of the ideas around which Kitchen Sync is based: identification and association of information with that identification. Later versions added more control over the actual use of the microwave, including new functions correlating cooking time to weight, but the basic information/identification structure has remained unchanged.

Mr. Java is exemplar of the second stage of Kitchen Sync. An intelligent coffee machine, it identifies the user through their cup, and feeds the user the coffee they want and the information they want. For example, as an expatriate Englishman, I have a double tall latte and listen to the latest news from London.


Third Stage Devices: Memory


The third stage of Kitchen Sync has components that are not only self aware but have some memory of their use. A Mr. Java coffee cup is tagged with a simple RFID tag, functioning similarly to a barcode. A given cup merely knows it is, say, object #1422722. A single step along is a coffee cup that knows when it was last used and will release this information when asked - perhaps by a coffee machine, or a desk that wants to make sure your coffee doesn't get cold. Perhaps more useful is a fridge door that knows when it was last opened - and if it's been closed since that point. The simple addition of memory adds a wealth of possibilities for an item.
  One can also think of this third stage of 'having memory' as being aware of temporal sequences. Mr. Java, a second stage device, functions in the instant: when a cup is presented, he makes coffee and plays the news. Each cup is treated as a separate isolated incident, unrelated to the one before it. Counter Intelligence, however, is aware of time and sequence - it's important to add the flour before putting on the icing, for example.
  We can continue along this trail: one project for the future is to produce a fridge which is aware of its contents, and can take action based on that awareness. We envisage a fridge that is not only aware how much milk it contains, but also orders more should that milk run out. Once a fridge - or a cupboard, a pantry, a larder - is completed, Kitchen Sync will have reached a stage where there is a serious possibility of using the intelligent Kitchen on a day-to-day basis.
  Eventually, we hope to assemble an entire intelligent kitchen environment and spend time working within the space and using the equipment on a day to day basis. In the long term, of course, we hope to see Kitchen Sync projects being used in commercial and residential kitchens.

PC Dinners / MicroChef

PC Dinners was the first Kitchen Sync project, formed as a collaboration between myself and Steve Gray, prior to conceiving an intelligent kitchen as a whole. In its simplest incarnation, PC Dinners was a microwave with a barcode scanner, controlled by a computer. It associated two sets of information with a barcode: cooking information, and a sound file. Both were tailored to the product, so French toast asked you to "Pliz remove ze toast from ze packet and put it in ze microwave, si vous plait." Frozen Danishes had the 'Danish Chef' saying something along the same lines - with the addition of the occasional "bork bork bork."
  Gray later added to the user interface, providing the facility to change cooking times and store new recommended times, and also added a simple weight-scaling function. The project was renamed MicroChef.

Mr. Java


Mr. Java is an intelligent coffee machine. It's based on an Acorto 2000s automatic coffee machine, which in its unaltered state makes a variety of hot coffee and milk based drinks at the touch of a button. By interfacing with the diagnostic serial port, we were able to control Mr. Java by means of a tag reader placed under the spout.
  The user places their cup under the spout, as usual. The reader located under the spout reads the tag on the bottom of the cup and transmits the result to a computer. The computer would then issue commands to the Acorto to make the appropriate drink, and play the associated RealAudio feed through the speakers.
  Mr. Java does not gather information on individual users' coffee use, although it provided that facility by letting users set their own URL for their audio feed, which would let users keep track of their own consumption. However, we did keep track of overall consumption, including dividing the data by day and by hour over time. For example, we saw a consistent daily pattern:

graph of daily purchases


Morning coffees peaked at 11am, and we saw another mid-afternoon peak at 3pm. Barely visible at 9pm is our first espresso peak; apparently, if you're still drinking coffee at 9pm, then it better be espresso.
  This kind of information was of great interest to many sponsors: both Kraft Foods, owner of Maxwell House, and P&G, owner of Folgers, spend a great deal of time and effort tracking usage statistics such as these. Presently, it's entirely done by hand: someone sits next to the coffee machine with a clipboard. Mr. Java's type of unobtrusive monitoring that can actually add value to the product being purchased has possibilities for a wide variety of applications.
  Mr. Java has been a great success. EDS purchased an entire system for their Dallas MarketSpace of the Future, and are presently considering assembling another ten systems for various offices and showrooms. Kyle Anderson, CEO of Acorto, sees Mr. Java as the missing element between a regular Acorto automatic espresso machine and the barista: it provides entertainment. A barista chats about the weather, tells jokes; Mr. Java adds back that functionality. There are currently plans to exhibit a Mr. Java in Acorto's main lobby.

Current Projects: Counter Intelligence


The kitchen counter is one of the most used portions of the kitchen: workspace is invariably prime real estate in food preparation. A wide variety of tools are used in conjunction with the counterspace in any food preparation: weighing scales, measuring cups, bowls, spoons, and - importantly - ingredients. Counter Intelligence tries to integrate itself into your work habits by serving as an interface between you, the recipe, and the food being prepared.
How can a kitchen help you make brownies? You pull out the cookbook, and start measuring ingredients into a bowl. Out of baking powder? Well, you remember you can use baking soda, but do you double or half the quantity? Oops. Just put the eggs in before the milk. Hope it doesn't matter - wonder why they're listed in that order in the recipe if it doesn't? If you use semi-sweet chips instead of dark chocolate, how do you adjust the sugar? Can't see how much butter to add: that dark chocolate from last time got on the page.
  Counter Intelligence takes away these problems. It's fully aware of a recipe: the sequence, the ingredients, possible substitutions. We're in the process of building it as a fully expandable system, enabling us to modify the user interface as we learn more. The current (extremely) prototype system uses a barcode scanner, a scales, and a keyboard for input, and a standard screen for output . It 'knows' a handful of recipes, can suggest substitutions for one or two products, and has a text-based interface.
We envisage a system almost entirely integrated into a standard workspace area. A scale built into the counter along with an RFID tag reader lets you identify and tare mixing bowls, whereas other tag readers, barcode readers or LazyFish could identify ingredients. A LazyFish would let you select ingredients and finished products by tapping their picture on the surface, with the entire recipe becoming an interactive experience. Perhaps instead of a line of text saying "Mix in two cups of flour", Counter Intelligence will have a pair of elves projected on your counter, apparently tugging at your real bag of flour.
  The possibilities of Counter Intelligence are practically endless. We are consciously not predicting an exact path of evolution for this project, or exact technologies we wish to work with. By letting it evolve with the technology and change as possibilities arise, we're free to create and invent entirely new concepts of kitchen interaction without being locked into an obsolete model.

Future Projects

Cool I/O


The concept of an intelligent fridge - Cool I/O - is one that seems fundamental to the intelligent kitchen. We see a fridge as performing, a number of functions.

Primary and most fundamental is the ability to identify objects within it. This simple sentence encompasses an entirely non-trivial problem: a later chapter deals with some of the technological possibilities that could make this occur. What has become clear is that once this hurdle is overcome, there are a vast number of advantages to everyone who interacts with a product, from manufacturer to consumer.

  We see an initial application in the commercial kitchen, which, by virtue of sheer turnover and capital investment, has much greater need for Counter Intelligence's information-tracking capabilities. Specifically, we see a fridge that can keep track of its contents, including, for any given item, the location in the fridge or fridges, dates that item entered the fridge or was used, and expiration dates.
  An obvious next step in this endeavor involves keeping track of desired contents. For example a home might wish to keep at least one gallon of 2% milk, not more than two days old, at all times. From this it is a simple matter to have automatic shopping list generation or, logically, online ordering to replace staple items through a grocery delivery service.
  An intelligent freezer would perform many of the same functions, but would perhaps be easier to prototype, as objects that go in or out of a freezer are generally either in Tupperware-type packaging or in their original packaging.
  We see the development of CoolIO or a similar intelligent fridge as a fundamental part of the Counter Intelligence vision.
  In the summer of 1997, the fledgling Kitchen Sync team worked on an intelligent fridge that kept track of all contents entering or leaving the fridge through a doorway-mounted barcode reader. We found that this user interface was entirely impractical, and massively inadequate for any consumer use. We expect that a fridge that knows what it contains will have to rely on an RFID-like tagging scheme.

Everything Bit: The Kitchen Sink


In an interconnected kitchen, even disposal units are part of a communications network that keeps track of comings & goings. We see the sink in such a kitchen at a minimum having a tag reader to read tags from reusable containers being washed - Tupperware and the like.
Envision the following scenario. You've had lasagna for dinner, and there's some left over. You put the leftovers in a Tupperware container, and put it in the fridge. "Is that lasagna?" asks the fridge - it remembers you made that for dinner. You confirm. Later on, feeling peckish, you pull out the leftovers and take half for a snack, putting the Tupperware back in. CoolIO remembers what was in that tagged container, and so assumes it still contains lasagna. Hungry again, you pull out the remainder and eat it for lunch the next day.
  You put the dirty container into Everything Bit, and wash off the food. As you do that, the RFID tag reader reads the tag and washes off the data, labeling it as empty. Next time you use that container, the fridge will ask you what's in the box.

Other Projects


We see all kitchen appliances as having the facility to be integrated into the Kitchen Sync environment: cameras above stoves can ensure that a watched pot never boils over. Tagged Tupperware can work in conjunction with your sink so it knows when it's dirty, when it's clean and what it's got in it. Dishwashers know what they have inside - and when what's inside needs to be clean. Trash cans sort recyclables and know when they're full.
  However, much of this level of automation is only possible when the entire kitchen as a whole is aware. The above projects, particularly Counter Intelligence and CoolIO present fundamental portions of the Kitchen Sync vision. Much of the brainstorming to create these ideas has been through the establishment of scenarios: given a situation, what could Kitchen Sync do to help you? We present an example, and encourage readers of this paper to do so within their particular fields of interest.

Scenario: Chocolate Cake


Kitchen!" you announce, bringing Kitchen Sync out of its sleep.
"I'd like to make a chocolate cake for desert tonight."
"I'm afraid we're out of butter: the delivery was delayed. We can substitute olive oil though - you liked that last time."

  The sounds of John Coltrane fill the air as you assemble the ingredients list projected on the wall, with the Kitchen only occasionally advising you on where you last put the baking powder. You put a mixing bowl down on the counter, and look at the wall. The recipe is replaced with a grinning foot-high character in a tall cook's hat, who points at the flour. You pick it up.
  "Four cups of flour." You start pouring.
"One cup... two... three... three and half... and stop."
You put the flour back on the counter.
"You can put that away now. You won't be needing it. And it'll make the place tidy."
  Guess you accidentally engaged the "Mother" mode. Still, you continue with the rest of the recipe, mixing and stirring. The Kitchen reminds you of the substition, and suggests you use low-fat chocolate - a suggestion you cheerfully ignore, despite a twinge of guilt as it updates the calorie count at the bottom of the page. It's only a matter of sliding the cake into the pre-heated oven and waiting until the Kitchen reminds you to take it out. And if you're in the shower when that happens? No need to worry: your Kitchen will remember to turn the oven off, even if you don't.

Theory & Concepts

Cloud of Bits


We've discovered a lot of ways to look at human-computer interaction, and the very concept of data, in the process of working with Mr. Java and Kitchen Sync. The first is a common enough realization at the Media Lab: that we exist in a cloud of bits, a set of information about your current condition. Today we mainly think of bits as perhaps graphics, webpages, QuickTime movies. In Personal Information Architecture, we go beyond this definition and see bits as a spectrum, ranging from the fixed and quantifiable to the fuzzy and intangible.
  For example, I am six foot two inches tall. That's a constant and relatively unchanging bit. Continuing along our spectrum of bits, I have a body temperature, pulse and blood pressure that are measurable and recordable using a variety of sensors. Nearer the other end of the spectrum, I may be hungry, or want a particular kind of coffee today. These are far less fixed and easy to measure: hunger is a function of blood sugar, but goes unnoticed with sufficient levels of adrenaline in the bloodstream.



In our initial design for Mr. Java, we had thought about a number of ways to recognise users of the machine. One possibility, for example, was IR transmitting badges, previously used on the Penguin Demo to great success. Stuffed penguins wore nametags that emitted a constant infra-red signature, saying, in effect, "I'm Irv. I'm Irv". When Mort, the other penguin, received this, Mort and Irv would have a conversation, as they knew they were facing each other.
The problem with this is that there's no implicit context. We wanted to avoid the problem of a coffee machine that spewed out espresso whenever you walked through the 3rd floor kitchen. In the plans for Kitchen Sync, there are many tag readers and ways to identify objects: it's important to know the context in which this is happening.
  Another way to think about the importance of context is in thinking about sharing bits. Unless you know what you're looking for, it's hard to figure out whether the stream of information you're looking at is biometric data from a human being on a bicycle, weather data from a probe at Base Camp on Everest, or an I Love Lucy rerun. "Bits be bits." Once bits leave their creating environment, it's important to ensure that they're implicitly and unambiguously labeled.




Barcode technology has a number of advantages: it's cheap, and it's widely available. Commercial products frequently come with barcodes, enabling easy recognition.

  However, there are problems with current barcodes as implemented in the UPC standard - that is to say, the labels on nearly every product you buy at the grocery store. They don't distinguish between different iterations of the same product - one can of tomatoes looks like another can of tomatoes. That's fine at the checkout, but difficult if you're trying to tell how many cans you have in your larder. If we're trying to keep track of how old milk is, for example, it's important to be able to distinguish between two cartons of milk that have the same barcode but were purchased a week apart.
One possibility would be for every UPC code to have two separate parts: an identification portion and a serial number portion. For example, a particular bottle of apple juice currently has the barcode 2-26284-17513-9. 2-26284 refers to the company who make the product, as assigned by the UPC council. 17513 is the company's code for "8 fl. oz. bottle Pressed Apple Juice." 9 is a check digit to ensure the computer has read the numbers correctly. Expanding this to include a serial number - thus, say, 226284-17513.0170222 would enable tracking of that particular bottle's history, including storage, sale, and environmental conditions during shipping.
  The most important change in barcodes will come when barcodes are no longer seen as identifying objects in themselves but as links to information. There is a practical limit on the quantity of information that can reliably be stored in a physical label space: there is no limit to the amount of information that can be linked to that label.
  The next step will come when you purchase a product which has its own individual webpage. A can of beans will come with its own individual webpage detailing such information as production date, transport history, and time spent on the shelf, all entered automatically as it moves along the retail chain. Two apparently identical packets of rice you purchased on two trips to the supermarket can have entirely different histories of transport, storage, and origin. This incredible quantity of information will begin to appear for high end items - a web-accessible history of your car, say - but as time goes on will continue down the value chain.




Radio Frequency Identification has the potential to be one of the most widely used and powerful identifying technologies we have. Tags can be battery powered or unpowered, and can be purchased in a variety of sizes and configurations to allow for a wide range of uses. They are contactless, and require neither touch nor line of sight. In particular, they work through plastic, wood, and other materials, and can be set up to work in harsh environmental conditions, where barcodes or less robust equipment would be unable to function.Our classic example of this is under the spout of Mr. Java, where a polyurethane-encased reader is regularly subjected to 245F espresso. The kitchen is no place for fragile technology.

  There are a wide variety of RFID tags. The simplest work in much the same way a barcode does, giving out a single pre-programmed number when placed in the vicinity of a reader. It's also possible to store a limited amount of information on the tags themselves. Research is currently underway to produce tags that report information about their physical environment, such as temperature or position, in addition to readable and writable storage.
  The current (temporary) argument against RF tags is that of cost. Compared to a printed barcode, the cost is presently prohibitive except in harsher environments unsuited to barcodes or more elaborate packaging systems, such as the EZPass toll-paying system.. Currently, a simple tag has a lower price limit of approximately ten cents: too much to put on a packet of cornflakes, but an entirely reasonable way to track the history of a $300 jacket. A tagged world will arrive, one bit at a time.



Biometrics is the term used for identification of people by their physical attributes, such as finger print recognition, face recognition, and the like. Much research is being done on their possibilities for security identification and the like. However, many people feel very uncomfortable about being identified in this way. We have made a conscious decision to avoid working with biometrics in Kitchen Sync as much as possible. Nothing says Big Brother quite like the phrase "fingerprint identification", except perhaps "retinal scan."

  There is a regretful tradition among scientists of ignoring such sociological issues with regards to new technology. We feel it is better to look for alternative forms of recognition technology rather than ignore this discomfort, and the very serious issues behind it. For example, Mr. Java functions in a security and privacy conscious manner. No individual usage data is kept, while providing the possibility for users to keep track of their own coffee consumption.


IR tags

Infrared technology works in an intuitive way. If the transmitter (say your remote control) can't see the 'eye' of the receiver (say your TV), it won't work. We call this line-of-sight. It has drawbacks: constant broadcasting is expensive in terms of power. However, remote control-type intermittent transmit devices are cheap and powerful. Only working line-of-sight can be frustrating, as anyone who's tried to change channels from off to one side of the television knows. This can also be a feature: a computer that doesn't "see" an object until you hold it up front of it makes sense to the user, and is a simple way to download data.



The point of all of these different methods of identification is that there are a plethora of technologies with a variety of attributes that can be adapted to nigh-on any use. Encoding of information is not a single-choice arena, and includes ones that I didn't mention, from magnetic strips on cards to touch recognition technologies. In considering any technology design it is important to take into account the wide variety of methods in which relevant data can be encoded.


The next step after identification is associating preferences with the object identified, be it person, cup, clothing or penguin. These preferences fall into two categories: bits and atoms. This is an important distinction in defining what we're looking to do in Kitchen Sync. Looking back at what was seen as a kitchen of the future in the sixties, through the eyes of the Jetsons, we see a kitchen concerned with manipulating atoms: mechanical hands come out holding frying pans into which other mechanical hands crack eggs. We're not trying to deal with manipulating atoms, we're interested in bits.
  Mr. Java, for example, associates two sets of information with each tag, with each individual's cup. One set is how they like their bits arranged, and the other set is how they like their atoms arranged. Arrangements of bits refers to, for example, the latest news from NPR, or the current weather report, or the sports scores. Mr. Java goes out across the web and picks up the RealAudio feed of your choice, and plays it while your coffee is being made. Importantly, we've provided the facility for users to create their own RealAudio or .au files by linking to a URL, which could contain their own personalized stock portfolio, or their messages. This also provides them with the means to track their personal coffee consumption without Mr. Java storing such personal individual data.
  This concept of preferences for bits and atoms is a powerful concept that can be extended far outside the realm of the kitchen. A car could recognise you through your key or key fob, and adjust the seats and mirrors (atoms) and the radio (bits) to your preferences. A washing machine could recognise clothing going in through flexible RFID tags and know you might not like your red T-shirt in with your otherwise white wash. All of these are examples of simple but powerful operating concept:
Within a context, recognise and associate preferences.




The field of domestic media is one that will truly come to fruition in the next decade. We currently have use of barcode and RFID technology almost entirely in industrial and commercial settings. As with the computer, the pager, and the microwave, we can expect this technology to start to be integrated in to home life.
  We're frequently asked how long it will be until we see Kitchen Sync technology entering the marketplace. Different pieces of technology will no doubt take different periods of time to really become useful. Mr. Java is currently starting to move into the marketplace, with the help of generous support and interest from EDS and Acorto. MicroChef / PC Dinners technology is already arriving in the home in a variety of forms, notably in the Japanese market. Longer term projects include Counter Intelligence and CoolIO, which currently appear to be in the ten-year range, as they require greater investment in an infrastructure and information base. Both will be effected by the degree of growth of shop-from-home services such as Peapod.
  We are currently at the Model T stage of computers. The Model T is still known, seventy years later, for being available in any colour you wanted, so long as it was black. Computers are currently available in any size you want, so long as they're a box. You can buy boxes that sit under your table, on your table, in your bag or in your pocket, but they're all boxes.
  Kitchen Sync is one example of technology designed to not be a box but instead be a coffee cup, a counter, a toy. A vast quantity of research at the Media Lab is engaged in making non-box computers. Kitchen Sync and the Counter Intelligence SIG show an extremely exciting and viable area of research in this arena with a great number of both commercial and research possibilities

Credits & Acknowledgements

Picture Credits

PC Dinners Webb Chappell
Mr. Java Webb Chappell
Mr. Java by Hour http://mrjava.media.mit.edu/byday.gif
Counter Intelligence Brad Geilfuss
Mort & Irv Webb Chappell
Sandy, Jofish,Mort Raffi Krikorian
Barcode http://www.milk.com/barcode/
Fingerprint http://www.linguafranca.com/9804/images/fingerprint.gif
IR Badge Webb Chappell
Rosie Jetson http://www.cybercomm.nl/~ivo/photo_ROSIE.html



My primary acknowledgement is to the members of the Personal Information Architecture group for all varieties of help from technical to emotional. Particular thanks must go to Kristin Hall for invaluable editing, friendship and support, Niko Matsakis, without whom none of this would have been possible, and my research advisor, Professor Mike Hawley - despite, and perhaps due to, his love for appalling puns in project names. A note of thanks is also due to Matt Lau for the name "Counter Intelligence."
  Another serious acknowledgement is due to those I live, work, rehearse and play with for the amount of time and support they have given me while I worked at the Lab instead of being with them. My family and my girlfriend, Allison Wolf, have given me constant, incredible and invaluable support, and Roadkill Buffet and my various theatrical endeavors have kept me sane. Without my friends, I would have and be nothing.
  A great deal of thanks is also due to the sponsors of the Things That Think consortium and Counter Intelligence SIG for providing funding for this research.