Engineering Projects/InteractiveSystem/Howard Community College/fall2012/p1-504-tjgr

Please delete everything in italics after filling it out.

Project documentation focuses on how the team organized itself to tackle the unknown. This is a creative process. The details learned should be captured in tutorials. Project documentation should focus on how the team organized itself to step out into the unknown. Failure is common. The single most effective way to increase engineering respect is to document failures in a creative way that stimulates, inspires and helps others continue the project. Projects are never done. There are always loose ends. There are always decisions that might have been made more clearly. There are always other alternatives that might have been considered. None of this stuff appears in a tutorial. It does appear in the team documentation.

Electronic Sections Expected

Problem Statement

To create an interactive motion and audio system using Xbox 360 Kinect, a projector, and a computer that will project a graphic design with an audio that responds to a person's motion while also projecting the person's silhouette.

Here is a visual demonstration of the problem statement. Although the example in the video is advanced, certain elements of our problem statement are clearly seen such as the graphics being interactive with the person, or the subject, displaying the silhouette, and being sound interactive as well. Example

Team Members

Tamara Barovskaia

Gratien

Summary

We plugged a camera and a projector into a computer. Using Po-Motion 30-day-free-trail, we were able to project a scene onto a wall. We put our hands in front of the camera and the projected scene reacted to the motion. However, the range of motion of the camera is very limited. We are replacing the camera with an Xbox 360 Kinect motion sensor. Using certain software, we were able to get the Kinect to work with a computer. The Kinect has several advantageous properties such as having superior technology, full-body sensors, audio-re-activity, and the flexibility of being used with a wide variety of software. Using Somantics Sparklers, we were able to have an interactive system with the Kinect and the graphic. No work has been done on creating a complimentary sound interactive system.

Poster

[Motion Art]

Story

The team members split up into 3 components. Tamara Barovskaia researched software to create a graphic that is continuously in motion. Gratien researched software that can connect the graphic and the kinect, so that when projected, the graphic would be motion interactive. Jessica "hacked" the Xbox 360 Kinect to get it to be recognized by the computer and be motion sensitive.

We discovered Somantics software and incorporated it in our system. Somantics works well with computer and Kinect. The interactive software is available for free download for Ipads, Windows, and Macs on the homepage. Various tutorials are also available. Somantics reacts to a person's motion, mostly focusing on the limbs. It picks up the input and produces an output on the computer of the person with the interactive graphics. Although, Somantics produces interactive graphics, there is no background necessarily. For example, the background will be of wherever the person will be. If the person is in their room or a classroom while using Somantics, the output will display the person, the interactive graphic, and the room the person is in.

Since we worked with Windows, here is the Somantics for Windows download Somantics Download - Windows

Here is a list of Somantics sample graphics and their tutorials. We used the Sparkles

Although we have not incorporated the interactive sound system in our project, there is a software available called Wekinator that might be helpful. A couple of very important applications of Wekinator include:

Creation of new musical instruments

Create mappings between gesture and computer sounds.

Creation of gesturally-controlled animations and games

Control interactive visual environments like Processing or Quartz Composer, or game engines like Unity, using gestures sensed from webcam, Kinect, Arduino, etc.

Creation of systems for gesture analysis and feedback

Build classifiers to detect which gesture a user is performing. Use the identified gesture to control the computer or to inform the user how he's doing.

Creation of real-time music information retrieval and audio analysis systems

Detect instrument, genre, pitch, rhythm, etc. of audio coming into the mic, and use this to control computer audio, visuals, etc.

Creation of other interactive systems in which the computer responds in real-time to some action performed by a human user.

Anything that can output OSC can be used as a controller Anything that can be controlled by OSC can be controlled by Wekinator

Here is the link to download Wekinator

The following are the requirements to download and operate the Wekinator: Chuck version 1.2.1.2 or later, and Java version 5 or later.

Here is the link to download Chuck

Here are the instructions to navigate the softwares and Wekinator, how to operate them, how to use the Kinect with the software

Instructions 

How to Make a Custom Interactive Graphic:

This can be accomplished by using Processing programming.

There are 2 different methods:

Software needed for the first method: 1. Processing.js 2. Anything.html 3. Anything.pde

Software needed for the second method: 1. Processing.js 2. Anything.html

All of these can be downloaded from Processing

What is programming. js? Processing program that makes data visualizations, digital art, interactive animations, graphs, video games, etc. using web standards without plugins. To make a custom graphic, go to the "Reference" tab. There are different categories for the codes used by processing.js. For example, to use a specific shape for a graphic, go under the "Shape" category and select the code for whichever shape needed.

Material List

1. Computer: Dell, PC, Windows. Quantity: 1. Cost: Provided by Howard Community College

2. Projector: EPSON. Quantity: 1. Cost: Provided by Howard Community College

3. Xbox 360 Kinect Motion Sensor. Quantity: 1. Cost: Provided by Howard Community College Engineering Lab

4. Software. Quantity: 5. Cost: Provided online for free download.

Software List

The following is the software used to "hack" into Xbox360 Kinect

1. OpenNI

2. Nite

3. Kinemote

4. Sensorkinect-unstable

Software used for the interactive graphics:

1. Somantics - Sparklers

Time

52 hours.

Next Steps

We were able to hack into the Xbox 360 Kinect and have it react with a computer. We used Somantics Sparklers as our interactive graphic. However, there needs to be a sound interactive system. The next group can work on creating a sound interactive system that will compliment the interactive graphic, possible using the Wekinator software. The next group can also try making custom graphics using Processing. Various categories for graphics are provided on Processing including their visual examples and programming codes.

This project relies heavily on the ability to understand programming since almost the entire system is software based. There is a great amount of difficulty in downloading, installing, extracting, and programming Wekinator software in addition to "hacking" the Kinect. This project demands patience and dedication. To assist the programming process, read the instructions carefully and slowly, do not skim through them.