Grab and Drop!

I got my inspiration from one of the readings that discussed how current technology does not respect our hand’s capacity to perform so many different tasks. So, I created a game that lets the users use their hands like they do in real lives. The users should move their hands, grab an apple on the screen, and drop it so that the apple falls into the basket that moves left and right with constant speed. The goal of the game is to put as many apples as possible in the basket.

Technical Methodology:
I used color tracking to locate the position of the hands. One glove has a red sticker attached to it and the other one has a blue one attached. [Figure 1] In the beginning, the processing loops through all the pixels of the video and finds the most red and blue points. After that, the program only checks the pixels near the first location, which makes the program not only more precise but also faster.  However, there were times that the program lost track of the hands. So, I changed the code so that it checks all pixels of the video whenever the basket reaches the endpoint of the screen. This way, I tried to minimize inconsistencies of color tracking. 
Moreover, flex sensors are used to detect the closing of the gloves. [Figure 1 & 2] It is attached to the middle finger and palm of the glove. Whenever the user closes their hands, the flex sensor is bent. Since flex sensors are analog sensors, I set a threshold value. If the value is lower than the threshold, Arduino sends “1” to processing, which indicates that the hand is closed.

[Figure 1]
[Figure 2]







Physical Construction:

At first, I dyed the whole glove as shown on the left. This was inappropriate because the area of the red or blue area was too big. Every time the program checked for the reddest (bluest) spot, it chose random points of the glove, making the color tracking jumpy. Sometimes, the program couldn’t detect the movement of gloves because the red area was too broad. The location of the reddest pixel didn’t change unless the user moved their hands significantly. Therefore, I decided to have a small piece of red and blue sticker on the glove as in [Figure 1].

Below is the schematic and prototyping shield of my project. 


Flex Sensor:
I used flex sensor to detect closing of hands. Previously, I tried to use separate metal pieces like switches. But, it turned out to be very unstable. Therefore, I used flex sensors and calibrated the value of the sensor to set the threshold. It worked pretty well until it broke during the show. I did not know that flex sensors are fragile and that they can change values easily. I did layers of heat shrink tubing to reduce the risk of breaking, but it turned out to be a failure. It would have been nicer if I used a stabler and more durable sensor since gloves were moving a lot.


Processing Main Code:

Processing Objects:

Arduino Code:


Problems and Possible Improvements:
1. My project kept detecting objects in the background during the show because people in different colored clothes were walking behind and screens in front of my project had colors, too. Kinect would have helped solve this problem. Since Kinect can detect proximity, I can only detect objects that are close enough. Although I wasn’t able to use it because there weren’t enough cameras, I would like to try it next time. 

2. The aesthetics of the project wasn’t so nice. Graphically, I did not have enough time to refine it because it took me too much time to realize the functions. I think it would be useful to learn photoshop etc. that would let me make graphical objects for my project. Physical construction of the gloves wasn’t appealing too. I tried to make it look neat by covering the flex sensor with a fabric and sewing it into the glove, but it still looked quite crude. I should work more on the design in order to attract and provide pleasurable experience to the users.

3. I wasn’t able to use full screen because using full screen meant webcam capturing a wider range of area. When I did it, the program was distracted by other objects moving behind. Therefore, I had to use a smaller display. If I considered this problem earlier, I might have been able to find a way to fix it. I learned how exhibition space and environment could have a huge influence on functions and effectiveness of projects.

User Testing

Feedbacks (everyone mentioned a same problem):

  1. The tracked color should be more distinct so it does not track off to another object.
  2. Better construction of the gloves. (The sensor falls off too easily)
  3. The color tracking is shaky which makes it hard to catch the apple.

Progress Report

I didn’t know how to release only one apple when I’m holding two apples in both hands. When I released one hand, two apples dropped at the same time, which was not what I intended. I solved the problem by adding a break function in the end, which escaped the loop, allowing me to drop just one apple.

Remaining Questions:

The movement of the hand is going to be shaky, because the color detecting function is not perfectly stable. I have to research on how to stabilize the movement of the hands. I’m trying to check only the parts that are near the first spot that was blue or red, so I can check less space and make the program not only faster but also less shaky.

Final Project Summary


My project is a game that has a goal of making a farmer happy by dropping apples to him. However, instead of using a keyboard or a mouse, the user uses their hands to drop the apples. The idea of farmer and apples might change, but the structure of the game will remain the same.

Technical Plan:

I’m going to use two gloves, each colored red and blue. There is going to be a flex sensor attached through the middle finger and the palm. This is going to be a controller for players. The program is going to identify the hands by its color. I will flip the location of the gloves horizontally in order to make it like a mirror so that players can more easily identify their hands. I will check the analog value of the flex sensor when it is straight and bent. By doing so, I can detect when the player is closing their hands.

Block Diagram:

  1. Entire Project

2) Arduino

3) Glove

The processing code will get constantly updated with the information of the location of the gloves and whether the hand is closed or not. If closing of the hand is detected, the program will identify the location of the glove that is closed. If there is an apple in that location, the player will hold the apple. And if the player opens the hand, the program will again detect the location of the glove. Then, an apple is going to drop from the location of the glove.

Additional Parts:

  • TV Screen
  • Camera
  • Glove (Black or white preferably)


  1. It will be hard to make all the components of the game working together well. The detection of the glove, identifying the closure of hands, catching and dropping the apples should work simultaneously. Combining these three components will be my first challenge.
  2. Physical construction of the gloves would be my next challenge. Since it will be moved around, the connection of the parts should be secure. Since I have less experience working with real objects, this will be the hardest part.
  3. Making the project aesthetically pleasing will be difficult. I want to make my game look like a completed piece, rather than a crude prototype on the screen. I would have to learn how to work with graphics and images a bit more.
  4. In the program, I should watch out for other blue or red objects on the screen. I should find out a way to exclude those factors and only detect the ones on the glove.
  5. It is hard to expect how users’ will use the glove. They might use it in a way that does not meet my intentions. Error might occur in that process. I think I should have instructions on the side so that users will know how to use it. I want to make it well so that it does not disturb the players from enjoying the game.


I made a glove that has the flex sensor and a colored tape attached to it. I got the program detect the movement and the closure of the gloves. I couldn’t incorporate it with the game due to time constraints, but I got the functions working. However, the flex sensor was pretty jumpy, so I have to figure out how to stabilize the readings. Also, I realized that I should also color the back side of the glove because the closure of the hand hides the colored tape attached on the palm.

Final Project Idea


I want to make a game that can be controlled using the hands. I haven’t decided on the details of the game, but I wish to use hands as the controller and also detect grabbing action of the hands.

Technical Requirements:

My program should be able to track the movement of the gloves all the time and react instantaneously. Quick reaction is a key to all kinds of games. I also want to detect the action of grasping with the use of Arduino. I will attach pieces of metal on each fingertips and a larger piece on the palm. The metals will act as a switch, so every time pieces of metal touch the one on the palm, the program will identify it as a grasp. (I might use conductive fabric instead) I might or might not have the function that identifies the opening of the grasp. This can be easily achieved if grasping motion can be detected consistently. (It is the same as dealing with buttons)

Equipment Needs:

My project would require a TV screen and a camera that can be attached above the screen. I also need to have a pair of glove and metal or conductive fabric to attach to the glove.

Block Diagrams of the Physical Construction:

5 Possible Difficulties:

  1. I should make sure that the processing identifies the movement of my hands correctly. To do so, I wish to use gloves of distinct color so that it is easier for the program to identify the gloves. However, it might be confused if an user is wearing a clothes with a similar color. So, I have to come upwith a way to cover the user, not blocking their vision, with something of different color.
  2. Another difficulty would be correctly identifying the grasp movement. Everyone has different ways of grabbing things. Metal pieces, or fabric conductor, should properly touch each other when the user does a grasping motion no matter who the user is.
  3. I have never worked with extra screen and camera. Connecting my laptop, Arduino, TV screen, camera and the glove with stability will be my challenge. I struggled a lot assembling all the parts together in my midterm project and I want to avoid that struggle this time by carefully planning out the physical aspect of my project. I would have no trouble making a box this time, but I would have to connect the Arduino and glove well because it is hard to know in advance how long the wire connecting the two should be.
  4. I have to make a game that is both fun and meaningful. I used to focus mostly on how to make things work. Now that I have to a project, it is hard to figure out what I really want to make. I can think of many games that use the functions I described above, but there are none that actually makes me excited. I think I need some more time to think about it.
  5. In the program, I want to have a virtual hand that follows the movement of the gloves. When the user makes a grasp, the virtual hand willtry to grab some object in the screen. I am not sure how to make the movement of the virtual hands grabbing an object smooth. I would have to search for other projects that have similar function and learn from them.

Response | Digitize Everything

Information on the Internet does have many useful applications. I can tell from my own experience that Google Translate has improved tremendously since I started using it. Before, it only showed translation for each word without considering grammar or context. Now, although not perfect, it understands idioms and phrases that mean something absolutely different from the words that form them. Also, the computer can teach themselves in an extreme speed that human cannot imitate. By constructing efficient algorithms, you can come up with better strategies for sports, purchase, and driving routes etc.

However, there is a flip side to all these great advantages massive data have. Since it is so cheap to create information, there are a lot of fake or wrong information, too. Since people get most of their information from the Internet, a group of people producing fake information can easily influence other people. Therefore, when it comes to matters that can shape your value, it is very crucial to take in information carefully. However, my experience tells me that extremists tend to write more in the Internet. Therefore, those comments tend to flood the Internet, hiding other sensible comments. Fixing this problem would be a big challenge for today’s society flooding with easily produced and reproduced information.

Moving Robot Arms Code

Processing gets value of the potentiometers through serial communication and changes the degree of the arms. I mapped the value of the potentiometer to range from 0 to the angle limit so that the arm does not rotate too much.

For Arduino:

For Processing:


Response | Design Meets Disability

I learned that design for disability does not always have to be universal. People with different tastes and needs will prefer different products. However, people usually ignore this fact because they think of disabled people as a homogeneous group. I believe machines like 3D printer or laser cutter would enable manufacturers to make specific products for individuals. These machines make it easier to customize products because all you need to do is make changes to the blueprint. Size, color and shape can be decided by the customer before the product is actually made. Then, people will be able to utilize their aids as a way of expressing themselves.

Also, it was interesting how bad design can sitgmatize people. Trying to make these ‘wears’ invisible, which never succeeds, might make people think that disability is something to be ashamed of. Mental burden that this perception puts on people might be more stressful than subtle functional inconveniences. Therefore, it is important to balance the tension between design and functionality in these products. Before, I used to think, like other design engineers, that functions come first. However, reading this article made me think in a different perspective.

Game Code

Following is an Arduino code:

Following is a Processing code:


Reach the End


  1. Click Start in the beginning.
  2. Small black circle is the player.
  3. Small black rectangle is the endpoint.
  4. Use two potentiometers to move the player. (left one for left/right, right one for up/down)
  5. Reach the endpoint without touching the colorful circles => Complete!
  6. If you touch one of the circles => Game Over!
  7. Click on the mouse to restart the game.

Problems Encountered (some still remaining):

  1. Serial Communication stopped when I tried to restart the game. I fixed it by sending ‘A’ to Arduino when I restart the game. This restarted the communication between Arduino and Processing.
  2. Obstacles used to be randomly generated, so, sometimes, it was concentrated on one space, making the game too easy. So, I separated the screen into four sections and generated same number of circles in each section. So, obstacles are more spread out on average.
  3. The value of potentiometer has fluctuations, so the player circle does not stay on one place but shakes.
  4. I focused on realizing the functions, so the code itself is pretty unorganized.