Magic Pencil

Harys Dalvi

Project from November 2020, written April 2021

One day, I decided it would be cool to control a graphing calculator without touching it. With my graphing calculator of choice, Desmos, loaded online, I got to work. This is the story of how I succeeded, but decided that it is more convenient to type on a keyboard than to control a computer without touching it.

My initial idea was to use Arduino. Specifically, I wanted to make a joystick mouse controller so I could control the motion and clicking of the computer mouse without using a physical mouse or trackpad. However, my Arduino didn't have permission to move the computer mouse, so I had to find another way. The other ways were more interesting anyways.

Inspired by Stephen Hawking's machine, I looked into the Python OpenCV module so I could use color tracking and a camera to sense my motions. The module allowed me to see how many pixels of a certain color range were in the vision of the computer camera, and where those pixels were. My first plan was to find my hand using skin color. I would move my hand to move the computer mouse with the python pyautogui module, and make a fist to click. For typing, I would use the Mac onscreen keyboard to click on the keys I would otherwise press.

There were multiple problems with this. The worst one was that my skin blended in with the color of my table, so it was difficult to detect accurately. Also the color range I needed was highly dependent on lighting, angle, and time of day; I didn't want to be recoding the color range every time I wanted to use the program. So having eliminated red, I moved on to the G of RGB.

Instead of using my hand, I used a green pencil (my "magic pencil") to move the mouse. I took the "center of mass" of all pixels in a green range that I defined based on the color of the pencil and made the mouse follow the pencil.

There was a problem: unlike my fist idea with my hands, I now had no way to click. I modified the code so I would move the mouse in the direction of that position of the pencil, instead of teleporting it to the location of the pencil. This worked for moving the mouse. If the pencil was still in the center of the screen, this meant that the mouse was stationary. When there weren't sufficient green pixels (if I removed the pencil), I took that to be a click.

Here is a selection of the Python code for how I followed the green pencil.

low_green = np.array([35, 80, 50])
high_green = np.array([90, 255, 255])
green_mask = cv2.inRange(img, low_green, high_green)
green_n = len(green_pos)
if green_n > 650:
    green_pos = green_pos.T
    temp = coords_G
    coords_G = [int(np.average(green_pos[0])), int(np.average(green_pos[1]))]
    pyautogui.moveTo((coords_G[0]+temp[0])/2.0*1440*1.5/1280-150, (coords_G[1]+temp[1])/2.0-150, duration=0)
    clicking = False
if not clicking and not coords_G[0] == 0 and green_n < 400:
    clicking = True
    coords_G[0] = 0
    pyautogui.click()

Although this worked, it was extremely tedious to type anything with a graphing calculator. I had to click the keys one by one on the Mac onscreen keyboard, including the shift key for things like ^ if I wanted to write x^2 for x squared. Also, in order to click, I had to very quickly move the pencil out of view of the camera. Otherwise, the mouse would move down as I moved the pencil, causing me to click the wrong key.

My first solution was to use another pencil, blue this time. The mouse would follow the green pencil, and click based on the presence or absence of the blue pencil. This worked. However, it was still hard to control the pencil precisely enough to type.

I remembered my original goal was to use a graphing calculator, not to control the computer fully. Therefore, I changed my model to something more specialized for graphing. I had a list of terms I could type into the graphing calculator, such as "r=" (for polar graphing), "x" (for Cartesian graphing), "pi", "e", "sqrt" (for square root), and digits 0-9. I would list one of these terms in green at any given point, and one in blue. I could move the green pencil into boxes to the right and left of the screen to cycle both the green and blue term forwards or backwards in my full list of terms. Then I could use a box to type the green term, and a separate box to type the blue term, depending on which reached my desired term first as I was cycling. Finally, I had boxes for delete and enter.

The main problem with this approach is that the detection of the green pencil remained inaccurate, often fluctuating. This frequently led to unwanted double clicks if the computer mistakenly thought that I had moved the green pencil into the center box (which was a neutral space without any command) and then back into the box I was actually in (the delete box, for example).

That's how I found multiple ways to control a computer without touching it, none of which were remotely convenient. Now, when I want to use Desmos, I use my hands and type on the keyboard. Still, I enjoyed working with my magic green pencil and trying to do what Stephen Hawking seemed to do so effortlessly.