This is a research project that aims to improve working memory in deaf children. The system uses a Microsoft Kinect camera to collect data of American Sign Language (ASL) gestures, and translates the gestures into text using the HTK Speech Recognition Toolkit. Children follow a tutor and play a 'CopyCat' game along the way. We are working on developing a documented dataset and have undertaken multiple deployments of this tool in Atlanta.
I developed a socket connection between the HTK Speech Recognition Toolkit backend and the Visual Studio frontend interface. I also designed the scheduling of the camera captures, the entire data collection workflow, as well as the frontend user interface. I also conducted HCI work on student-system interactions.
I have recently been very interested in exploring how we can push the boundaries of our interactions with physical objects and artificial intelligence and explore their intersection in novel use cases. For a long time, I wanted to bring together my passion in playing the violin and machine learning - and so I came up with this idea for designing an AI that learned how I play the violin in my Indian classical (Carnatic) style and improvised with me. Our team was successful in building such a system that I could actually jam and make music with - but we also took it one step further, and used a neural network to get the AI to generate its own classical music compositions. We reached a point where the AI actually played better violin than me! (While I had 12 years of classical music experience, the AI was trained on 150 years of classical music knowledge.)
This project won the FIRST PRIZE at the MHacks 2016 hackathon at the University of Michigan, Ann Arbor. The project also received initial seed funding from the 1517 Fund. Please see the Devpost website or contact the author for details.
With the coming of the IoT age, we wanted to explore the addition of new experiences in our interactions with physical objects and facilitate crossovers from the digital to the physical world. Since paper is a ubiquitous tool in our day to day life, we decided to try to push the boundaries of how we interact with paper.
We develop a workflow that enables a user to convert ANY piece of paper with text on it into an interactive touch interface that communicates with nearby mobile and desktop devices.
This project won the Philips-MIT Award for the Best Focus-Based Document Summarization, at the HackMIT 2016 hackathon at the Massachusetts Institute of Technology. Please see the Devpost website or contact the author for details.
We apply machine learning on crime statistics in Philadelphia to determine the safest walking path from point A to B. We used a Support Vector Machine to perform the classification and developed a front-end supported by Google Polymer. We developed this tool in less than 36 hours at the University of Pennsylvania.
This project was presented at the Pennapps 2016 hackathon at the University of Pennsylvania. Please see the Devpost website or contact the author for details.
This HCI study was conducted in collaboration with Google. We used the Google Cardboard and the Google Expeditions Field Trips Application to study the potential of creating affordable virtual reality-based learning experiences for children in low-resource contexts.
The Online Learning Community is a holistic model to incorporate mobile technologies into a larger online learning environment, primarily geared towards a low-resource context. Our model is centered on the design of an online, learning community with participation from students, tutors, NGO staff, and curriculum designers. Our software includes a suite of tools that have been used as design probes and prototypes for the study of the space of education in low-resource contexts. We have performed multiple field deployments of our project in diverse underserved communities, and conduct an ethnographic inquiry of the impact of these tools on teaching and learning. We also study the feasibility of these ICTs for Education.
Please note that none of the software is consumer ready. This software is part of field studies with a strong focus on research, and our tools are not consumer-ready products. My advisor for this project is Neha Kumar.
Unsupervised Harvesting and Utilization of Recognizable Acoustics: This is a software toolkit that allows researchers to analyze dolphin audio files using various machine learning techniques. I worked with the Wild Dolphin Project since 2014 on developing different kinds of technologies to aid marine biologists and divers. The toolkit consists of an Ocean Noise Detector (which runs the raw dolphin acoustic data through an FFT, to filter out the unwanted segments of data), a Discovery Tool (that analyzes animal (here, dolphin) vocalizations and discovers patterns in the data), and a front-end display interface to view the signals. One instance of the front-end interface was developed using the Noldus Observer XT event logging software (shown below).
Co-developed a mini-course, which is hosted on the edX platform. See this link for the full course. The course introduces our model for a mobile learning platform for underserved communities, and subsequently describes our research methods, research findings from prior work, and future plan. This course was created as part of the Flipped Presentations for the Third Annual ACM Conference on Learning at Scale 2016, in Edinburgh, UK.
"A driver, upon entering her/his car, is asked to speak out a few sentences to the sensors mounted on the dashboard of the car. The device (with the support from an Arduino and Raspberry Pi set up) analyzes the input voice, and determines if the driver is fit to drive. If she/he is not fit to drive, the tool dispatches text messages to close family members/friends alerting them of the driver's location, following which they may take action."
This project won the IBM Bluemix Award for the Best Use of the IBM API, as well as the Runner-Up Prize at the HackDuke: Code for Good 2016 hackathon. Please see the Github website or contact the author for details.
Worked as a User Experience and Mobile (Android) Application Developer for three months with an NGO to develop a math learning application for students in low-resource rural schools in Tamil Nadu, India. The tablet application also functioned as a design probe to understand math aptitude levels of students in these schools. The UI design of the application was iterated upon over a period of six weeks.
This application is now currently used by two NGOs in 12 different villages across the state of Tamil Nadu, India. Currently, I do not own the application, and it is not in the open source domain.
Please see Github for the code or contact the author for details.