During the 36 hour hackathon BitCamp, Yana Gevorgyan and I remade the classic word game Mad Libs with a twist: by incorporating image recognition. In a traditional game of Mad Libs, a player would have to come up with spontaneous words to fill in the blanks for a story. In our reimagined mad libs, players scan objects around them as nouns to be filled into a story.
MadLads was selected out of 68 submissions as the Best First-Time Hack. View Devpost Submission
Role: Game development, front-end development, user flow, UX design, visual design
How it Works
We used ml5.js, a pre-trained, web friendly image recognition library that can identify 1000+ objects. We applied the model to the environment facing camera on the phone to constantly identify objects. We also played around with the probability of the recognition and decided to set the accuracy quite low so that the app will sometimes report hilarious observations. Lastly, we used p5.speech to narrate each and everything that the machine sees to add to comedic value.
Once the user selects all objects, the objects are plugged into the first passage of Francis of the Filth, a book that we both enjoyed.
Sketching + Ideation
A simple and affirmative user journey
We designed an onboarding screen that contains a sneak-peak into the camera-based gameplay of MadLads, with a clear call-to-action start button. During the game, we prompted users to scan a variety of objects before confirming on a selected one, as well as pacing them throughout the journey by showing how many words they have left.
Using visual design to guide players
Because the main focus of the game is the camera display, we kept the visual design simple yet inviting. On the start-screen, we established an inviting on-boarding state that shows users the camera-based and AI integrated gameplay to prepare them for the game to come. The app doesn’t immediately choose the first thing the user decides to scan, instead, they can experiment on a variety of objects and the press a confirm button once they have decided on an object.
Closing Thoughts
With MadLads we wanted to create a fun and experimental game that anyone with a smartphone can play. Using my pre-existing knowledge of design and development, I was excited to bring image recognition and web speech into a game that many of us are familiar with. MadLads is just the start of many ML-related projects I wish to explore in the future! 
BitCamp 2019
Demo Day

You may also like