Hi, I'm Sam.
a face recognition system at PPCU
About
Facebook is doing some fancy stuff, Google is doing some groundbreaking research. Why shouldn’t we join? All resources are publicly available to make a working face recognition system, and this case study is about implementing & deploying something that could make our everyday life easier a bit.
Purposes:
The goal is a completely safe upgrade to our current security system that lets people who have signed up pass without their ID tag, using only their faces for identification. Why? For fun, for experience. This project is led by no financial motivations, but for pure science.
It is hard to trust such a system, especially when it is intelligent… This is why we have decided to make the whole source code along with the design steps available for anyone.
What Sam is:
- a camera that looks at newcomers
- a trained recognition system that guesses who they might be
- a database of registered people who can be invited by Sam
- an activator that moves the turnstile
What Sam isn’t:
- a tool for counting people that logs their activities
- a tool for faculty members to check students’ presence
- definitely not an AI with personality (that would be cool tho.)
There are 3 main steps:
- Pair IDs and faces
- Train a recognition algorithm
- Make that turnstile turn
In order to decrease the possibility of Sam letting in people who otherwise could not come in, we have to train it to distinguish people better. That’s the reason behind the current phase.
Training Sam
HOW DOES A RECOGNITION SYSTEM WORK?
If you are curious about core concepts and what should the Neural Network (a.k.a Sam’s brain) learn - take a look at my online slideshow which demonstrates completely ridiculous usage of the Parallax effect.
Embedding Sam
This is the most tricky part. We have something that works with a GTX 1080 Ti… fine… what doesn’t work with a 1080 Ti? The true challenge is to find some computing unit that is small enough to align with the current design of the entrance system, while being able to compute a Convolutional Neural Networks responses (the brain of Sam).
By courtesy of Zsedrovits Tamás, we got a Jetson TX1 board to run Sam’s recognition software onboard. Currently we are working with PyTorch 0.3 that is still in beta, but extremely convenient for fast prototyping. Our plan is to deploy the best models in Caffe2 which is rather focusing on running things fast.
Connecting Sam
Currently the cam is recording in show-time, and Sam processes the recorded frames by selecting whether there are any face on them. If no recognizable face was found then to make room for more valuable memories “empty” frames gets removed automatically. The ID matching happens via matching a log file of the card readers.
To make this whole process smoother and less energy consuming the ID-face matching will occur instantaneously by making Sam listen on the card readers directly.
Deploying Sam
Once Sam is trained to map images into a space where everyone has a separate cluster we are ready to deploy the algorithm. This means that every computing step will be optimised for inference, to make face recognition more smooth and reliable. Our workflow follows the pattern suggested by the Facebook AI Researchers (FAIR).
Workflow:
Personal interface
As we have passed the initial benchmarks now we are working towards a trustworthy interface where one could sign up for the services of Sam. Here one can manage all the accessible data and services assigned to the user.
Basic features:
- Visualize the 2D embeddings (t-SNE) of the personal database
- Manage recordings and logs
- Suggest modifications and report malfunctions
Microcontroller
A closely related project to “Connecting Sam”, where a special fail-proof secondary system handles
two main cases:
- If one is required to use the ID badge to be authorized, we should log it to improve performance.
- If one is recognized by Sam we should mimic the same signal as the card reader would emit to preserve the security system’s integrity.
Tomi
His job is mainly to organize photos and data to be easily stored and retrieved from database. He is also developing the website that shows you all photos of you and gives you the opportunity to help us by filtering out photos incorrectly classified as you.
Marci
Hardver guy, knows a lot about access control systems, RFID technology and electronic stuff. Survived at least 9 electric shocks so far.
Levi
He is the brain surgeon of Sam.
He has the main role in designing and implementing the circuitry that communicates with the security system. Practically he is the guy who plugs random wires in your microwave oven with a laptop on the other side and hacks it. Possibly may have solutions for problems you didn’t know you had before meeting him.
Andris
The ultimate super-user of all time. The classic admin guy: he is above all the network protocolls, black-green terminal and stuff. When it comes to deploying linux systems - like what Sam is running on - you can approach the same problem in almost infinite number of directions… Andris knows the best ones.
Máté
It is completely up to him whether users personalize and depict Sam as:
- some laser eyed, bear riding, two-handed katana swinging old-school terminator
- or as a friendly bot that helps grannies cross the street… like GlaDOS
Joke aside, his work in design is as important as the backend of the project, especially for making Sam more like a thing that you would like to face when you come in.
Csabi
Runs the AI business in the background, practically maintains the edge and the cloud services. Originally began the project alone but soon many cool guys appeared with invaluable perks.
Contact us
Many cool guys have taken role in this project so far, and that’s why projects like this can happen here.
If you have any questions regarding the
Python / C++ code base || Hardware implementation || Machine Learning theoretics
or you would like to join us feel free to contact us at the address below