Detecting Bias in Autonomous Systems
Mission Statement
Our Mission is to mitigate the bias of both implicit and historical racial, gender, and ideological biases in hiring systems that utilize either humans or artificial intelligence in its decision making.
Project Timeline
National Conference on Undergraduate Research
April 15, 2023
On the same day as the MPSA conference, nearly 800 miles away, Team DeBIAS was also presenting at the University of Wisconsin at the National Conference on Undergraduate Research. The mission of the NCUR is to promote undergraduate research scholarship and creative activity done in partnership with faculty or other mentors as a vital component of higher education. Similar to MPSA, the conference followed a poster presentation style, where our discussants and other participants could look around the different posters and ask us questions. The conference went smoothly, and Team DEBIAS is grateful that we had the opportunity to present in-person at WIsconsin! Fun fact, our inclusion in NCUR's lineup makes Team DeBIAS the very first research team in Gemstone's history to attend two in-person conferences simultaneously!
Our experts!
Pictured (top to bottom, left to right): Dr. Natalie Tran, Dr. Juliet Aiken, Dr. Sonia Kang, Dr. Christo Wilson, and Dr. Andras Tilcsik
Team DeBIAS poses in front of the posters of other fellow Gemstone teams!
Pictured (left to right): Daniel Zhu, Phil Matthew, Johnny Rajala, Rachel Antony, Seth Gleason, Joanna Ihm, Kyle Truong, and Soham Nagaokar
​
Not Pictured: Benjamin Bral and Aarushi Malhotra
The NCUR kids pose in front of their poster at the University of Wisconsin during their poster presentation session!
​
Pictured (left to right): Kyle Truong, Johnny Rajala, and Seth Gleason
Our experts!
Pictured (top to bottom, left to right): Dr. Natalie Tran, Dr. Juliet Aiken, Dr. Sonia Kang, Dr. Christo Wilson, and Dr. Andras Tilcsik
Team DeBIAS poses in front of the posters of other fellow Gemstone teams!
Pictured (left to right): Daniel Zhu, Phil Matthew, Johnny Rajala, Rachel Antony, Seth Gleason, Joanna Ihm, Kyle Truong, and Soham Nagaokar
​
Not Pictured: Benjamin Bral and Aarushi Malhotra
Pictured above is the iconic cheese curd, one of the main food exports of Wisconsin. Our NCUR team did a cheese curd crawl after the conference, ranking the best and worst cheese curds in the area!
Submitted Final Paper
May 1, 2023
Based on the feedback given to us by our discussants across all three of our conferences, we were able to make even more improvements to our research paper, to make it even more comprehensive of alternate perspectives as well as making it easier to follow along with to a new reader. With it turned in, our journey on the four-year Gemstone research program has come to a close. Thanks for following along! Or, if this is the first event you're seeing, scroll down to see how we got here!
Midwest Political Science Association Conference
April 15, 2023
Continuing DeBIAS's conference season, we qualified to present at the Midwest Political Science Association Conference in-person at the Palmer House In Chicago. The MPSA was founded in 1939 and is dedicated to the advancement of scholarship in all areas of political science by promoting the professional study, communication and teaching of political science. Our presentation sessions as this conference consisted of two poster sessions (Machine Learning, Text Analysis, and Measurement and Rectifying Historical Injustices), in which we displayed a large poster that summarized the important concepts and results of our research. From there, participants could walk around, read our poster, and ask us any questions they had about our research, similar to the Under Graduate Research Day presentation we attended last Spring. Similar to the Gemstone Thesis Conference, we also had dedicated discussants who had read our paper beforehand, asked us pertinent questions, and provided crucial feedback for our research that could be incorporated in a future draft. All in all, the conference went very well, and are happy we had the opportunity to present in Chicago. In fact,m our little excursion made Team DeBIAS the first Gemstone team to present at an in-person conference post-pandemic!
Our experts!
Pictured (top to bottom, left to right): Dr. Natalie Tran, Dr. Juliet Aiken, Dr. Sonia Kang, Dr. Christo Wilson, and Dr. Andras Tilcsik
Team DeBIAS poses in front of the posters of other fellow Gemstone teams!
Pictured (left to right): Daniel Zhu, Phil Matthew, Johnny Rajala, Rachel Antony, Seth Gleason, Joanna Ihm, Kyle Truong, and Soham Nagaokar
​
Not Pictured: Benjamin Bral and Aarushi Malhotra
The MPSA Kids pose in front of our poster at the Palmer House during the conference!
Pictured (left to right), Joanna Ihm, Soham Nagaokar, Aarushi Malhotra
Our experts!
Pictured (top to bottom, left to right): Dr. Natalie Tran, Dr. Juliet Aiken, Dr. Sonia Kang, Dr. Christo Wilson, and Dr. Andras Tilcsik
Team DeBIAS poses in front of the posters of other fellow Gemstone teams!
Pictured (left to right): Daniel Zhu, Phil Matthew, Johnny Rajala, Rachel Antony, Seth Gleason, Joanna Ihm, Kyle Truong, and Soham Nagaokar
​
Not Pictured: Benjamin Bral and Aarushi Malhotra
The MPSA kids snap a quick pic in front of an iconic Chicago landmark during their time off! Can you guess where they are?
Our experts!
Pictured (top to bottom, left to right): Dr. Natalie Tran, Dr. Juliet Aiken, Dr. Sonia Kang, Dr. Christo Wilson, and Dr. Andras Tilcsik
Team DeBIAS poses in front of the posters of other fellow Gemstone teams!
Pictured (left to right): Daniel Zhu, Phil Matthew, Johnny Rajala, Rachel Antony, Seth Gleason, Joanna Ihm, Kyle Truong, and Soham Nagaokar
​
Not Pictured: Benjamin Bral and Aarushi Malhotra
Team DeBIAS poses for one last time after presenting our final thesis to our discussants and loved ones!
Pictured (left to right, top to bottom): Phil Matthew, Aarushi Malhotra, Soham Nagaokar, Joanna Ihm, Benjamin Bral, Johnny Rajala, Seth Gleason, Daniel Zhu, Kyle Truong, Dr. Steven Sin (mentor), Rachel Antony
Gemstone Thesis Conference
April 14, 2023
To kick off DeBIAS's conference season, we presented at the virtual Gemstone Thesis Conference. This Thesis Conference is held annually for senior Gemstone teams to present and defend their research in front of an audience, which includes a panel of experts, friends, and family. These panel of experts are collectively referred to as "discussants", and are given a copy of our entire research paper beforehand. This allows them to become well-versed in our research project, and in turn, ask poignant questions during their designated question answering session. To this end, we created slides that summarized our research topic, process, and findings, as well as prepared for potential anticipated questions our discussants may ask us, as well as how we could potentially address them. While we were well prepared for the presentation, we were also a bit nervous, as this Thesis Conference was considered to be the main endpoint of the Gemstone program, which we have been preparing for these past four years. Fortunately, the Conference went off without a hitch, allowing us to cap off our Gemstone experience with a bang!
Finalized Results and Conclusion
Spring 2023
With conference season fast approaching, now was the time for our team to finalize the findings of our research. With our four-year research investigation nearing completion, the natural question that people who read our research had, was: What did you find? Initially, answering this question was actually quite difficult, as we initially hypothesized that the Indeed resume ranking algorithm would be biased against applicants from HBCUs by consistently ranking them worse as compared to their non-HBCU counterparts, when controlling for other factors. However, our data suggested the opposite, that this bias was not there on a large scale view. This meant that we also could not continue with our second hypothesis of using AI techniques to de-bias Indeed's algorithm, as no general bias was found to begin with. It was only when we analyzed the data by individual variables (such as major and experience length) that we uncovered bias against HBCU students. These new findings, that we were able to find based on advanced statistical analysis, proved that bias against minority applicants did exist, just in specific sub-populations, unlike what we initially believed, which became the backbone of our results and discussion moving forwards.
Undergraduate Research Day
April 27, 2022
For our final event of the academic year, Team DeBIAS attended the University of Maryland's Undergraduate Research Day, an event where we conducted a poster presentation of our current research to local visitors. The event was great practice for future conferences, which will most likely have a similar presentation structure. Additionally, the event was a great opportunity for our team to present our initial quantitative findings for the first time.
Our experts!
Pictured (top to bottom, left to right): Dr. Natalie Tran, Dr. Juliet Aiken, Dr. Sonia Kang, Dr. Christo Wilson, and Dr. Andras Tilcsik
Team DeBIAS poses in front of the posters of other fellow Gemstone teams!
Pictured (left to right): Daniel Zhu, Phil Matthew, Johnny Rajala, Rachel Antony, Seth Gleason, Joanna Ihm, Kyle Truong, and Soham Nagaokar
​
Not Pictured: Benjamin Bral and Aarushi Malhotra
Team DeBIAS poses in front of the posters of other fellow Gemstone teams!
Pictured (left to right): Daniel Zhu, Phil Matthew, Johnny Rajala, Rachel Antony, Seth Gleason, Joanna Ihm, Kyle Truong, and Soham Nagaokar
Not Pictured: Benjamin Bral and Aarushi Malhotra
Finalized First Three Chapters of Thesis
Dec 6, 2022
For the Fall 2022 semester, we decided to take a step back and review the first three chapters of our thesis to ensure that they were still up to date with our current research. This is especially pertinent to do in long-term research programs such as Gemstone, where there is plenty of time for research to change direction drastically, a phenomenon we were not immune to. While originally we were looking solely racial bias, we expanded our scope to include prestige bias. This expansion allowed us to investigate other variables, such as US News ranking and school size. This, in turn, allowed us to have a more comprehensive analysis and control for more factors related to how the prestigious the general public views certain institutions. We also took this opportunity to add more sources and sections to our literature review.
Giving Day Fundraiser
March 9, 2022 - April 9, 2022
With our team now having concrete data, we turned to the next resource we needed to complete our research: money. Future aspects of our research may require the use of more intense computing resources, such as Amazon Web Services, to properly train our AI models. However, since these are paid services, our team needed more funds to continue our research. After spending a significant amount of time creating a fundraiser page for our research and reaching out to friends, families, and community members, we were able to raise a grand total of $4,080 with 137 unique donors, raising more than any other Gemstone team that semester! Thanks to the generosity of the local community, we now have the necessary funds to continue our research and we would like to humbly thank anyone who donated to our campaign!
Our experts!
Pictured (top to bottom, left to right): Dr. Natalie Tran, Dr. Juliet Aiken, Dr. Sonia Kang, Dr. Christo Wilson, and Dr. Andras Tilcsik
Our donation page from the Spring Giving Day. Thanks to the generosity of our friends, family, and peers, Team DeBIAS was able to raise a total of $4,080 from 137 unique donors. We would like to thank anyone who donated or spread the word!
Our donation page from the Spring Giving Day. Thanks to the generosity of our friends, family, and peers, Team DeBIAS was able to raise a total of $4,080 from 137 unique donors. We would like to thank anyone who donated or spread the word!
​
Our experts!
Pictured (top to bottom, left to right): Dr. Natalie Tran, Dr. Juliet Aiken, Dr. Sonia Kang, Dr. Christo Wilson, and Dr. Andras Tilcsik
Below is a heatmap of where our team got donations from. The majority of donations came from Maryland, New Jersey, and California, although we had donations from all around the nation.
Below is a heatmap of where our team got donations from. The majority of donations came from Maryland, New Jersey, and California, although we had donations from all across the nation.
Preliminary Data Analysis Completed
Spring 2022
Using our preliminary data, we then moved on to complete data analysis. Our goal was to find if there was a linear relationship between HBCU status and Indeed rank, while also controlling for the type of degree. After using MatPlotLib to plot our data between the three variables, we found there to be a statistically significant correlation, indicating that the Indeed resume ranking algorithm is in fact biased against HBCU applicants, even when considering outside factors like the level of degree. This was a big step for our team, as it was the first time that our hypotheses had been confirmed by hard data, proving the validity of our research.
Preliminary Data Collection Completed
Spring 2022
With official backing from the Institutional Review Board, it was time for our team to begin our initial data collection. This took the form of downloading individual resumes from the job hiring site, Indeed. Since the resume repository of candidates at Indeed is massive, we had to use advanced settings in order to narrow our search by search radius and job type. From there, our team fed these resumes to a homemade parser with the ability to extract the information necessary to conduct our research. Said information included Degree Level and School Name. The reality of our data also caused some issues with the information we needed. For example, a previous version of research had us looking at applicants' zip codes to determine if the ranking algorithm was biased against those from redlined areas. However, this became problematic when we realized that the vast majority of Indeed users do not include their address on their resume, including zip code. Encountering unforeseen roadblocks, such as this one, forced our team to adapt and rework our research to fit the reality at hand. Eventually, our team settled on switching to HBCU enrollment as our new proxy for race, and using school name, a variable found in most resumes, to measure it.
Do Good Presentation
November 1, 2021
For the final major event of 2021, our team had to present our current research at Gemstone's Do Good event. The presentation would be similar to the Library Award, where we would present the current status of our research to a panel of judges. However, instead of university librarians, our team was to present to Gemstone faculty and alumni. These judges would have much more intimate knowledge of the Gemstone research process, and consequently, were more likely to ask critical questions regarding our research. Preparing for this event required significantly more defensive preparation regarding why certain research decisions were made, along with an explanation as to how our research positively benefitted society. While our team did not win any awards this time around, our in-depth presentation and detailed defense enabled us to fine-tune our presentation skills, making us more than ready for our future presentation events.
Our experts!
Pictured (top to bottom, left to right): Dr. Natalie Tran, Dr. Juliet Aiken, Dr. Sonia Kang, Dr. Christo Wilson, and Dr. Andras Tilcsik
Fun fact: this is Team DeBIAS's first professional photo!
Pictured (top to bottom, left to right): Seth Gleason, Kyle Truong, Daniel Zhu, Benjamin Bral. Johnny Rajala, Joanna Ihm, Aarushi Malhotra, Phil Matthew, Soham Nagaokar, and Rachel Antony
Fun fact: this is Team DeBIAS's first professional photo!
Pictured (top to bottom, left to right): Seth Gleason, Kyle Truong, Daniel Zhu, Benjamin Bral. Johnny Rajala, Joanna Ihm, Aarushi Malhotra, Phil Matthew, Soham Nagaokar, and Rachel Antony
Library Awards for Undergraduate Research Presentation
October 4, 2021
For team DEBIAS's next major event, we were tasked with presenting our research proposal to a panel consisting of librarians from the University of Maryland. We were judged not only our research content, but also how our team utilized the resources provided by the library system to guide how we found sources and how we planned our methodology. Our team qualified for the award, along with two other Gemstone teams, and as a result of prudent planning of our presentation, along with deeply understanding our project to ensure we could answer any questions the judges asked us, our team actually won the first place award! This gave us access to $2000 in funds, which would greatly assist our future research endeavors; however, our main takeaway from the event was how to plan presentations as a team, along with a newfound confidence in presenting our research to a panel of judges.
IRB Submissions for Approval
Fall 2021
A crucial component of any research is first getting approval from appropriate regulatory agencies, and our team was no exception. Our research required approval from the Instituional Review Board, so our team had to plead our case. Ensuring to the IRB that our research contained no personal data violations and that uploading synthetic resumes to Indeed's database did not violate TOS proved particularly challenging in confirming. We were forced to think critically and creatively about the relationship between our research purpose and methodology, forcing multiple reworkings of our research process. But, eventually, we were able to settle on a methodology that the IRB liked, enabling us to begin data collection in proper.
Scanning Script Completed
Fall 2021
With the Gemstone experience officially halfway over, it was time for our team to start implementing the practical components of our research. Part of our project would involve scanning preexisting resumes and using their qualities to generate brand new resumes. While the generation aspect was already being provided by GPT-3, an AI text-based generative model, we still needed to provide a script that could convert our already existing resumes into usable data by GPT-3. The Python script we created did exactly that, although it went through many iterations: it started by using regular expressions to grab pertinent information, but we recently had to switch to string indexing to reduce errors and ensure edge cases were handled.
Methodology Refinement
April - May 2021
After gaining valuable perspective from various experts, it was time for our team to apply that advice to our methodology by introducing multiple new details in order to further flesh out our procedure and make it more doable. Part of this process included narrowing down a resume ranking site, determining which job positions to apply for, selecting potential locations, reviewing resume revision methods, and determining what order to tackle our various methodologies.
Prospectus Completed
Spring 2021
After finishing our major event of the semester, our team decided that our time was best spent meeting with experts who have done research in the fields of AI and hiring bias. Our goal for our meetings with them was to ensure that our current methodology was feasible and to discuss if any parts of our procedure did not make sense. We also brainstormed various workarounds to parts of our methodology that needed fixing which would expedite our methodology refinement later on. Special thanks to Dr. Natalie Tran, Dr. Juliet Aiken, Dr. Sonia Kang, Dr. Christo Wilson, and Dr. Andras Tilcsik, who agreed to meet with us to further improve our methodology!
Our experts!
Pictured (top to bottom, left to right): Dr. Natalie Tran, Dr. Juliet Aiken, Dr. Sonia Kang, Dr. Christo Wilson, and Dr. Andras Tilcsik
Our experts!
Pictured (top to bottom, left to right): Dr. Natalie Tran, Dr. Juliet Aiken, Dr. Sonia Kang, Dr. Christo Wilson, and Dr. Andras Tilcsik
Proposal Defense
February 17, 2021
For the first event of 2021, Team DEBIAS had the monumental task of presenting our first proposal defense. A proposal defense is when, a research team presents the premise of their research and a draft of their procedure to a panel of experts, including the mentor and librarian. While not everyone on the team is in charge of presenting, extensive preparation is required of all teammates, as all members of the panel will ask thorough questions about all parts of the project to ensure that it is financially and practically sound. While this event is quite nerve-wracking on its own, the tension was compounded by the fact that this proposal defense had to be delivered virtually due to the Coronavirus pandemic, an irregularity that almost no other team in past years had to deal with. However, we managed to persist regardless and deliver an extensive presentation that successfully demonstrated the importance and practicality of our project by answering any and all questions from the expert panel.
Our experts!
Pictured (top to bottom, left to right): Dr. Natalie Tran, Dr. Juliet Aiken, Dr. Sonia Kang, Dr. Christo Wilson, and Dr. Andras Tilcsik
Our experts!
​
Pictured (top to bottom, left to right): Dr. Natalie Tran, Dr. Juliet Aiken, Dr. Sonia Kang, Dr. Christo Wilson, and Dr. Andras Tilcsik
Fun fact: this is Team DeBIAS's first professional photo!
Pictured (top to bottom, left to right): Seth Gleason, Kyle Truong, Daniel Zhu, Benjamin Bral. Johnny Rajala, Joanna Ihm, Aarushi Malhotra, Phil Matthew, Soham Nagaokar, and Rachel Antony
Methodology Draft 1 Completed
November 12, 2020
The preliminary procedure for our research project has been decided upon for the first time since DEBIAS's conception. This part of the writing process can often prove particularly frustrating and nerve-racking, as it's the first time research teams take a deep, critical look at what they want to accomplish and must determine if reaching their initial goals are possible and feasible. Almost always, at this point, teams will often have to rework their procedure to meet logistical and time constraints and in the process, shift the focus of their project; team DEBIAS was no exception. However, by the end, we were able to create a procedure that both accomplished what we originally set out to do, and was realistic with our given resources.
Introduction and Literature Review Draft 1 Completed
October 8, 2020
The formalization of the first half of our paper, which included various elements introduced in the prospectus, has been completed. The difference between this assignment and the previous one is that the ideas introduced in our prospectus now required formal evidence, in the form of the citing of other academic papers to prove legitimacy. Additionally, this is the first version of our paper to include a literature review, which is a summary of all the relevant, preexisting research that we found, organized into sections, which will serve to provide readers with the background knowledge necessary to understand our research. Granted, as this was only a first draft, future revisions would be necessary.
Source Research Begins
Summer 2020
We officially began amassing a multitude of sources to gather formal evidence that our research problem exists, investigate how different types of bias manifest in machine learning systems, and understand what past research had been done in the field of bias in artificial intelligence. Specifically, our STEM team organized previous papers in this topic based on the type of bias the paper focused on, in order to determine what biases and methodologies have not been tested yet. Meanwhile, our social sciences team was tasked with studying how unconscious bias surfaces in our everyday lives, so that they could better understand it and start brainstorming how to implement measures to combat it.
Prospectus Completed
Fall 2020
A Prospectus represents a preliminary draft of the first half of a research paper, and contains an introduction to our research problem, proof that it exists, preliminary research done by other academics in our chosen topic, and our general procedure, among other details.
Our first team picture!
​
Pictured (top to bottom, left to right): Dr. Steve Sin (mentor), Philip Matthew, Seth Gleason, Johnny Rajala, Daniel Zhu, Kyle Truong, Rachel Antony, Joanna Ihm, Aarushi Malhotra, and Soham Nagaokar
​
Not Pictured: Benjamin Bral
Our first team picture!
​
Pictured (top to bottom, left to right): Dr. Steve Sin (mentor), Philip Matthew, Seth Gleason, Johnny Rajala, Daniel Zhu, Kyle Truong, Rachel Antony, Joanna Ihm, Aarushi Malhotra, and Soham Nagaokar
Not Pictured: Benjamin Bral
Team DEBIAS is Born
April 3, 2020
Gemstone leadership brings together 10 bright, dedicated, and passionate individuals from varying disciplines for the purpose of figuring out how to remove conscious and unconscious bias from automated hiring systems.