2020 Oct 31
This content is an excerpt from an interview I did with fellow SharpestMinds alum Amber Teng.
Q: Did you always know that working in data science was what you wanted to do? What inspired you to pursue a career in natural language processing? And could you tell us a bit about your work on the Primer.ai applied research team looks like?
Not at all! I donât think many of us who work in data science today could have anticipated the rise of this field. I didnât even really know about the widespread use of applied statistics in the private sector until my senior year of undergrad.
When I graduated with my B.A. and started my first job in marketing, I figured out that that wasnât the right fit for me relatively quickly. I started researching my alternatives to see if there might be a career I could transition into that would be better-suited to my personality and values (and frankly, better-paying, as the entry-level marketing salary was only enough to live paycheck-to-paycheck in the Bay Area).
After a few months of digging, I landed on data science. It was intellectually challenging work, poised to make an enormous impact both economically and in society at large. Not only that, but I noticed that people in data science careers often cared more deeply about ethics than I had seen elsewhere. To see that people in the field genuinely cared about how their work would impact people really spoke to me, and is what ultimately helped me decide on making the transition.
That said, I was still unsure, because I had had negative experiences with math and computer science in undergrad, and I wasnât sure that I could hack it (haha). In my first quarter at Stanford, I got the worst grades I had ever received in my life in a calculus course and a computer science (CS) course, which caused me to seriously question whether I was cut out for those kinds of subjects.â˝
So during these early stages, I did my best to focus on objective measurements of my potential instead of my own feelings about whether I could succeed in data science: âwell I scored X on the SAT, and the average score for CS majors on the SAT was Y (where X > Y), so I should be able to learn the math and other material just as well as other folks in this field.â
Later, during my M.S. in Stats, I made the connection that the primary reasons for my underperformance as an undergrad were a lack of good study habits and a lack of interest in math as a subject. In high school, I could get away with waiting until the night before to study for the test, and never reading the textbook outside of class, but that was no longer the case at Stanford.
I improved my study habits over time, and by the time I took 2 statistics courses in senior year, I was able to ace them both.
Similarly, once I started learning about some of the fascinating and unexpected ways that math and stats are being applied to the real world via data science and machine learning, my mind really awoke to the benefits of math, and suddenly the motivation to master it was there. I got straight Aâs in my M.S. coursework for all 3 semesters that I was enrolled.
It is still a little crazy to think that just a few years ago I truly disliked both math and programming, yet here I am now, using them both every day and genuinely enjoying it. I really want to emphasize how important it is not to put yourself into a âmath personâ/ânot a math personâ box, and the same goes for programming.
Both skillsets are simply tools, and these tools have incredible power to make you more effective at any other area or interest you care about making a difference in, whether thatâs art, law, a social science, or a more traditional synergy like engineering. If you can push through those early feelings of resistance and intimidation, there are wonderful feelings of competence and accomplishment waiting for you once youâre able to start using these tools for the things you care about.
As for why I chose natural language processing (NLP) in particular, there are a few reasons.
On a career level, I saw the NLP community as more welcoming to people from unconventional backgrounds, relative to an area like computer vision where I was really only seeing people from CS, math, physics, and electrical engineering backgrounds.
On a more personal and interests-based level, I see NLP as the field best suited to helping solve the problem of information overload. There is an endless amount of information to consume, which is contributing to a heightened level of stress for everyone, as well as impairing the productivity of people in knowledge work careers.
I love NLP because I can contribute directly to helping people cut through the noise and get down to what they need to know in order to live their lives, make informed decisions, and do their work more effectively.
My work at Primer is directly relevant to the problem of information overload. At Primer, weâre leveraging powerful, cutting-edge NLP models to extract structured information from noisy, unstructured text data. This helps our customers get at the information they need much faster than having individual humans poring over the data themselves. Some analysts are working 12 hour days simply because they have no way of quickly reading and digesting the deluge of information theyâre responsible for staying up-to-date with, and we want to change that.
My team, Applied Research, is tasked with training, testing, and making deep learning models available for Primerâs products, then integrating those models into our data pipeline or exposing them for use via an API. We also create reusable scripts and resources that allow people to train their own models on their own data. The work involves not just model experiments and engineering, but also plenty of collaboration with other teams that work more directly on our products and infrastructure.
In terms of the week-to-week, Iâd say half the time goes to writing code for model training/evaluation, data preprocessing, and other typical machine learning tasks, and the other half goes toward communicating about the work: discussing plans, specifications, and progress with product managers, working with our data labeling team to create datasets for new and existing tasks, as well as presenting to the company at large about new developments and improvements of our models.
â˝ I would learn later that these kinds of experiences seem to be uniquely demoralizing to women - Claudia Goldin showed that, in economics, men are not deterred by receiving bad grades in introductory coursework, but if women feel that they are underperforming or that their competitive advantage might be in a different subject, they will often switch to something else. (source)