Anjali is the Lead for India Product & Design team at ThoughtSpot, where she leads product, user experience, and enablement. Prior to ThoughtSpot, she was with Capillary Technologies leading ideation, creation, and launch of new products. Anjali also led the product team at Myntra. Before moving to Bangalore, Anjali was working in Bay Area, California with companies like Box.net and Enterprise Solutions Inc. She holds a Master’s in Computer Science from Purdue University and yet another Master in Business Administration from Indian School of Business, Hyderabad.
Almost a decade back, renowned thought leader and author, Tom Davenport, and the US’ first Chief Data Scientist, DJ Patil proclaimed that ‘Data Science is the sexiest job of the 21st century’. This exciting headline, which has since become an ‘iconic’ statement in the technology industry, led to the demand of data science talent to skyrocket as organizations sought to unlock transformative growth through AI, ML and data analytics. The high demand pushed salaries through the roof and gave an enviable sheen to anyone in the field – as the career of the future.
In India, data science jobs were expected to see 150,000 new openings in 2020, which was an increase of about 62% as compared to 2019. Data science professionals with 3-10 years of experience got salaries in the range of INR 2.5 million to INR 6.5 million, and those with more experience can draw upwards of INR 10 million to INR 18 million according to Michael Page’s Talent Trends 2021 report. However, like most technology trends, the hype and the reality of what data science can accomplish aren’t aligned.
Expectation vs Reality
In 2021, the mismatch of expectation and reality is causing data science to lose its luster. Rapid growth and demand in digital transformation especially after the pandemic have led organizations to invest heavily in advanced analytics and next generation technologies like IoT, blockchain and quantum computing. But for most organizations, their investments in AI and machine learning have failed to produce the promised results. Moreover, the demand for data science professionals is at an all time high, and the supply has not been able to catch up.
The data scientists themselves are not to blame. For the most part, they’ve done what they were trained to do. Instead, it is the shortage of talent as well as the gaps between data science talent, the AI and machine learning technology that many are trained to use, and the business problems they seek to solve that are creating this friction and preventing the promised value from being delivered. Globally, according to a MIT Sloan review, as many as 85% of big data and machine learning projects fail to produce ROI.
Moreover, in the last few years, the Indian economy has been rising and the private sector has been doing well. Since every company has wanted to join the AI race, they are creating their own data science teams. However, these companies have not done their due diligence before hiring data science professionals or developing the practice in-house. Companies needed to have a clear vision and strategy as to how their AI investment is going to play out in the long run. Many companies mislead consumers and potential candidates by saying that they are using AI and deep learning, while the story is completely different. When later companies find no tangible value from large data science teams – data science professionals are the first to get laid off as ‘non-essential’ personnel, during crisis.
Reimaging data science
The issue here is two-fold. Those trained in data science have incredible education when it comes to computer science, mathematics, and statistical models. However, most of them are undertrained in business applications and domain expertise. And many times, upper-level management might have a lack of immediately realizing the value of data science to core business functions and revenues.
This lack of training and awareness is the culprit, keeping data science from delivering the value many expect – and this has led to the gap between the business and technical subset of people. Further, there’s been little to no emphasis on requisite communication and data storytelling skills. It’s this overemphasis on technical skills and under emphasis of business and soft skills that have led to not only a communication gap, but an imagination gap.
Having said that, many organizations have done exciting work to help address these issues, including co-development of curriculum with universities to broaden instruction from math and tech skills to include business acumen, communication, and versatility. In fact, the most successful data scientists themselves aren’t just sitting back; they’re actively self-training and upskilling themselves in these areas, too. It’s these individuals, who can bridge the analytics with business outcomes, who will have the ‘sexiest’ careers of the coming decade.
Going forward, if business value is not seen, companies are going to cherry-pick the talent and get rid of the rest. With so many people. Data science professions may not rise like software engineering during crisis – like the pandemic – but the demand for experienced and proven professionals who bring business value through data insights is always going to be there.