Five Observations About Recruiting Big Data Professionals

The evolution of IQ Workforce continues.  For the past 6-years our team has recruited in the web analytics > digital analytics > customer analytics space.  We started out as web analytics experts, then took on all of the channels of digital analytics and then took on online / offline combinations of customer data for our clients.

Over the past 6-months we have added two additional disciplines:  Big Data and Predictive Analytics.  This has been a natural evolution… not something that we had planned.    

We are creatures of habit and would have been perfectly happy to continue to place the best digital & customer analytics talent on a contract and perm basis.  That does not appear to be an option going forward.  The times they are a changing. 

There has been a learning curve for our team and I thought it would be helpful to share some of the things that we have noticed in recruiting big data professionals.  There are no huge revelations here, but there are five things that we didn’t know 6-months ago. 

In case you are not sure what the phrase big data means, it is a pretty broad term for data sets that are too large and complex to mange with conventional database tools.  Since we are collecting exponentially more data every year, this is becoming an issue for more and more organizations.  Every day 2.5 quintillion bytes of data are being created.

Here are our observations:

1. There is much more emphasis on the technical side of analytics.  Most big data systems have to be built on top of Hadoop or some distributed environment.  There are very few packaged applications that run on top of Hadoop.  That seems likely to change in the next couple of years, but in the meantime it leaves companies taking on big, expensive architecture and development projects.

   There is certainly analysis being done on this data, but as these systems get built companies will be focused primarily on technical talent.

   At eMetrics last year I heard a joke that is worth repeating:  The definition of a data scientist is an analyst that lives in San Francisco.

   This is funny because it is true.  The title "Data Scientist" has been thrown around just as much as the phrase "Big Data" and it has begun to lose meaning.  My definition of a data scientist is someone that does analysis, visualization and sharing of insights from huge, complex data sets.   There is a technical / engineering component to many of these roles (for the reasons that we just mentioned) but there are also business and quantitative components.

   Data scientists are just as hard to find right now as web analysts were in 2006.

   Right now the tech jobs outnumber the analysis jobs.  How long will it take for that to switch?  Probably a year or two.    If you run a search on Indeed for Hadoop or "big data".  You will see that the vast majority of the listings are for technical positions.  They are mostly engineers, architects and developers, with the occasional data scientist sprinkled in.

   IQ Workforce has never crossed over into purely technical positions before… but digital analytics is different.  Companies have been implementing packaged digital analytics products.  This required some help from IT when it came to tagging, but the emphasis has largely been on the business and quant ends of the spectrum.

   We have been turning down searches for SAS developers, SQL developers and Javascripters for years.  We might have to start to rethink that philosophy if we are going to be valuable partners to big data clients.  We need to start relationships with R developers.  Does anyone out there know one???  Please send them our way.

2. Another thing that I noticed is that there is not a robust big data community.  As a recruiting firm you want to try to make yourself a valuable member of a professional community.  That was a lot easier to do in digital analytics, where there has been a robust community for as long as we have been here.  Leaders in the space like Eric Peterson, Jim Sterne and Jim Novo worked hard to set up the platforms and institutions that led to a culture of camaraderie and knowledge sharing among peers.

   The big data community is young and fragmented.  Most companies are still wrapping their heads around this stuff and creating these positions.

3. There are dozens (hundreds?) of start-ups and early-stage companies in the big data space right now.  Run a search on Indeed for data scientist positions.  Tell me how many of the companies you have heard of before.  For me it was very few.  Current listings include:  Zoosk, Dataminr, Knewton, AdSafe, SEOmoz, Flurry, Chegg AddThis, 23andME, Matchbox, PrecisionDemand, Fluitec Wind and Cloudera.  Not exactly household names… and that is only two pages of search results. 

   We are expanding our database with talent from dozens of start-ups and early stage companies that are positioning themselves in various corners and aspects of big data.  The NY DAA Symposium focused on this issue, inviting representatives from dozens of big data start-ups to present their value proposition.

   If you run a search on LinkedIn for data scientists you will see the same thing.  The vast majority work for companies with weird names in NY or San Francisco that have 11-50 employees.

4. There are lots of foreign nationals working in big data.  If you want to recruit a data scientist, an engineer or a distributed system architect you should be prepared to sponsor a work visa.  If not, you are taking a small field and cutting out 50% (I just made up that number, but it is probably conservative).  If you layer a geographic requirement on top of that, you are talking about a VERY small group of potential candidates.

   This is similar to the searches that we do for statistical modelers around the country.  If you want a 2-3 year data scientist or a 2-3 year statistical modeler you can either catch lightening in a bottle or sponsor a visa.

5. There is a lot of false advertising out there.  A lot of the resumes that we read have data scientist titles, but most of them are either programmers or some other kind of analyst that works with relatively large amounts of data from a variety of sources.

   I am not splitting hairs…  it is not the same thing.  It is not just a question of scale.  Big data shops have major issues that someone that works with SQL or Teradata has never dealt with.  They are much messier, much more IT-intensive and you can’t necessarily pick up those skills as you go.

   It is very similar to the experience that we had when social media first got hot.   There were a million self-professed social media experts on LinkedIn, but only a handful that actually did it for a living.

   The same is true on the job posting side.  If you actually read through a lot of the data scientist job descriptions, there has been a lot of scope creep with the title.  Yes, there are more and more big data shops every day but most companies just have data warehouses.  That is a different job.  It is no less valuable to the company, but it is not big data and your analysts are not data scientists.  They are analysts.  Even if the do live in San Francisco.