So, you need to grow your data team? Job titles vary considerably across the London employment market, with different businesses blending coding, cleansing, analysis, architecture and visualisation responsibilities across frequently conflicting job titles.
Do you need a Data Engineer or a Senior Data Engineer? What skills can you expect at each level? And how do you effectively attract target applicants?
We researched skill sets across 2,254 Data Engineers to show you how tooling impacts the size of the talent pool from which you can draw. Coupled with our experience creating data roles at all levels across our broad customer base, we’ve provided the following common overviews.
Data Engineers will be able to build data pipelines that collect information from various sources (such as APIs). These data streams are then cleansed and standardised, before being deposited into a centralised data warehouse or data lake where they can be leveraged for reporting.
Data Engineers should be able to build basic automation scripts using Python, and should have a good understanding of SQL. Ideally, they should be able to build stored procedures and underlying data structures, but will likely have the support of a wider, more senior team for advanced design and architecture.
You can expect to pay £55k – £65k for a Data Engineer at this level.
Senior Data Engineers
Senior Data Engineers will have a more holistic understanding of the business' data flows and will be capable of undertaking business-critical data projects without supervision.
At the upper end of the salary scale, they will likely have experience with Big Data tools such as Hadoop or Spark, and will have worked closely with Data Scientists to implement predictive modelling.
A Senior Data Engineer will have strong Python/R/Scala experience and will be able to build intricate data pipelines utilising Cloud technologies such as AWS, Google Cloud Platform or Azure.
You can expect to pay £70k – £85k for a Data Engineer at this level.
What does the talent pool look like?
In London alone, 2,254 people in our community of tech professionals self-identified as Data Engineers.
As seen in the table below, specific skill sets limit numbers considerably. You can imagine the demand for the 89 of that 2,254 who are already working with Cloud technologies!
Attraction is key here: why should the 4% with in-demand skills want to work for your business? High salaries are one reason, but autonomy and scope to grow are perhaps more important.
Leverage Existing Skill Sets.
Data Warehouse Developers are proactively transitioning to Data Engineering wherever possible. Those stuck with legacy data warehouses will not get that opportunity, but may learn Python, Java or Scala in their own time. You can tap into a highly skilled talent pool by hiring based on business awareness and a proactive enthusiasm for learning, as opposed to very narrow, specific experience with Cloud or Hadoop.
Hire on potential.
Data Engineering in the Cloud is an emerging employment market. This not only means high demand for a small skills base, but it also means knowledge rapidly becomes redundant. The strongest teams are often made up of natural learners who are driven to advance their skills on a consistent basis, as opposed to those who happen to have been on the right project at the right time. Hire on an ability to learn and natural problem solving, as opposed to specific, barely-known tooling.
Be language agnostic.
Most traditional Data Warehouse Developers started with SQL, learnt to use ETL tools, then moved on to creating scrips for cleansing and migration. Hiring only for Python, Java or Scala – especially at the level of coding skill required – limits talent pools. Be flexible and test for a natural leaning towards adaptability at interview.
Understand data skill sets.
Data Analysts and Data Scientists are not Data Engineers. Understanding the split between classifying, loading and cleansing data (Data Engineers) vs utilising it to provide business insight (Data Analysts and Scientists) is key to hiring successfully in this space.
Inspire your learning-focused team.
Collaborative teams learn best but, without at least one team member who currently possesses advanced, aspirational knowledge, what are your team aiming for? This means either one extremely senior hire – which, in this market, is very expensive – or a contract resource who can help embed knowledge in key team members, thus creating the skills most businesses are currently unable to hire for with strong loyalty and retention built in.