Competition for skilled Data Engineers is high and they are in limited supply. So, what can you do to compete with big-budget companies that are fighting to hire the Data Engineers you need?
Having spoken with 317 actively job-seeking Data Engineers, reviewed the profiles and career history of an additional 800 UK-based Data Engineers, interviewed our customers for their own success stories, and pored over the data contained in 19 relevant market reports produced by us over the past 12 months, we’ve detailed your 4 best options to hire.
1. Prepare to pay full whack
There is perhaps no talent pool with more power at the moment than London’s tech professionals. With companies constantly on the lookout for skilled talent to hire, technologists are always assured of healthy interest should they decide to change jobs and are constantly approached, even when they’re not looking. It is essential that you, as the employer, ensure that you know the going market rates so that you can offer an appropriate salary that does not low-ball top talent.
Data Engineers tend to be high-budget hires, ranging from £50k to £70k for junior to mid-level, with Senior and Lead Data Engineers commanding salaries of £70 to £120k and above.
Further compounding the issue, Data Engineers are in consistently high demand from large corporations. In the past year alone, big names like Facebook, Shell, ASOS, Sky, and Amazon have expanded their teams of Data Engineers, some by as much as 83% – and their efforts are ramping up. The weight these brands carry, along with the resources they have at hand, exponentially reduces the likelihood of your ability to compete.
2. Develop existing skills, in-house
Instead of discounting existing employees due to lack of necessary skills to execute the job, perhaps it’s time to look at them with different eyes. If you have an employee with suitable foundational skills who is interested in shifting to a different position, this could be perfect for them.
Not everyone is interested in ‘climbing the ladder’ to add increasingly senior job titles to their portfolio, nor is everyone interested in entering management positions. For those who value personal development and job satisfaction above status, lateral moves are an enticing option that allow them to stay within the organisation whilst still achieving the satisfaction of personal and professional growth.
Whilst seemingly ‘slower’ than hiring the finished product, encouraging lateral moves reaps long-lasting rewards. Interviewing, onboarding, and training takes precious time and money. When an employee leaves, the time and resources you’ve invested leave with them. Considering internal transfers is a powerful way to reduce the stagnation that results in employee turnover, whilst retaining valuable knowledge within your business.
Investing in your current team not only breeds loyalty, it also alleviates long and frustrating hiring cycles for highly coveted professionals. Your employee can learn their new skills in their current role, certainly until you find their replacement, whilst your hiring team can focus on replacing a less coveted skill set.
3. Cross-train for long-lasting value
OK, so what is cross-training, and how can you approach it?
Cross-training is the act of taking an applicant from a different specialisation – for example, a Python Developer or a Data Warehouse Engineer – and giving them the skills they need in order to succeed as a Data Engineer.
When looking for candidates to cross-train, you need to consider the foundations they will need in order to quickly pick up new skills and add value to your company.
To cross-train someone into a Data Engineering position, they will likely need knowledge of:
- Data architecture
- Relational databases (such as SQL)
- ETL (Extract, Transform and Load) processes
- Analytical reporting
They may also have Programming experience (most likely C# or Python) and some knowledge of Cloud technologies such as Azure, AWS or GCP.
The more capabilities they have, the easier they will be to train – but the more they will probably cost to hire! Once you find a candidate with the necessary foundations, you will need to determine what skills they require in order to perform in the role of Data Engineer.
You may need to teach them:
- How to work with unstructured or non-relational data systems
- Software engineering (using languages such as Python, Scala, or Java)
- ELT processes
- How to use Hadoop or Spark
- How to use Cloud technologies (such as Azure, AWS or GCP)
Roles ideally suited to cross-training into Data Engineering:
- Data Warehouse Developer
- BI Developer
- Business Intelligence Developer
- SQL Developer
- Data Architect
- Python Developer
- Database Developer
- Oracle Developer
- ETL Developer
- SSIS Developer
- Data Analyst
- .NET Developer
A Talent Point Perspective
One of our favourite success stories involves exactly this! While sourcing a Data Engineer for one of our customers, we found an applicant that had a strong foundational knowledge of data architecture, having come from a job as a BI Developer. He had a strong interest in transitioning into a Data Engineering role – particularly in using newer tech such as AWS, Google Analytics, and GCP – but his current company simply did not have a need for it.
By considering a non-traditional prospect, our customer was able to gain a highly motivated new employee who went on to exceed expectations, becoming Lead Data Engineer in the space of a few months!
4. Target professionals looking to escape legacy tech
If you’re open to cross-training but feel that hiring a junior tech professional is unsuitable, then an alternative option is to target more senior-level professionals who may be ready for a change.
There is a huge untapped pool of experienced tech professionals who are eager to learn and use new technologies and tooling, but who may have found themselves stuck in a mire of legacy tech that offers no way forward.
By offering them an opportunity to transfer into Data Engineering – a rapidly growing field with endless opportunity – you can capture more experienced applicants and offer them a much-needed next step into a future-proofed position.
To summarise, your options to develop industry-leading Data Engineers are:
Ensure you have full visibility of the market and prepare to pay accordingly.
Attempting to pay below market rates is not only unlikely to secure the best talent but has a far-reaching impact on your employer brand. Bad news travels fast in the tech community and sites such as Glassdoor – as well as developer communities like GitHub and Stack Overflow – make any negative comments more public than ever, causing lasting damage to your brand.
Look internally for those who would jump at the opportunity to move laterally.
With the dual benefit of keeping hiring costs at a manageable level and retaining current staff, internal transfers allow employees who already know your business and your culture to bring a unique viewpoint into their new department – something that would be impossible with a totally new hire.
Appeal to experienced professionals who want to modernise.
Tap into the frustrations of experienced professionals who feel limited by legacy tech. They may currently hold a position in a large corporation where replacing outdated tech is time-consuming and prohibitively expensive, yet hunger to reignite their learning and explore the vast opportunities provided by a career in Data Engineering.
Enhancing your team with experienced professionals adds valuable business experience, confidence and new perspectives about how to tackle challenging problems based upon previous experience.
Cross-train for long-lasting value.
It is important to remember that, in order to successfully cross-train a new hire, they will need to be highly motivated, with an entrenched interest in self-learning and upskilling. Without that drive to succeed, you will have great difficulty converting them from one career path onto another.
When you find the right person – with the right motivation to succeed – cross-training is a proven, effective way to fill roles.