Big Data jobs and careers
As the use of data in business gets more and more prevalent the ability to quickly and effectively produce data algorithms that can analyse large complex datasets is crucial. In order to do this, the right data infrastructure needs to be in place.
Why consider a career in Big Data?
Data Engineers build databases, data warehouses and data lakes to service the needs of the Data Analysts and Data Scientists in their businesses. This area is fast growing and Data Engineers are in massive demand as so many companies race to unlock the value in their data.
Typical role titles include
Big Data Engineer
Business Intelligence Developer
Business Intelligence Engineer
Big Data Developer
Data factory Developer
Data Availability Engineer
Data Science Engineer
Big Data Software Engineer
Data Warehouse Engineer
SQL Server Engineer
Data Infrastructure Engineer
Data Infrastructure Programmer
Data Infrastructure Developer
Data Analytics Engineer
Big Data Architect
Big Data Specialist
Big Data Lead
Head of Big Data
Data Architecture specialist
Head of Data Architecture
Career progression example
Junior Data Engineer
Senior Data Engineer
Lead Data Engineer
Head of Data Engineering
At an entry level, junior/graduate Data Engineering professionals on average will be earning between £30k-£35k. At the mid-level you’d be looking to earn £50k-£60k and seniors would be looking to earn £70k+.
Is it right for me? The skills it takes...
The skills needed can be split into 3 categories; programming skills, data science understanding and problem-solving.
Programming skills: Java | Python | Golang | Cython | Software Design - A background in APIs e.g. web APIs and static/dynamic libraries | ML Libraries - e.g. TensorFlow, CNTK, MLib, Spark, Kafka, Kinesis, Redshift.
What qualifications does it take?
A BSc, Masters (MSc) or Doctorate (PhD) in a maths, physics, statistics, artificial intelligence (AI), software engineering, engineering or computer science field with technical skills in programming. Demonstrable experience working with large data sets or alternatively candidates will work in Business Intelligence and then do online courses /self-learning/ get trained up on Big Data Engineering and Architecture and can move into these roles through this route.
Our Data recruitment team
Our latest Data jobsSearch all jobs
— One of the largest and widely respected UK Insurance companiesLocation:LondonSalary:50k to 70kYou will work on interesting Machine Learning projects alongside a team of established Data Scientists. You will be supported by a dedicated team of data engineers. The company is fully committed to the applications of machine learning across all business areas. Our client is one of the largest, longest standing, and well-respected UK Insurance companies. Within finance, they are one of the UK’s Juggernauts, but despite this, they have managed to retain a professional and welcoming working culture. Their Data Science team is constantly growing and they will be investing heavily over the next 12 months. Because of their continued interest in the area, they have recently partnered with one of the UK’s leading Red Brick Universities for research and collaboration. What you will be doing This role will involve developing and implementing machine learning platforms in order to improve the realisation of benefits in the Pricing Department. You will be working closely with a team of experienced Data Scientists and work across different teams within the department. This is a fantastic opportunity if you have a curious nature and want to continue your personal development in machine learning. Useful Experience • Good command of Python or R.• Solid experience using SQL (particularly relational databases)• Experience with Microsoft Azure.• Machine learning experience in Linear Models and Tree Based Methods. Able to deploy this in a business environment. • Being a willing and able communicator. What you will get in return for your experience Due to the flexibility and willingness to upskill people coming in the salary band is between £50,000 - £70,000. On top of the salary, the package includes 30(+2 optional buys) days holiday, private medical care, a very generous discount on all insurance offerings and partnerships, and the opportunity for a sizable bonus predicated on personal performance. The working environment has many testimonials which highlight that it is a wonderful place to work, and is head and shoulders above many insurance/finance companies. Their staff retention rate is a good indicator of this. What’s Next Please get in touch with Kye Williams with an up to date CV today. Don’t hesitate to call/email to discuss in more detail.
— A fantastic Charity supporting children across the globeLocation:LondonSalary:43000Join one of the leading charity institutes helping young children across the world. They have rich data streams across 70 different countries. A fantastic working culture with people who really care about their work and their impact. Our client is a Global charity working across 70 countries helping vulnerable young people and women overcome any challenges they have. The data science team supports their mission by delivering high-quality services, developing platforms to support business optimisation, and innovating new processes. If you are passionate about Data Engineering and the rights of Women & Children then this is the perfect role. What you will be doing This is a senior position for a data engineer to act as a technical lead on developing data structures, data processes, and delivering a better user experience. You will oversee the rest of the team to implement data handling processes, build reports, and manage data sets. This is a fantastic opportunity to step up into a leadership role. Useful experience Strong experience in SQL (query tuning and optimisation in particular). Experience with Sharepoint, Power Pivot, and TOGAF (development framework), would be helpful for quick integration. The ability to communicate technical information to people in other departments clearly. Knowledge of the legislation regulating data. Some project management experience or interest in progressing into it. DBS check will be an essential requirement. What you will get in return for your experience In return for your skills, you will receive a salary of up to £43,000 ad a good benefits package. More importantly, it is a fantastic opportunity to make a difference in a role and help the organisation create global change. People at the company are truly passionate about what they do and staff retention is much higher than most companies. What’s next Please get in touch with Kye Williams with an up to date CV today. Don’t hesitate to call / email to discuss in more detail.
— Friendly Insurance FirmLocation:WiltshireSalary:Up to £45KWork for a fast growing Insurance firm in a brand new role.Have a real impact by shaping the role as you see fit.Amazing working environment and friendly culture. This is a great chance to join a fast growing business with an amazing working culture. Working between the Marketing and Business Intelligence teams you will be exposed to multiple teams in the business and regularly will report to the Strategy Director. What you’ll be doing You will be the main Data Analyst for Marketing working as part of a larger team of Data Analysts experts who are delivering high quality insight analysis across a range of business areas.This is a nice, broad role analysing all marketing activities across the business including online and offline channels, customer data and more. What experience you’ll need to apply Experience of marketing data analysis or customer insight analysis in a commercial setting Solid experience of Google Analytics, Adobe Analytics or similar tools. Excellent MS Excel skills Excellent communication skills Data Visualisation and dashboard building skills (Power BI / Tableau / QlikView / QlikSense) What you’ll get in return for your experience A very competitive salary of £35,000 - £45,000 (dependent on experience) an excellent benefits package and an annual bonus What’s next? Please get in touch with Alex with an up to date CV today. Don’t hesitate to call / email to discuss the finer details.
— FinTech FirmLocation:NewportSalary:Up to £45KDevelop end-to-end solutions to progress the PowerBI suite. Gain experience with modern cloud services for data extraction, manipulation and analysis. Work across multiple products in a technologically driven culture This FinTech client have recently introduced and acquired new products to support its established service. They require a PowerBI developer to progress their PowerBI suite and bridge the gap between development and analysis as they continue to expand the BI Team. What you’ll be doing You will work collaboratively with one other PowerBI Developer within the wider BI team to gather requirements and build reports across multiple product areas. You’ll improve the PowerBI suite by undertaking complex queries, optimising data models and producing compelling visualisations to support the future data strategy. What you’ll need to apply Minimum 1 years’ experience of end-to-end PowerBI development SQL Server exposure Experience with cloud services will be advantageous Passionate to progress and learn new technologies Stakeholder engagement What you’ll get in return for your experience A base salary of up to £45k plus 15% bonus and great benefits including a training budget, share scheme and more. What’s next Please get in touch with Scott with a CV today, don’t hesitate to email/call to discuss the finer details.
— Growing Financial Services FirmLocation:SalisburySalary:Up to £50kLead reporting for the BI and Pricing Teams. Work with modern Cloud, BI and warehouse technologies. Define reporting standards for the Strategy Department. This firm is building out its Strategy Department to enable further data-driven decision-making across Product, IT and Pricing Teams. They require a Lead BI Analyst to lead a small team of analysts to optimise data and reporting activities. What you’ll be doing As the Lead, you will be responsible for all things BI and report into the Director of Strategy. This will require working with numerous teams to understand their data needs and deliver actionable insight. You’ll sit above three BI Analysts to provide support to the team ensuring that necessary data can be acquired and conveyed to the wider business. There won’t be line management responsibilities, instead, this position will be heavily focused on analysis, reporting and decision-making. As this is a brand-new position, you will be paramount in defining standards shaping BI. What you’ll need to apply Minimum of 2 years’ experience in BI or Data Analysis Experience with a modern BI tool (PowerBI, Tableau, QlikView or similar) Solid SQL and Excel Ability to mentor and guide junior analysts Statistical analysis experience preferred Degree in a numerical discipline preferred What you’ll get in return for your experience A great salary of up to £50k plus benefits and to work with numerous modern BI and Cloud technologies. What’s next Please get in touch with Scott with an up-to-date CV today, don’t hesitate to call/email to discuss the finer details.
— Health Tech CompanyLocation:BathSalary:Up to £32kWork alongside a team of experienced developers. Design and build ETL solutions. Autonomous role in a collaborative environment. This established development company is continuing to grow and need a specialist ETL developer to work closely with their team of developers in Bath. This position will work on projects for external clients to deliver a wide range of data migration requirements. What you’ll be doing You will work alongside a team of experienced developers and external clients to scope out their data migration needs. This will include identifying, documenting and validating the migration scope by liaising with clients while also maintain the existing ETL set-up. You’ll shape the analytics at this firm by designing and implementing new ETL solutions and inform all relevant parties on the end-to-end process. Due to these responsibilities, you will be a well-organised developer, capable of working on different projects while engaging internal and external stakeholders. What you’ll need to apply Minimum 1 years’ experience in ETL Development (e.g Talend, SSIS, Informatica, Alteryx) Solid SQL Project Delivery and stakeholder management Experience with object-orientated programming desirable Agile/Scrum desirable Proactive and self-organised What you’ll get in return for your experience A competitive salary of up to £32,000 DOE plus bonus and benefits including flexible working, social events, training and development budget and more. What’s next Please get in touch with Scott with an up to date CV today, don’t hesitate to call/email to discuss the role in more detail.
— Operational Data FirmLocation:GloucestershireSalary:Up to £65KWork with Cloud, Big Data and CI/CD technologies. Build modern, scalable infrastructure. Flexible organisation that encourages remote working. This organisation are building an analytics platform to progress their data science capabilities. This platform will define the data structure of the organisation and enable clients to be integrated via a variety of serverless technologies. What you’ll be doing As the Senior Data Engineer, you will stamp out a data structure that enables Data Science activities consistent with the long-term data strategy. This will require deploying a diverse range of modern applications that facilitate the integration of existing sources and implement original technologies. You’ll be responsible for each tool implemented, ensuring that all available data can be acquired and accessed by the wider team for analysis. What you’ll need to apply Minimum 3 years’ experience in Data, Cloud or Software Engineering Must have CI/CD experience Must have experience using Git Cloud development experience, AWS OR GCP preferred Experience with a dataflow tool (Kafka, Airflow, Luigi or similar) Modern programming experience (Python, Scala, Java or similar) Apache Hadoop, Spark or MapReduce desirable Degree in Computer Science or similar What you’ll get in return for your experience A salary of up to £65,000 plus great benefits including working from home What’s next Please get in touch with Scott with an up-to-date CV today, don’t hesitate to call/email to discuss the finer details.
— Operational Data FirmLocation:GloucestershireSalary:Up to £50kWork with Cloud, Big Data and CI/CD technologies. Build modern, scalable infrastructure. Flexible organisation that encourages remote working. This organisation are building an analytics platform to progress their data science capabilities. This platform will define the data structure of the organisation and enable clients to be integrated via a variety of serverless technologies. What you’ll be doing As the Data Engineer, you will develop the data structure that enables Data Science activities consistent with the long-term data strategy. This will require deploying a diverse range of modern applications that facilitate the integration of existing sources and implement original technologies. You’ll be responsible for supporting each tool that is implemented, ensuring that all available data can be acquired and accessed by the wider team for analysis. What you’ll need to apply Minimum 2 years’ experience in Data, Cloud or Software Engineering Must have CI/CD experience Must have experience using Git Cloud development experience, AWS OR GCP preferred Experience with a dataflow tool (Kafka, Airflow, Luigi or similar) Modern programming experience (Python, Scala, Java or similar) Apache Hadoop, Spark or MapReduce desirable Degree in Computer Science or similar What you’ll get in return for your experience A salary of up to £50,000 plus great benefits including working from home. What’s next Please get in touch with Scott with an up-to-date CV today, don’t hesitate to call/email to discuss the finer details.
— Financial Services InstitutionLocation:BournemouthSalary:Up to £45kExcellent opportunity to work for a fast-growing financial services firm in Bournemouth. Gain wider experience looking at customer data, internal business data and financial reporting data. Join an innovative and forward-thinking company. Joining the Analytics Team at this friendly financial services firm, you will be given the chance to grow and get exposure to a wide range of customer, internal business, operational data and financial data. What you'll be doing As the SQL Developer working as part of the wider Analytics Team, you will use your SQL skills to answer business questions. You will be working with the MS suite and Visual Studio to manipulate data, take data from multiple sources and then provide it to the data analysts. You'll have plenty of interactions with Senior Managers in this business across the different areas and departments to support them with their data needs. What you'll need to apply Minimum 2 years’ experience with SQL Good working knowledge of Excel Visual Studio experience preferred SQL Server experience desirable Object-orientated language desirable What you'll get in return for your experience A great salary of up to £45,000 DOE plus a great benefits package and a bonus. What's next Please get in touch with Scott with an up-to-date CV today. Don't hesitate to call/email to discuss the finer details.
Data case studies
Alistair Rogers, Data Scientist
- Alistair is an experienced Data Scientist with working with clients across government, healthcare and banking sectors.
- While successfully working in London as a Data Scientist for one of the Big Four professional service companies, he was planning to move to Bristol and approached us to get an understanding of the local market, especially opportunities and skills in demand within Data in the South West well ahead of the move.
- We advised on opportunities and businesses in and around Bristol as well as in-demand skill sets so he could gather relevant exposure in advance.
- Once ready to make the move, we were able to match Alistair perfectly with an open opportunity. Equipping him with tips and advice, we guided him through the interview process and negotiated the salary that was in line with his expectations. Alistair started as a Data Scientist with OnCorps, a start-up FinTech firm.
"Alex and ADLIB were fantastic to work with and absolutely made the daunting experience of finding a job in a new city a lot easier! They take the time to understand the fields that they are recruiting for and even go to developer meetups to learn more about the areas that they recruit in. Would definitely recommend working with Alex and ADLIB."
Alistair Rogers, Data Scientist
Two niche hires in record time
- SunLife are a financial services company and one of the largest life insurance companies in the world.
- As part of a small restructure, they were looking to expand their Performance Optimisation team, by adding two analysts.
- Low quality of applications and scarcity of top analytics talent can make finding specialists extremely difficult.
- By tapping into existing candidate pools and relationships, we were able to suggest a range of suitable candidates with a wide range of experience, resulting in SunLife making two great Performance Analyst hires.
“ADLIB helped Sunlife hire two great analysts in record time , who are already embedded in to the team and delivering great pieces of work.”
Billy Mohan, Digital Performance Manager at SunLife