Senior Data Engineer

Company: CAT Technology, Inc
Location: Schaumburg
Posted on: November 23, 2020

Job Description:

Senior Data Engineer Location Schaumburg, IL 6 Months - Contract to Hire Manager has been rejecting resumes for a couple reasons. Either they do not include the required write up which should list how many years of each skill listed. Or the Write up does not match the resume. Example Write up says 5 years of DataBricks experience, but resume only shows 1 year. Description This person needs to figure out how to use Databricks Apache Spark, to organize their big data sets for this initiative. Needing strong exp with Bid Data, working with large Data sets and tables, and need to be able to speak in-depth about this stuffhellip not just surface level. Projects the candidate will be working on bullCreate and maintain data pipelines between on-premise data center, Azure Data Lake Storage, and Azure Synapse database using Databricks and Apache SparkScala. bullThis role is for a senior data engineer that will join a team responsible for managing a growing cloud-based data ecosystem consisting of a metadata driven data lake and databases that support real time analytics, extracts, and reporting. bullThe right candidate will have a solid background in data engineering and should have a few years of experience on a major cloud platform such as Azure Top Responsibilities bullBuilding and maintaining a data processing framework on Azure using Databricks bullWriting code in Apache SparkScala bullWorking with existing Databricks Delta Lake tables to optimize for CDC performance using techniques bullWorking with existing Databricks Notebooks to optimize or address performance concerns bullCreate new Databricks Notebooks or stand-alone Apache SparkScala code as needed bullWillingness to learn existing on-premise data management tools as required, such as Ab Initio Software toolsskills bullDatabricks bullApache Spark bullScala programming bullAzure. Skillsattributes bullData engineering experience - 5 years bullCloud platform experience ndash 2 years bullVersion Control (Git or equivalent) - 2 years. Nice to have bullData Integration Tools (SparkDatabricks or equivalent) 2 years bullVersion Control (Git or equivalent) 2 years bullScripting (LinuxUnix Shell scripting or equivalent) 2 years bullNetezza experience. Normal 0 false false false EN-US X-NONE X-NONE MicrosoftInternetExplorer4 Style Definitions table.MsoNormalTable mso-style-name"Table Normal" mso-tstyle-rowband-size0 mso-tstyle-colband-size0 mso-style-noshowyes mso-style-priority99 mso-style-qformatyes mso-style-parent"" mso-padding-alt0in 5.4pt 0in 5.4pt mso-para-margin0in mso-para-margin-bottom.0001pt mso-paginationwidow-orphan font-size10.0pt font-family"Times New Roman","serif"

Keywords: CAT Technology, Inc, Schaumburg , Senior Data Engineer, Engineering , Schaumburg, Illinois