Previous Job
Previous
Data Engineer
Ref No.: 18-09920
Location: Denver, Colorado
Diversant/Data Engineer
Diversant is searching for a Data Engineer for our client located in Denver, CO. This is an AWS, Python ETL, SQL Server project. The complexity of this project will be in the business processes and source systems which are (A) Not standardized or documented, (B) still under active development. Engineers personality must be patient and able to accommodate strong source code. This is a Direct Placement Opportunity. 

This person MUST have experience using Python as the production ETL Tool.


Experience:

Python skills/experience: Python 3.6, PEP8 compliant code that lists error/warning free (Pylint, Pyflake) and that follows team coding conventions that will be under development. Code will be developed in Python virtual environments and versioned via Github. Code require will require proper treatment of various text encodings, error handling and recovery, logging, standard lib based CSV import/export (CSV data quality issues may eliminate Pandas as a library), string and data transformations, DBAPI connections to SQL Server and PostgreSQL, AWS Boto3 API calls at the resource and client level. Python code will run under Windows Server and optionally under Linux environment. Technical knowledge of Window NT Services architecture and behavior needed for delivering data capture code.
 
ETL skills/experience: Must understand job control, logging, monitoring, restart, initial and incremental data pulls, archive/failure bin tracking of raw assets, CDC techniques, surrogate key assignment, upsert/merge, pivot/transpose, audit control fields, data quality screening, etc.
 
SQL Server 2016+ skills/experience: In-memory tables, column store indexes,  temporal tables, transactions, sequences, indexes. Optional: Stored procs, index and database tuning.
 
AWS skills/experience: Must understand IAC (infrastructure as code) concepts and be able to use CloudFormation or equivalent (Terraform, Chef, Ansible) to help build dev, QA and production environments from code. Respect for strong naming conventions and consistent resource tagging are important. Experience with AWS console, CLI, and Boto3 API.