Previous Job
Previous
Hadoop / Spark Engineer
Ref No.: 17-17534
Location: Mountain View, California
Description
Responsibilities
  • Own it – take full engineering responsibility from Dev to Test to Deploy
  • Lead the architecture, design, and development of components and services to enable Machine Learning at scale
  • Identify and recommend the most appropriate paradigms and technology choices for batch and realtime Client scenarios
  • Write highly efficient Spark jobs to process TB of data
  • Collect heuristics and optimize Spark jobs on large Hadoop clusters
  • Perform DevOps on AWS to ensure resiliency, availability, security, and HA/DR
  • Partner with a x-functional team of Architects, Data scientists, Data engineers, Analysts, PMs, and customers
  • Write clean, efficient code and deployment artifacts in Java or Python or Chef
  • Teach and mentor other engineers on the team

Qualifications
  • BS in Computer Science. MS Preferred.
  • Strong CS fundamentals including data structures, algorithms, and distributed systems.
  • A minimum of 5+ years of experience on distributed computing and on solving complex cluster computing problems
  • Deep hands-on experience with Spark 2.0 and Hadoop
  • Deep hands-on experience with SMACK architecture or similar
  • Strong programming experience in core Java
  • Strong experience with DevOps tools and processes
  • Strong Experience with RDBMS, NoSQL, or Graph Databases.
  • Working knowledge of AWS architecture, concepts, and tools
  • Some experience with Python and Chef
  • Experience with Agile methodologies
  • Amazon EMR experience would be a big plus
  • Excellent communication skills