Job Details

Email to Friend Save Back

Senior Data Engineer

BRISTOL-06010, CT, US
11/28/2019

-

Required Skills

Tableau, BI tools

Company

Infinity Consulting Solutions, Inc

Experience

3 to 6 Year(s)

Job Description

Long Term Contract 18+ months

Bristol, CT

Responsibilities:

Serve as a Technical lead and key coding contributor to the team

Build scalable analytics solution, including data processing, storage, and serving large-scale data through batch and stream, analytics for both behavioral & ad revenue through digital & non-digital channels

Lead the transformation of a peta-byte scale batch-based processing platform to a near real-time streaming platform using technologies such as Apache Kafka, Spark and database technologies such as Snowflake

Capture functional requirements and translate them into robust, scalable, operable solutions that work well within the overall data architecture

Mentor other software engineers by developing re-usable frameworks

Review design and code produced by other engineers

Embrace the DevOps mentality to build, deploy and support applications in cloud with minimal help from other teams

Requirements:

3+ years' working experience with Spark, Hive, Message Queue or Pub/Sub, Streaming technologies
6+ years of experience developing data pipelines using mix of languages (Python, Scala, SQL etc.) and open source frameworks to implement data ingest, processing, and analytics technologies

Experience leveraging open source big data processing frameworks, such as Apache Spark, Hadoop and streaming technologies such as Kafka

Hands-on experience with newer technologies relevant to the data space such as Spark, Airflow, Apache
Druid, Snowflake (or any other OLAP databases)

Experience developing and deploying data pipelines and real-time data streams within a cloud native i
nfrastructure preferably AWS

Proven strength in SQL, data modeling, and data warehousing

Experience designing solutions for structured and unstructured data

Experience with Git, Continuous Integration and related tools

Good unit testing and integration testing skills

Good problem-solving skills and passionate about data, technology and innovation

Preferred:

Experience with BI tools like Tableau etc.

Prior experience building internet scale platforms – handling Peta- byte scale data, operationalizing clusters with hundreds of compute nodes in cloud environment

Experience with open source such as Spring, Hadoop, Spark, Kafka, Druid, Kubernetes