Job Details

Email to Friend Save Back

Data Engineer

PITTSBURGH-15212, PA, US
02/24/2020

-

Required Skills

Tableau, SQL

Company

Infinity Consulting Solutions, Inc

Experience

5 to 7 Year(s)

Job Description

Job Description

The Data Engineer is responsible for building and supporting systems to improve and transform data practices for Research.

This role will focus on the research data pipeline, warehouse, databases, and BI tooling.

He/she will work with business analyst, data scientists, and other data engineers to facilitate ETL/ELT processes that move, clean, and store data.

The engineer will also be tasked with creating data accessibility points and tooling to enable reporting insights with ease of use and maintenance in mind.

The data engineer is expected to provide input to end-state design and schema while enforcing best practices.

Responsibilities

Design, build and maintain efficient and progressive data infrastructure for research across disparate research silos in San Francisco, Boston, and Pune focusing on creating a transparent data environment.

• Engage in a variety of tactical projects including but not limited to ETL, storage, visualization, reporting, web- scraping, and dashboard development

• Utilize ETL tooling to build, template, and rapidly deploy new pipelines for gathering and cleaning data

• Analyze existing data stores/data marts, clean, and migrate into a centralized data lake

• Work with Technology and Research leads to implement central and/or virtualized warehousing solutions

• Develop APIs for accessing data, for use by business users (i.e., researchers and portfolio managers)

• Configure Tableau dashboards and reports while serving as SME for end consumers of data

• Identify and deploy advanced BI tooling on top of datasets including AI/Client/DL techniques and algorithms

• Assist in the design and development of enterprise data standards and best practices

• Use modern tooling to focus on progressive technology and expand business capabilities and time to market

Work closely with business analysts, data scientists, and technologists through full project lifecycles which will provide deep insight on research needs, business processes, and research practices.

• Gather requirements and analyze solution options

• Develop solutions and define and execute test plans

• Define and implement operational procedures

• Automate the research and review of data quality issues to ensure data accuracy and reliability

• Resolve data integrity and data validation issues.

• Produce ad-hoc queries and reports for non-standard requests from Data Scientists and Data Consumers.

• Become SMEs on the full suite of solutions delivered by the Research Data Engineering team with an eye to identify, analyze, and interpret trends or patterns to identify new solution options, define process improvement opportunities and generate value opportunities for our business partners

Provides application software development services or technical support in more complex situations. May also be responsible for requirements gathering and BRD/SRD preparation.

Has thorough knowledge of the Software Development Life Cycle. Conducts reviews of the test Plan and test Data.

May allocate/ coordinate work within a team/project. Analyzes, defines and documents requirements for data, workflow, logical processes, hardware and operating system environment, interfaces with other systems, internal and external checks, controls, and outputs using standard development methodology.

Coordinates coding, testing, implementation and documentation of solution.

Ensures that expected application performance levels are achieved.

Contributes to defining time tables and project plans.

Analyzes and estimates feasibility, cost, time, and compatibility with hardware and other programs.

Conducts analysis of organizational needs and goals for the development and implementation of application systems.

Proposes innovative, creative technology solutions. Contributes to the achievement of related teams' objectives.

Job Qualifications:

• 5+ years of experience as a data engineer, software engineer, or similar

• Strong Experience building ETL pipelines and knowledge of ETL best practices

• Experience with overall data architecture and data routing design

• Familiarity with data quality control tools and processes

Strong communication skills and a keen attention to detail Technical Qualifications:

The candidate is not expected to have expertise in all technical areas listed but should be highly proficient in several of these including: • SQL, MongoDB, R, Python, SSIS, Pentaho/Kettle, Excel, Tableau, Kafka, Hive/Spark, Parquet

• Experience with CI/CD, container, and frameworks: GitLab, Selenium, Docker, Kubernetes

• Disciplines: Microservice Architecture, Design Patterns

• Environment Tooling: Agile, JIRA, Confluence

• Familiarity with RDBS and/or NoSQL and related best practices

Bachelor's degree in computer science engineering or a related discipline, or equivalent work experience required, 6-8 years of experience in software development required, experience in the securities or financial services industry is a plus.