Data Engineer Subject Matter Expert in Charlotte, NC at Signature Consultants

Date Posted: 3/10/2020

Job Snapshot

Job Description

Data Engineer Subject Matter Expert Summary:

Signature Consultants is seeking a Data Engineer SME for an opportunity in Charlotte, NC. The Data Engineer SME will support or collaborate with application developers, database architects, data analysts and data scientists to ensure optimal data delivery architecture throughout ongoing projects/operations.

Responsibilities:

  • Develop and maintain automated ETL pipelines (with monitoring) using scripting languages such as Python, Spark, SQL and AWS services such as S3, Glue, Lambda, SNS, SQS, KMS.
  • Implement and support reporting and analytics infrastructure for internal business customers.
  • Develop and maintain data security and permissions solutions for enterprise scale data warehouse and data lake implementations including data encryption and database user access controls and logging.
  • Develop data objects for business analytics using data modeling techniques.
  • Develop and optimize data warehouse and data lake tables using best practices for DDL, physical and logical tables, data partitioning, compression, and parallelization.
  • Develop and maintain data warehouse and data lake metadata, data catalog, and user documentation for internal business customers.
  • Work with internal business customers and software development teams to gather and document requirements for data publishing and data consumption via data warehouse, data lake, and analytics solutions.
  • Design, build, and manage analytics infrastructure that can be utilized by data analysts, data scientists, and non-technical data consumers, which enables functions of the big data platform for Analytics.
  • Develop, construct, test, and maintain architectures, such as databases and large-scale processing systems that help analyze and process data in the way the Analytics organization requires.
  • Develop highly scalable data management interfaces, as well as software components by employing programming languages and tools.
  • Work closely with a team of Data Science staff to take existing or new models and convert them into scalable analytical solutions.
  • Design, document, build, test and deploy data pipelines that assemble large complex datasets from various sources and integrate them into a unified view.
  • Identify, design, and implement operational improvements: automating manual processes, data quality checks, error handling and recovery, re-designing infrastructure as needed.
  • Create data models that will allow analytics and business teams to derive insights about customer behaviors
  • Build new data pipelines, identify existing data gaps and provide automated solutions to deliver analytical capabilities and enriched data to applications.
  • Responsible for obtaining data from the System of Record and establishing batch or real-time data feed to provide analysis in an automated fashion.
  • Develop techniques supporting trending and analytic decision making processes
  • Apply technologies for responsive front-end experience
  • Ensure systems meet business requirements and industry practices
  • Research opportunities for data acquisition and new uses for existing data
  • Develop data set processes for data modeling, mining and production
  • Integrate data management technologies and software engineering tools into existing structures
  • Employ a variety of languages and tools (e.g. scripting languages)

Required Experience:

  • Degree in Computer Science, Engineering, or related fields.
  • 8-15 years of experience
  • Experience building enterprise-scale data warehouse and data lake solutions end-to-end.
  • Experience transitioning on premise big data platforms into cloud based platforms.
  • Knowledgeable about a variety of strategies for ingesting, modeling, processing, and persisting big data.
  • Experience with native AWS technologies for data and analytics such as Redshift Spectrum, Athena, S3, Lambda, Glue, EMR, Kinesis, SNS, CloudWatch, etc.
  • Write secure, stable, testable, maintainable code with minimal defects.
  • 4+ years of experience with one or more query languages (e.g. SQL), schema definition languages (e.g. DDL), and scripting languages (e.g. Python) to build data solutions.
  • Prior experience with Infrastructure as Code technology such as Terraform or Cloudformation

About Signature Consultants, LLC

Headquartered in Fort Lauderdale, Florida, Signature Consultants was established in 1997 with a singular focus: to provide clients and consultants with superior staffing solutions. For the ninth consecutive year, Signature was voted as one of the "Best Staffing Firms to Work For" and is now the 14th largest IT staffing firm in the United States (source: Staffing Industry Analysts). With 28 locations throughout North America, Signature annually deploys thousands of consultants to support, run, and manage their clients' technology needs. Signature offers IT staffing, consulting, managed solutions, and direct placement services. For more information on the company, please visit www.sigconsult.com. Signature Consultants is the parent company to Hunter Hollis and Madison Gunn.