Data Engineer

Description

Job Summary:

We are seeking a highly skilled Data Engineer to join our team and play a pivotal role in designing, implementing, and optimizing our data infrastructure. The ideal candidate will have extensive expertise in AWS Glue and Amazon Redshift, along with strong development experience in Python, PySpark, and ELT/ETL (Extract, Transform, Load) pipelines. A deep understanding of data warehousing and cloud-based data solutions is essential.

In this role, you will be responsible for maintaining data integrity, availability, and security. You will work closely with cross-functional teams to convert complex data into valuable insights that drive informed business decisions. If you are passionate about data engineering and excited to make a meaningful impact, we invite you to apply for this opportunity at Sunward.

Requirements

Essential Job Duties:

Main responsibilities: 

• Design and implement data integration solutions using AWS S3, Glue, Python, SQL, and Redshift to meet internal client requirements while ensuring data accuracy and reliability.
• Develop complex data transformations and mappings within Redshift to support data migration and transformation tasks.
• Identify and address bottlenecks and optimization opportunities in ETL processes, troubleshooting and resolving technical issues as needed.
• Provide guidance, knowledge sharing, and mentorship to team members to enhance their ETL skills.
• Collaborate with cross-functional teams, including data engineers, architects, and business analysts, to deliver high-quality solutions.
• Design, build, install, test, and maintain highly scalable data management systems.
• Ensure data systems align with business requirements and industry best practices.
• Develop standardized processes for data mining, modeling, and production.
• Utilize various tools to integrate systems, identify opportunities for improving data quality, and redesign data for new use cases.
• Perform data analysis to troubleshoot and resolve data-related issues.
• Work closely with data and analytics experts to enhance the functionality of data systems.
• Establish data governance practices and frameworks to ensure data quality and consistency.
• Document business data structures and types.
• Collaborate with stakeholders, including executives, product managers, data teams, and design teams, to address data-related technical challenges and support infrastructure needs.
• Perform other duties as assigned.

Required Skills/Abilities:

• Proficiency with AWS data stack (Glue, S3, and Redshift) and ETL (Extract, Transform, Load) processes and tools. 
• Experience in batch data and real time ingestion. Strong experience in performance tuning in Python, PySpark, and SQL.
• Experience with data warehousing concepts, relational databases and dimensional modeling.
• Familiarity with version control systems and CI/CD pipelines.
• Strong analytical skills and problem-solving ability.
• Excellent interpersonal and negotiation skills.
• Excellent work organizational/prioritization skills and attention to detail.
• Strong proficiency with Microsoft Office applications. Familiar and comfortable with information technology, particularly digital means of communication.

Knowledge:

• Strong proficiency in the best practices developing a data pipeline, and administration. Must have experience with AWS Glue and python/PySpark.
• Understanding of data warehousing concepts, data modeling, and developing data dictionaries.
• Strong AWS data stack experience

Education and Experience:

• Minimum of 2 years of experience in ETL, Data Engineering, or related field.
• Bachelor’s degree or more than 2 years of related experience.

Physical Requirements:

Prolonged periods of sitting at a desk and working on a computer.