Lead Software Engineer Scraping

New York, United States
Type
Permanent

Role:

Our client is an innovative FinTech company and the market leader in providing data and business intelligence to help customers make faster and smarter investment decisions. They are looking for a Lead Software Engineer to drive and own theit web scraping initiatives. This role will lead the design, development, and maintenance of scalable scraping solutions as part of their data technology team. The ideal candidate is an expert Python developer with strong experience in web scraping and related technologies.

Responsibilities

• Provide effective technical leadership and hands-on technical mentorship to developers.
• Lead the design and development of web scraping applications and services, ensuring they are scalable, efficient, and meet performance requirements.
• Drive technical decisions, ensuring that best practices are followed.
• Provide leadership in applying design patterns and adhering to SOLID engineering principles, with a focus on reusability, modularity, scalability, and fault tolerance.
• Utilize hands-on experience to design, build, and support web scraping solutions, particularly using Python and relevant libraries (e.g., Scrapy).
• Promote a culture of continuous improvement and high-quality code delivery.
• Partner with the broader technology leadership team to ensure service reliability through testing and release procedures.
• Stay current with industry trends in web scraping, data technologies, and emerging tools.
• Identify opportunities for innovation and process improvement,
• Ensure compliance with security and firmwide architecture standards.

Requirements

• Strong understanding of technical design patterns and SOLID engineering principles.
• Experience mentoring engineering teams, particularly in the context of data extraction and scraping projects.
• Expertise in Python development, particularly with web scraping libraries like Scrapy/Beautiful Soup.
• Proven experience in developing and maintaining large-scale web scraping solutions.
• Familiarity with C#/.NET technologies is a strong plus.
• Strong problem-solving skills and a proactive attitude.
• Strong understanding of API design principles and best practices
• Experience with CI/CD pipelines and automated testing frameworks.
• Familiarity with cloud platforms (AWS/GCP/Azure)
• Experience with software development lifecycle tools such as JIRA, Confluence, GitHub, etc.
• Excellent communication and collaboration skills.

This position offers comprehensive benefits and is eligible for an annual discretionary bonus.
Fully remote candidates will be considered.
Cannot provide Visa sponsorship at this time.

• share

Related jobs

Full Stack Engineer - React/Node

Practice
Development & Integration
Location
New York - United States
Type
Permanent
Job Title: Full Stack Software Engineer (Front-End Focus) Location: New York City, NY (Hybrid or Onsite)About Us: A rapidly growing healthcare techn...

Explore this job

PEGA LSA with KYC/CM - 12 months - Start ASAP - Part Remote working

Practice
Development & Integration
Location
Remote - - United States
Type
Contract
PEGA LSA with KYC/CM - 12 months - Start ASAP - Part Remote workingMy client, is seeking a highly skilled Pega Lead System Architect (LSA) with exp...

Explore this job

PEGA LSA - Stockholm - Hybrid - Start ASAP - 12 months (KYC/CPM)

Practice
Development & Integration
Location
Sweden - - - None Specified -
Type
Contract
PEGA LSA - Stockholm - Hybrid - Start ASAP - 12 months (KYC/CPM)We are seeking a highly skilled and motivated Pega Lead System Architect (LSA) to jo...

Explore this job

PEGA SSA - Sweden - MUST BE LOCATED ALREADY IN SWEDEN - 12 months - Start ASAP

Practice
Business Applications
Location
Stockholm, NORWAY - - Norway
Type
Contract
PEGA SSA - Sweden - MUST BE LOCATED ALREADY IN SWEDEN - 12 months - Start ASAP Next Ventures are seeking several experienced PEGA SSA's to help delive...

Explore this job

Senior JavaScript Developer

Practice
Development & Integration
Location
London - United Kingdom
Type
Permanent
Our client in the Hospitality Industry is looking for a Senior JavaScript Developer to join their team.Join the dynamic team in the hospitality indu...

Explore this job

Senior AI Engineer

Practice
Development & Integration
Location
New York, New York - United States
Type
Permanent
Job Description: Backend AI Engineer (LegalTech Startup)Location: New York City (Onsite, 5 days per week)About Us: A dynamic NYC-based startup at ...
meer...

Opslaan en terug Naar vacature

Meer vacatures van Next Ventures

Staff Engineer (Go)

Who are we? is on a mission to provide global private market visibility. Our industry-leading platform combines advanced AI tech with local-for-local research. It delivers the highest quality information on the companies
1 dag geleden- Volledige vacature bekijken

Senior Presales Lead, Public Sector

Your ImpactAs a highly personable, technically curious and enthusiastic Axon Sales Engineer, you consistently strive to be the trusted customer advisor for the value and technical aspects of Axon’s globally leading
4 dagen geleden- Volledige vacature bekijken

Senior Software Engineer

What you will do:Work with a team of engineers to develop high-quality softwareParticipate to all phases of the project lifecycle – gathering requirements, designing solutions, building newinterfaces, integration with
5 dagen geleden- Volledige vacature bekijken

Lead Electrical Design Engineer

Bedrijfsomschrijving: Locatie: regio AMSTERDAM. Deze onderneming is een vooraanstaande speler in de cleantech-sector, gericht op de ontwikkeling van innovatieve technologieën voor CO2-afvang. Geboren uit de Europese
1 dag geleden- Volledige vacature bekijken