BigData/PySpark Developer

BerryBytes specialized in AI capabilities powered by computer vision and cloud-native microservice development. We are pushing the limits on real-time interactivity. We are looking for ML engineers who like to work with exceptional people on challenging problems.

We are looking for PySpark engineers familiar with data management concepts -- data modeling, effectively organizing large data volumes; both structured and unstructured, data quality, data security, etc and Hands-on experience with Spark / Python and PySpark in the Hadoop ecosystem. 

Experience in building a Recommendation system with PySpark is a big plus. 

Skills & Experience

• Proven experience in developing solutions using Spark architecture and PySpark for data engineering pipelines, transformation and aggregation of data from a variety of sources into a data lake.
• At least 2 or more years of relevant experience in developing PySpark programs using APIs. Expertise in different file formats like parquet.
• Experience with troubleshooting, fine-tuning Spark and python based applications for scalability and performance.
• Experience in data ingestion, processing and analyzing data using Spark/SQL from disparate sources. Knowledge in using Spark-Submit and Spark UI.
• Experience in creating and then performing operations on Spark RDD.
• Experience in creating Spark Data Frames from RDD, HIVE and Parquet files and then performing Joins and Aggregations on Dataframes.
• Experience in processing data from Python and other API modules. 

Nice to have

• Prior experience in building a Recommendation system is a big plus. 
• Knowledge in Docker. Kubernetes would be an added advantage.
• Logging (e.g. splunk, logstash, sentry) and Elastic Search (ELK).
• Prior experience in Machine Learning projects would be advantageous.

Notes:

• The deserving candidate will be awarded the best salary and bonus!
• onsite opportunities in the US and possible H1b sponsors as well.