Databricks Custom Agent Engineer
KPI Partners
- Experience
- 5+ years
- Location
- Remote · Bengaluru, India
- Job type
- Contract (6 months)
- Azure Databricks
- Python
- Databricks Agent Framework
- LangGraph
- RAG
- MCP
- data engineering
- data pipelines
About the role
Ship custom agents that run reliably inside Azure Databricks. Make data workflows faster and more scalable through agent automation. Turn RAG and tool-calling patterns into production-grade agent capabilities. Keep Databricks jobs and agents performant under real data volumes.
What you’ll do
- Design and implement custom agents using Azure Databricks and related frameworks
- Develop and maintain Python-based applications for data processing and analysis
- Utilize the Databricks Agent Framework and LangGraph for enhanced agent functionality
- Integrate RAG (Retrieval-Augmented Generation) and MCP (Multi-Channel Processing) tool calls to streamline agent communications
- Collaborate with data engineers and data scientists to ensure seamless data flows and data quality
- Conduct performance tuning and optimization of Databricks jobs and custom agents
- Document architectures, processes, and methodologies to facilitate knowledge sharing and team collaboration
What we’re looking for
Must have
- Azure Databricks
- Python
- Databricks Agent Framework
- LangGraph
- RAG
- MCP
What makes this role challenging
- Building custom agents that stay reliable inside Azure Databricks at production scale
- Wiring RAG and tool-calling into agent flows without breaking data quality
- Tuning Databricks jobs and agents for performance on real workloads
- Keeping pace with fast-moving agent frameworks and cloud data tooling
What success looks like
- First 30 days
- Ramped on the Azure Databricks environment and shipped a first custom agent or agent component
- By 90 days
- Owning custom agent development end-to-end, including RAG and tool-calling integrations
- First year
- Set the patterns for scalable, maintainable Databricks agents and improved data workflow performance
Tech stack
- Databricks
- Snowflake
- Microsoft Azure
- AWS
- Google Cloud
- Microsoft Fabric
- Power BI
- Apache Spark
- Apache Kafka
- Hadoop
- Oracle
- ServiceNow
About KPI Partners
KPI Partners is a global IT services and consulting firm that helps enterprises modernize their data, analytics, cloud, and AI ecosystems. With 19+ years of experience, it delivers enterprise data migration, cloud data architecture, AI consulting, and BI modernization, working with platforms including Databricks, Snowflake, Microsoft Azure, AWS, and Google Cloud.
IT Services & Consulting — Data, Analytics, Cloud & AI · ~628 employees (2026); sources range 500-1000
Databricks Custom Agent Engineer
Azure Databricks & Genie Agents; Python; Databricks Agent Framework / LangGraph; RAG & MCP/Tool Calling
Description
We are looking for a skilled Databricks Custom Agent Engineer to join our dynamic team at KPI Partners. The successful candidate will be responsible for designing, developing, and deploying custom agents within the Azure Databricks environment. You will work closely with cross-functional teams to optimize data workflows and enhance the capabilities of our data processing units. This role requires a strong focus on building efficient, scalable, and maintainable solutions that leverage Databricks and related technologies.
Key responsibilities
- Design and implement custom agents using Azure Databricks and related frameworks.
- Develop and maintain Python-based applications for data processing and analysis.
- Utilize the Databricks Agent Framework and LangGraph for enhanced agent functionality.
- Integrate RAG (Retrieval-Augmented Generation) and MCP (Multi-Channel Processing) tool calls to streamline agent communications.
- Collaborate with data engineers and data scientists to ensure seamless data flows and data quality.
- Conduct performance tuning and optimization of Databricks jobs and custom agents.
- Document architectures, processes, and methodologies to facilitate knowledge sharing and team collaboration.
- Stay updated with the latest trends and technologies in cloud computing and data engineering.
Skills required
- Proficiency in Azure Databricks and its ecosystem.
- Strong programming skills in Python with a focus on data manipulation and analysis.
- Experience with the Databricks Agent Framework and LangGraph.
- Knowledge of RAG (Retrieval-Augmented Generation) and MCP (Multi-Channel Processing) concepts.
- Familiarity with data engineering best practices and data pipeline construction.
- Ability to troubleshoot and optimize complex data workflows.
- Strong communication skills and the ability to work collaboratively in a team environment.
Yrs : 5 + Yrs
Budget : 2.5 Lakhs/ Per month