Introducing MLOps Portable Prediction Servers

Most organizations invested in machine learning are experiencing what we at DataRobot call the AI production gap, which refers to the transition of models from the data science teams who develop them to the IT and DevOps teams who are responsible for deploying and monitoring them in production.

Production AI Challenges

Enterprises increasingly rely on AI and machine learning to build a competitive advantage, but the challenges involved in developing these assets do not end when the model is complete. In fact, that’s just the start of a new challenge—the journey to production—a journey that according to market surveys, fails at an alarming rate.  

Of the models that are deployed, few are equipped with monitoring capabilities sufficient to draw attention to health and performance over their lifetime. This is incredibly important because models that are accurate today are rarely accurate tomorrow. They require constant monitoring, retraining, or replacement.

IT and DevOps teams are in a tough position. They need to support many teams across their organization, some of which use different tools, languages, libraries, and environments. These teams seldom take production deployment into consideration, and certainly not from an enterprise perspective. The result is a proliferation of IT tools or large engineering efforts to build out custom frameworks to manage models in production.

To be successful with production AI across the enterprise, IT and DevOps teams need a standard and centralized approach for deploying and managing production models. Ideally, this approach uses familiar DevOps tools to containerize models and artifacts and includes out-of-the-box health and performance monitoring capabilities. 

New in DataRobot MLOps Release 6.3: Portable Prediction Servers

In MLOps Release 6.3, DataRobot introduces Portable Prediction Servers. An MLOps Portable Prediction Server is an easy-to-use Docker container that can host one or more production models. The models are accessible through a production-grade REST interface for predictions and prediction explanations.

Your DevOps and IT departments are already familiar with Docker. They can now easily integrate production models into pipelines and applications across the most popular cloud platforms, including Amazon Web Services, Microsoft Azure, and Google Cloud Services, and on-premise platforms without time-consuming software engineering efforts. And container orchestration tools, such as Kubernetes, provide a straightforward path to scale out as model demand requires.

If you’re using DataRobot MLOps 6.3, you can now define external prediction environments for a centralized view of where models are running. It doesn’t matter if the models were built using DataRobot or developed externally, outside of DataRobot, by your data science team. Your MLOps engineers and administrators can use this new functionality to quickly identify the locations of all production models.

You can use DataRobot Portable Prediction Servers or an internal DataRobot Prediction Server to deploy models to these external prediction environments. Managing all deployments from DataRobot MLOps ensures adherence to your organization’s governance and approval policies through a tightly gated workflow.  

The MLOps product guides you through the process of deploying a model with a Portable Prediction Server. To start the server that contains a model, you simply download the package and issue a single docker run command.

You use REST APIs to interact with the server. You can make real-time and batch predictions using CSV or JSON files. You can even request prediction explanations to understand the reasons behind a model’s outcomes.

Monitoring Portable Prediction Servers with MLOps Agents 

The flexibility to deploy models in a variety of environments can present challenges in the monitoring of their health and performance. That’s why the Portable Prediction Servers use the agent concept familiar in DevOps to track models in your preferred infrastructure while monitoring them centrally. Just imagine being able to monitor models created in any location and deployed on any infrastructure with minimal time or performance overhead.

The combination of the Portable Prediction Server and the Monitoring Agent offers your MLOps administrators deployment flexibility and a single pane of glass from which to observe model performance. Visit here to learn more about MLOps Agents.

Learn More About the Portable Prediction Server

This new model deployment option is included with your DataRobot MLOps license. It is generally available in Release 6.3.  

For a demonstration, see this video on the DataRobot Community page. For information about how to use the Portable Prediction Server, see instructions for your cloud platform:

• Azure Kubernetes Service (AKS)
• Amazon Elastic Kubernetes Service (Amazon EKS)
• Google Kubernetes Engine (GKE)

About the author

Chris Cozzi

Chris Cozzi is a product manager on DataRobot MLOps. Before he joined DataRobot, Chris worked on product management in enterprise analytics and healthcare marketing.

Meet Chris Cozzi