What is ML scale? The where and the when of ML usage
The topic this week is really at the heart of the presentation I have been working on for this year's speaking season. You either found that compelling or you did not, but you should take it for what it is worth that this topic is one that I’m really interested in understanding in depth. It was one that I invested my time in and felt was worth sharing with others during a formal talk.
When you sit down to think about machine learning a lot of hype jumps out and a little bit of truth is hiding somewhere in the middle of that. Some workflow is going to exist where a business driven use case occurs. Within that workflow some type of definable and repeatable task would benefit from machine learning to drive some inflection point. Maybe it is a straightforward recommendation engine or maybe you need to figure out what is in a photograph or a block of text to trigger some type of action. Doing that one time or a couple times might be a fun little proof of concept to setup and execute. Working out the details to do it every time over and over again across hundreds of thousands or millions of executions is a different story altogether.
Executing a machine learning model at scale either by calling an API or serving the model up is going to reveal edge cases and unintended consequences. That is one of the things that is going to happen. Something that you thought had a really solid fit and was going to execute with a high degree of accuracy will face the challenges of the real world and problems will ensure. You have to be ready for those by having a planful method of keeping things within pre planned tolerances. That is how you ensure your return on investment path works vs. having one good performance problem breaking down your entire financial use case. People are really starting to talk about the importance of scaling in real-world machine learning examples.[1]
I was one of those people talking about the importance of scaling. This week we are diving deep into the topic of ML scale in this newsletter. Earlier this week I was a speaker at a virtual conference on artificial intelligence called Ai4 2021 Retail, Supply Chain, & Marketing Summit, which occurred from April 7-8, 2021.[2] My topic was, “The ML scale problem: Thinking about where and when to use ML, ROI models, synthetic data, repeatable frameworks, and teams.”
Brief Presentation Abstract:
During this talk I will focus on the strategy behind both ML scale and MLOps decisions at scale. Conceptually I have been breaking down the categories of ML use cases into three buckets: Bucket 1: “Things you can call” e.g. external API services. Bucket 2: “Places you can be” e.g. ecosystems where you can build out your footprint (AWS, GCP, Azure, and many others that are springing up for MLOps delivery). Bucket 3: “Building something yourself” e.g. open source and self tooled solutions. Those 3 buckets are then considered across four phases: 1) Initiating: What is scale exactly? 2) Analyzing: Use cases “Scale vs. Maturity” 3) Directing: Leaning into ML scale. 4) Platforming: What is compelling about MLOps? It should be an entertaining 25 minutes.
Here is the first half of the talk transcribed from the actual presentation audio and edited a little bit for better grammar… the other half of the talk will arrive in your inbox in 2 weeks...
Hello, I’m Dr. Nels Lindahl. I’m a clinical systems IT director at a fortune 5 company, the author of Graduation with Civic Honors, an avid writer, tech chaser, sports card collector, and a major TensorFlow enthusiast.
Research Disclaimer
This presentation includes original research. All the content is mine and reflects my current views on the topics under consideration. Those thoughts were current at the point of this publication, but things in this space change rapidly.
Section 1: Initiating: What is scale exactly?
Conceptually I have been breaking down the categories of ML use cases into three buckets:
Bucket 1: “Things you can call” e.g. external API services
Bucket 2: “Places you can be” e.g. ecosystems where you can build out your footprint (AWS, GCP, Azure, and many others that are springing up for MLOps delivery)
Bucket 3: “Building something yourself” e.g. open source and self tooled solutions
Bucket one is the easiest because all you have to do is go out and consume it. You just need to connect to it, send some information out to it and get some information back, and you're ready to go. Bucket two is really about places where you can be totally within an ecosystem where you can build out your footprint for the endeavor. AWS, Azure, and GCP and many others that are springing up for MLOPS delivery ...and I do mean many others are ready to provide you an ecosystem. You're starting to see other ecosystems besides the major three, they're popping up and they're going to provide a different workflow in a different place where you can serve up your ML models and to be able to get going in this space.
Now the third category or bucket three is where you're building something yourself, these are the open source and self-tooled solutions and I tried to make this bucket just a little bit bigger because I feel like that's where a lot of this gets started. And a few years ago, it was the primary place and now we are seeing a shift. We're seeing that movement into other buckets. Those APIs are so easily available and you can get into these ecosystems where you can get going so quickly. Things are moving around and changing. That categorization of the three buckets helps me think about where things are for use cases and where things are gonna happen. It's a very tactical question versus strategic one.
“Seeing the strategy beyond trees in the random forest takes a bit of perspective. Sometimes it is easier to lock in and focus on a specific project and forget about how that project fits into a broader strategy. Having a targeted focused ML strategy that is applied from the top down can help ensure the right executive sponsorship and resources are focused on getting results. Instead of running a bunch of separate efforts that are self-incubating it might be better to have a definable and repeatable process to roll out and help ensure the same approach can be replicated in cost effective ways for the organization. That being said… of course you need an ML strategy.”
→ Machine learning strategy
→ The budget level investment
→ Budget level KPI for your ML spending
→ Compendium of KPIs including ROI
→ ML use case delivery
→ Ongoing MLOps
So it's not just about the three buckets and what you're doing and what you're considering. It's about how you put that pitch together. Your machine learning strategy has to be able to tell people what you're trying to accomplish and not only what you're trying to accomplish but what it's gonna do for the organization.
Now if you're going out and you're sharing that machine learning strategy and you're trying to get the organization to commit their precious funds to your strategy. You're probably talking about making a budget level investment. Now that budget level investment is gonna require you to do a few things that have some commitments.
Now, those commitments are probably going to include a budget level key KPI (key performance indicator) for your ML spending. How do I go about managing that? What do I do? Well, you knew, you knew it was gonna be a compendium of KPIs, including return on investment. So I started to build those things throughout the process so that I'm tracking and modeling and understanding exactly what's gonna happen within the deployment of my machine learning strategy. That is done so that I can communicate at all levels the success or you know, in some cases you may have some failure you need to retool and explain to people what you're doing, but your compendium of KPIs is what's gonna be able to build that communication strategy and let you be successful.
After you've done those first four things then and only then can you switch over and start doing some ML use case delivery. That's where the fun happens, right? And as you start delivering your ML use cases, you're gonna find out quickly that you're going to need an ongoing MLOps strategy to be able to sustain and continue and to do all the things that you committed to when you ask for those funds to have a budget-level commitment.
Selection 2: Analyzing: Common ML use cases “Scale vs. Maturity”
So, In this slide here what I did was take a look at some different common APIs from AWS, Azure, and GCP and just kind of looked at them in terms of scale versus maturity. So you can think about this as bucket one. These are things you can call. So it's part of your machine learning strategy.
You must make it be about purpose, replication, and reuse. That's going to be at the heart of getting value back for your organization and going out and using an API is a great way to get value back really quickly because if you go out and use AWS enterprise search or GCP Vision API or the AWS transcription service. You already have all the training, all the model building, all the hard work was done by the folks that are housing and working with these APIs. So that investment up front was spent already, but not by you or your organization. So you can get right to the point where you're able to inject this into your workflow and be able to do some fun and interesting things really quickly.
“From a budget line item to actually being operationalized you have to apply your machine learning strategy in a uniform way based on potential return on investment. After you do that you will know you are selecting the right path for the right reasons. Then you can begin to think about replication of both the results and process across as many applications as possible. Transfer learning both in terms of models and deployments really plays into this and you will learn quickly that after you figured out how to do it with quality and speed that applying that to a suite of things can happen much quicker. That is the power of your team coming together and being able to deliver results”
And yes, that's what I keep coming back to. I keep circling back to strategy. So your machine learning strategy must be at the forefront of what you're trying to communicate. You want it to be definable and repeatable; you want to be able to share that vision. You want your elevator pitch to be from the top down. People have to know it is happening. It has to be part of what the organization knows is happening. That's so powerful because when you're thinking about the where and when to use your machine learning you're gonna get the best bottom-up work from the folks that know can think about the examples in practice. Everybody can think about what could happen when and the people can bring all their SME level ideas about machine learning use cases and match that up to your machine learning strategy when those two things come together that's where success is going to be formed for your organization.
Now that success is not just from the ideation phase. It's from being planful and operationalizing in a successful way. It is in part from your return on investment model. That requires ongoing monitoring so that you can really understand exactly what's going on so that you really know from quarter to quarter from month to month is this effort successful. Is this model working? Does it need tuning? Do you need to do something with it?
At the same time though, you can get into a space where maybe you're making decisions about synthetic data versus other data strategies. Now if you're building your machine learning model and you know exactly what to do and you have a really good idea about what the data looks like during your entire process and you can build data synthetically that looks like your real data that probably means that you have a good understanding of exactly what's going on and you could be in a position where building some synthetic data and having a lot more data available for training and other things may work out really well for you. Otherwise you may have to look into other data strategies to get a little bit more complex, but I think being able to create synthetic data is something that you should definitely take a look at and it's worth further consideration.
So you need to make sure you have repeatable frameworks and they're key for defined MLOps plans. And once you get into this process where you're building pipelines and you're operationalizing your machine learning, you're gonna find out that doing it in a definable and repeatable way is gonna be key to that success.
But, It's not gonna work if you don't have the right teams. You may have to go out to a vendor you may have to build those internal teams. You may have to do things to get it right. But I'll tell you if you have the strategy and the ideation going but you don't have the team to execute it it may be time to put on the brakes and just think a little bit longer and dig more in and figure out who you can bring in to make it successful.
Section 3: Directing: Leaning into ML scale
So, this is the directing slide. Here we're gonna you know look about leaning into ML scale This is bucket 2 and this is bucket 3. So we are talking about places you can be in ecosystems where you can build stuff out and you know bucket 3 where you can be building something out yourself, these are just general use cases.
And I think this is a really great way to just kind of think about what's possible within your machine learning strategy. You could do some image recognition, some product recommendations, build some monitoring systems, maybe some account prioritization. Maybe you're in a business where lead prioritization makes sense. Not every business is going to find sound pattern matching is going to make sense or geospatial analysis. Just because you can build something out with a machine learning model doesn't necessarily mean it's the right thing to do for your business.
Now, this is a good sampling of things you can do and can jump into right now. So, maybe look at those things and figure out where the best things you can do in your strategy, you're going to need the teams to come together to be successful implementing and executing your strategy, but by all means you don't want to try to do everything.
“Part of your machine learning strategy must be about purpose, replication, and reuse. That is going to be at the heart of getting value back for the organization. Definable and repeatable results are the groundwork to predictable machine learning engagements. Machine learning is typically applied in production systems as part of a definable and repeatable process. That is how you get quality and speed. You have to have guardrails in place that keep things within the confines of what is possible for that model. Outside of that you must be clear on the purpose of using machine learning to do something for your organization.”
I think it's really important to really think about where you can go and what you can do within those machine learning engagements. So now that we took a look at what's possible and looked at a lot of those use cases we can dig into MLOps.
Stay tuned for the next part of this talk to arrive in your inbox in just two weeks...
Links and thoughts:
I listened to Nilay and Dieter host the Vergecast podcast this week
While I did watch part of the WAN show with Linus and Luke I’m not very amused with Linus’ effort to diversify platforms at the moment
I did enjoy the Deep Learning News #10 this week
Top 5 Tweets of the week:
Footnotes:
[1] https://ai4.io/retail/ You are getting most of the content in the newsletter this week, but you don’t get to hear my delivery of it which makes it more fun...
[2] Why Scaling Matters https://www.codementor.io/blog/scaling-ml-6ruo1wykxf
What’s next for The Lindahl Letter?
Week 12: Confounding within multiple ML model deployments
Week 13: Building out your ML Ops
Week 14: My Ai4 Healthcare NYC 2019 talk revisited
Week 15: What are people really doing with machine learning?
Week 16: Ongoing ML cloud costs
I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed reading this content, then please take a moment and share it with a friend.