That one with a machine learning post
Right now my week by week roundup for The Lindahl Letter has 72 topics. This very missive you are reading right now marks my 1 year anniversary of writing Substack posts. That milestone now represents 52 out of the 72 topics in the backlog. Not only was I able to hit the 52 straight weeks of publishing milestones, but also I was able to publish the first year of my Substack posts as an edited and packaged eBook, “The Lindahl Letter: On Machine Learning” [1]. This particularly missive Substack post this week will be a slightly longer one with a few lessons learned and some observations. Overall this ongoing weekly writing effort has been about my efforts to chronicle my machine learning research. Every week I spend time digging into a topic and considering how to communicate about it from my perspective. This effort was not designed to be a purely reactionary dialogue that only responded to topics. My analysis was intended to be deeper than just initial reactions veering into the category of actual consideration.
Long-haul writing related observations:
Keep an idea backlog - When a topic comes to mind it should always be added to the backlog. Sometimes that means quickly writing it down in Google Keep and other times it means just writing it in the backlog real-time. Feeding the backlog with solid topics is really the best way to keep the writing engine running week after week.
Have a writing routine - You are going to want to keep to a set writing routine and plan around vacations and other interruptions. My writing routine is straightforward and involves waking up early on Saturday morning to do the bulk of the writing and waking up early on Sunday morning to close out the effort and pivot to working on writing other academic content.
Keep planning - Working ahead is a nice idea, but it is really hard to do in practice over the long haul. You can always keep planning and refining where you are going in terms of a long term writing effort. Sometimes I’m super productive and a few weeks of work will get completed early, but over time I have learned that I can only backfill for a vacation or travel day from time to time. This writing effort ends up being a just in time type of writing exercise.
Stay focused in the window - Staying focused on the topic at hand and not diverging. This is way harder than it should be, but is always the gate that determines if the topic at hand gets focused on and closed out or if my attention jumps from topic to topic and nothing gets completed.
Protect your golden hour - Know your golden hour of writing focus and creativity. My Fitbit watch vibrates as an alarm on Saturday morning to let me know it is time to get out of bed and start writing. A small window exists when nobody is awake and I can super focus on creating epic prose. Generally within the first two hours of waking up my golden hour of productivity occurs and it is downhill from that moment throughout the rest of the day. Peak productivity is glorious and you have to protect that time and manage it wisely.
Machine learning observations:
Major change is on the way - Really interesting groups like EluetherAI and Hugging Face are working to make waves and it feels like major change is on the way in 2022. People are coming together to build projects and share things that are changing machine learning. These changes will be decentralized and not driven by the major corporations.
Ethical modeling is important - Timnit Gebru (formerly of Google) started the Distributed Artificial Intelligence Research Institute (DAIR) and other people are digging into ethical considerations related to machine learning.
Models need ever increasing compute time - Models are requiring more and more computing power to train as they get bigger. Training a large language model may be cost prohibitive to people outside of the largest companies. Actually serving up the models is much less computing intensive. At some point in the next 5 years we may see a situation arise where the cost of the GPU time is seen as more than just a monetary one. We may see the energy required to train up the biggest models be considered as part of the equation.
ML is getting democratized - We are seeing more and more ML models get built into everyday software applications like photography, word processing, and spreadsheets. The more the trained models just become extensible to everyday driving ML usage will just become democratized to the point that it is expected instead of being impressive.
General vs. specific use - ML models now exist for so many different applications and they are tailored to highly specialized (specific) use cases. General models are probably coming at some point, but they are not here. Even the largest language models (foundational models) are specialized to language. We could see some of these models moving into more generalized use cases where they are working to do things. GTP-3 and copilot code development is veering in that direction.
Open source abounds - I’m still amazed that this technology is freely available. At some point I’m going to have to dig into the new government initiatives to encourage security within open source that recently was shared as a readout [2].
Learn about frameworks and pipelines - Frameworks are well developed and have been pressure tested at scale. Pipelines and automation save time. Fewer machine learning team members are needed to deliver this way. The process has been well documented, and the path is clear.
Success exists - Yeah, people have proven it works. A lot of the first-time-doing-this gotchas are managed away in this model based on leveraging community knowledge and practice.
Model management is a real thing - Serving multiple models and model management is hard. This is either going to end up getting built into the process and pipelines or completely simplified. Either way the efficacy of models will have to be controlled over time to avoid drift.
Data quality is key - None of this replaces the deep work required to wrangle the data.
As we reach the end of the 52nd post in this series we should take a look back at my previous efforts to sum up your machine learning readiness. I attempted to build out my consideration of that topic into 5 questions to help get you started or moving along your machine learning journey.
What are you actually trying to do? Do you understand the use case in a way that is definable and repeatable?
How are you collecting and storing data? You are going to need a method to collect data and understand your success metrics for tracking.
Are your use case and data collection/storage aligned?
Should you partner with a vendor, or are you ready internally to drive this forward? This is really a question about how you are going to bring your use case to production.
Have you considered the ROI, compendium of KPIs, and budget-level investments that are going to be required? Be strategic with your machine learning efforts. I’m surprised that this list did not start with that one when I first created it last year.
My biggest takeaway from this entire machine learning research process is that being planful into efforts that are definable and repeatable remains good advice. Almost as good as my central mantra on this one related to always being strategic with your machine learning efforts. It is entirely possible that as machine learning becomes more and more omnipresent and built into things that you are going to experience being included in the machine learning strategies of other organizations. This is already particularly true within the worlds of smartphones, IOT, and the vast network of APIs emerging. Just because you are a part of another organization's strategy does that mean you should abdicate your responsibility to consider and know your strategy. Be planful, be strategic, and consider the path forward along your machine learning journey and how to measure your success along that journey.
Links and thoughts:
“How to build an AI/ML powered priority lane”
“Jeff Dean: AI isn't as smart as you think -- but it could be | TED”
From Yannic Kilcher, “This Team won the Minecraft RL BASALT Challenge! (Paper Explanation & Interview with the authors)”
Top Tweet of the week:
Footnotes:
[1] Lindahl, N. (2022). The Lindahl Letter: On Machine Learning. Morrisville, NC: Lulu Press, Inc. https://www.lulu.com/en/us/shop/nels-lindahl/the-lindahl-letter-on-machine-learning/ebook/product-4gz6rj.html
[2] The White House. (2022, January 13). Readout of White House Meeting on Software Security. Briefing Room: Statements and releases. Retrieved January 16, 2022, from whitehouse.gov/briefing-room/statements-releases/2022/01/13/readout-of-white-house-meeting-on-software-security/https://www.whitehouse.gov/briefing-room/statements-releases/2022/01/13/readout-of-white-house-meeting-on-software-security/
What’s next for The Lindahl Letter?
Week 53: Machine learning interview questions
Week 54: What is a Chief AI Officer (CAIO)?
Week 55: Who is acquiring machine learning patents?
Week 56: Comparative analysis of national AI strategies
Week 57: How would I compose an ML syllabus?
I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed reading this content, then please take a moment and share it with a friend.