During the last lecture we jumped in and looked at 10 machine learning algorithms. This week the content contained within this lecture will cover from a machine learning perspective reinforcement learning and 3 types of supervised learning. Those types of supervised learning will include the general use case of supervised learning, unsupervised learning, and the super interesting semi-supervised learning. Like the model for consideration used in the last lecture I’ll cover the topics in general and provide links to papers covering the topic to allow people looking for a higher degree of depth to dive deeper into academic papers to achieve that goal. My general preference here is to find academic papers that are both readable and are generally available for you to actually read with very low friction. Within the machine learning and artificial intelligence space a lot of papers are generally available and that is great for literature reviews and generally for scholarly work and practitioners working to implement the technology. My perspective is a mix between those two worlds which could be defined as a pracademic view of things. All right; here we go.
Reinforcement learning - Welcome to the world of machine learning. This is probably the first approach you are going to learn about in your journey. That’s right, it's time to consider for a brief moment the world of reinforcement learning. You are probably going to need to start to create some intelligent agents and you will want to figure out how to maximize the reward those agents could get. One method of achieving that result is called reinforcement learning. A lot of really great tutorials exist trying to explain this concept and one that I enjoyed was from Towards Data Science way back in 2018 [1]. The nuts and bolts of this one involve trial and error with an intelligent agent trying to learn from mistakes using a maximization of reward function to avoid going down paths that don’t offer greater reward. The key takeaway here is that during the course of executing a model or algorithm a maximization function based on reward has to be in place to literally reinforce maximization during learning. I’m sharing references and links to 4 academic papers about this topic to help you dig into reinforcement learning with a bit of depth if you feel so inclined.
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996). Reinforcement learning: A survey. Journal of artificial intelligence research, 4, 237-285. https://www.jair.org/index.php/jair/article/view/10166/24110
Sutton, R. S., & Barto, A. G. (1998). Introduction to reinforcement learning. https://login.cs.utexas.edu/sites/default/files/legacy_files/research/documents/1%20intro%20up%20to%20RL%3ATD.pdf
Szepesvári, C. (2010). Algorithms for reinforcement learning. Synthesis lectures on artificial intelligence and machine learning, 4(1), 1-103. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.308.549&rep=rep1&type=pdf
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. https://arxiv.org/pdf/1312.5602.pdf
Supervised learning - You knew it would only be a matter of time before we went out to some content from our friends over at IBM [2]. They note that within a world where you have some labeled datasets and are training an algorithm to engage in classification or perhaps regression, but probably classification. In some ways the supervised element here is the labeling and guiding of the classification. Outside of somebody or a lot of people sitting and labeling training data the supervision is not from somebody outright sitting and watching the machine learning model run step by step. Some ethical considerations need to be taken into account at this point. A lot of people have worked to engage in data labeling. A ton of services exist to help bring people together to help do this type of work. Back in 2018 Maximilian Gahntz published a piece in Towards Data Science that talked about the invisible workers that are doing all that labeling in large curated datasets [3]. Within the world of supervised learning being able to get high quality labeled data really impacts the ability to make solid models. It’s our ethical duty as researchers to consider what that work involves and who is doing that work. Another article in the MIT Technology Review back in 2020 covered the idea of how gig workers are powering a lot of this labeling [4]. The first academic article linked below with Saiph Savage as a co-author will cover the same topic and you should consider giving it a read to better understand how machine learning is built from dataset to model. After that article, the next two are general academic articles about predicting good probabilities and empirical comparisons to help ground your understanding of supervised learning.
Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C., & Bigham, J. P. (2018, April). A data-driven analysis of workers' earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI conference on human factors in computing systems (pp. 1-14). https://arxiv.org/pdf/1712.05796.pdf
Niculescu-Mizil, A., & Caruana, R. (2005, August). Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning (pp. 625-632). https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.60.7135&rep=rep1&type=pdf
Caruana, R., & Niculescu-Mizil, A. (2006, June). An empirical comparison of supervised learning algorithms. In Proceedings of the 23rd international conference on Machine learning (pp. 161-168). http://www.cs.cornell.edu/~alexn/papers/empirical.icml06.pdf
Unsupervised learning - It’s a good thing that you were paying very close attention to the explanation of supervised learning above. Imagine that the humans or in some cases the vast collectives of humans labeling training sets just stopped doing that. Within the unsupervised learning world the classification within the machine learning problem space is going to be handed differently. Labeling and the creation of classification has to be a part of the modeling methodology. This topic always makes me think of the wonderful time capsule of a technology show about startups called Silicon Valley (2014 to 2019) that was broadcast by HBO. They had an algorithm explained at one point as being able to principally identify food as hot dog or not hot dog. That’s it the model only could do the one task. It was not capable of correctly identifying all food as that is a really complex task. Trying to use unsupervised learning for example, based on tags and other information identifying different types of food in photographs is something that people have certainly done with unsupervised learning approaches. I’m only sharing one paper about this approach and its from 2001.
Hofmann, T. (2001). Unsupervised learning by probabilistic latent semantic analysis. Machine learning, 42(1), 177-196. https://link.springer.com/content/pdf/10.1023/A:1007617005950.pdf
Semi-supervised learning - All 3 of these different types of learning supervised, unsupervised, and semi-supervised are related. They are different methods of attacking a problem space related to learning as part of the border landscape of machine learning. You can imagine that people wanted to try to create a hybrid model when a limited set of labeled data is used to help begin the modeling process. That is the essence of the process of building out a semi-supervised learning approach [5]. I’m sharing 3 different academic papers related to this topic that cover a literature review, a book about it, and the more advanced topic of pseudo labeling.
Zhu, X. J. (2005). Semi-supervised learning literature survey. https://minds.wisconsin.edu/bitstream/handle/1793/60444/TR1530.pdf?sequence=1
Chapelle, O., Scholkopf, B., & Zien, A. (2009). Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks, 20(3), 542-542. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=4787647
Lee, D. H. (2013, June). Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML (Vol. 3, No. 2, p. 896). https://www.kaggle.com/blobs/download/forum-message-attachment-files/746/pseudo_label_final.pdf
Conclusion - This lecture covered reinforcement learning and 3 types of supervised learning. You could spend a lot of time digging into academic articles and books related to these topics. Generally, I believe you will start to want to look at use cases and direct your attention to highly specific examples of applied machine learning at this point. Fortunately, a lot of those papers exist and you won’t be disappointed.
Links and thoughts:
“[ML News] This AI completes Wikipedia! Meta AI Sphere | Google Minerva | GPT-3 writes a paper”
Top 4 Tweets of the week:
Footnotes:
[1] https://towardsdatascience.com/reinforcement-learning-101-e24b50e1d292
[2] https://www.ibm.com/cloud/learn/supervised-learning#toc-unsupervis-Fo3jDcmY
[3] https://towardsdatascience.com/the-invisible-workers-of-the-ai-era-c83735481ba
Research Note:
You can find the files from the syllabus being built on GitHub. The latest version of the draft is being shared by exports when changes are being made. https://github.com/nelslindahlx/Introduction-to-machine-learning-syllabus-2022
What’s next for The Lindahl Letter?
Week 84: Neural networks (ML syllabus edition 5/8)
Week 85: Neuroscience (ML syllabus edition 6/8)
Week 86: Ethics, fairness, bias, and privacy (ML syllabus edition 7/8)
Week 87: MLOps (ML syllabus edition 8/8)
Week 88: The future of publishing
I’ll try to keep the what’s next list forward looking with at least five weeks of posts in planning or review. If you enjoyed this content, then please take a moment and share it with a friend. If you are new to The Lindahl Letter, then please consider subscribing. New editions arrive every Friday. Thank you and enjoy the week ahead.
Share this post