I’ve just begun my second deep learning course with Prof Stanley Kok at NUS - for FT5011. Based on the first lecture alone, I have a feeling that this is going to be a great course. Stanley seems to be a very good explainer of concepts, and despite having learnt deep learning and neural networks before, there were still nuances from the first lecture that I want to write about. ...
And so, I am reproducing it here for reference. My prompt: What are some best methods of gaining mastery in doing deep learning, for instance, dealing with quirks of different environments, shapes, data types, etc? Its response: Great question—and honestly, the fact that you’re thinking about quirks (envs, shapes, dtypes) is already a sign you’re moving from “tutorial deep learning” toward real mastery. I’ll break this into mindsets, habits, and deliberate practice loops—because deep learning skill compounds less from reading papers and more from debugging scars. ...
Training a CNN from scratch only got my accuracy up till 70%, whereas with transfer learning, I managed to get my accuracy up to 90%. What do I mean by transfer learning here? It means that I used a Resnet50 pretrained model as a departure point, and only fine tuned the final linear layers. The entire model implementation looks like this: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 def build_model(num_classes=2, freeze_backbone=True): transfer_model = models.resnet50(weights='DEFAULT') if freeze_backbone: for param in transfer_model.parameters(): param.requires_grad = False transfer_model.fc = nn.Sequential( nn.Linear(transfer_model.fc.in_features, 500), nn.ReLU(), nn.Dropout(), nn.Linear(500, num_classes) ) for param in transfer_model.fc.parameters(): param.requires_grad = True device = torch.device("cuda" if torch.cuda.is_available() else "cpu") transfer_model = transfer_model.to(device) return transfer_model Here, I rely on a build_model function to instantiate my model. ...
Learnings: Using Alexnet model with kernel settings of size=11, stride=4, padding=2, and image size of 227x227, I could reach 70% accuracy with training of 30 epochs. This, unfortunately, is the same level of accuracy as my Simplenet in the previous post Training Using a simple neural network from the previous post, I could reach an accuracy level of 0.7. However whilst implementing the following CNN from chapter 3 of the Pytorch book, I realized that I had quite a few issues trying to get it to converge. ...
After having completed my CS5242 course on neural network and deep learning, I realized that there’s a lot more practice that I’ll need to do in order to deploy useful models out into the wild. Which means that I would really want to work on and fully master the fundamentals. One of them being Pytorch. And so, I began on a Pytorch book by Ian Pointer. In the first exercise, I implemented a simple 2 layer neural network to classify whether an image is a cat or a fish. This is not new to me, but still, I think there’s a couple of things about this implementation that I should learn about: ...
So I’ve decided to revive this project after a long while haha. What exactly is this project? Idk, technical writing, as well as sporadic posts about life. I have a substack, but that’s a little more for polished posts about general topics in life, whereas this one is for rough sketches of everything I find interesting, as well as an online documentation of my journey with certain things. And, as the current domain name suggests, I am officially in my deep learning phase in my life, and I want to carve out a space to write about everything that I learn on that front. ...
I am following along to the coursera implementation of the transformer. For what it’s worth, it’s quite am ambitious exercise, taking students through the exact implementation step by step, as well as providing unit tests for each function along the way. However, I am at the part where I have implemented the encoder layer, and I feel like I am losing track - of all the different things happening, of all the weights and variables produced, and of the series of transformations on the data. ...
For my assignment, I have to partially implement the T5 transformer model. However, the only thing I have for implementing it is intuition. I don’t understand many parts of it deeply enough. And the full code for the implementation is sprawling - and also, because of its sprawling nature, makes breaking down the components difficult. There is quite a good guide for it here, where the authors take us through each aspect of its implementation with notes. I think this is a good starting point. ...
So I was playing with Pytorch over the weekend, trying to get my fundamentals right by following along to the tutorials in the documentations. In my first exercise, I wanted to try to implement a shallow neural network with 1 hidden layer (784-4-10) to try to classify images from the FashionMNIST dataset. I have learned quite a bit about what’s needed to get it right. Model illustration generated by ChatGPT: ...
A historic event, reproduced below. Apology and reflection 今日头条的朋友们： Dear friends of Jinri Toutiao: 我真诚地向监管部门致歉，向用户及同事们道歉。 从昨天下午接到监管部门的通知到现在，我一直处在自责和内疚之中，一夜未眠。 I earnestly apologise to regulatory authorities, and to our users and colleagues. Since receiving the notice yesterday from regulatory authorities, I have been filled with remorse and guilt, entirely unable to sleep. 今日头条将永久关停“内涵段子”客户端软件及公众号。产品走错了路，出现了与社会主义核心价值观不符的内容，没有贯彻好舆论导向，接受处罚，所有责任在我。 Jinri Toutiao will shut down once and for all its “Neihan Duanzi” app and its public accounts. Our product took the wrong path, and content appeared that was incommensurate with socialist core values, that did not properly implement public opinion guidance — and I am personally responsible for the punishments we have received [as a result]. ...