I am following along to the coursera implementation of the transformer. For what it’s worth, it’s quite am ambitious exercise, taking students through the exact implementation step by step, as well as providing unit tests for each function along the way. However, I am at the part where I have implemented the encoder layer, and I feel like I am losing track - of all the different things happening, of all the weights and variables produced, and of the series of transformations on the data. ...

For my assignment, I have to partially implement the T5 transformer model. However, the only thing I have for implementing it is intuition. I don’t understand many parts of it deeply enough. And the full code for the implementation is sprawling - and also, because of its sprawling nature, makes breaking down the components difficult. There is quite a good guide for it here, where the authors take us through each aspect of its implementation with notes. I think this is a good starting point. ...

So I was playing with Pytorch over the weekend, trying to get my fundamentals right by following along to the tutorials in the documentations. In my first exercise, I wanted to try to implement a shallow neural network with 1 hidden layer (784-4-10) to try to classify images from the FashionMNIST dataset. I have learned quite a bit about what’s needed to get it right. Model illustration generated by ChatGPT: ...

A historic event, reproduced below. Apology and reflection 今日头条的朋友们： Dear friends of Jinri Toutiao: 我真诚地向监管部门致歉，向用户及同事们道歉。 从昨天下午接到监管部门的通知到现在，我一直处在自责和内疚之中，一夜未眠。 I earnestly apologise to regulatory authorities, and to our users and colleagues. Since receiving the notice yesterday from regulatory authorities, I have been filled with remorse and guilt, entirely unable to sleep. 今日头条将永久关停“内涵段子”客户端软件及公众号。产品走错了路，出现了与社会主义核心价值观不符的内容，没有贯彻好舆论导向，接受处罚，所有责任在我。 Jinri Toutiao will shut down once and for all its “Neihan Duanzi” app and its public accounts. Our product took the wrong path, and content appeared that was incommensurate with socialist core values, that did not properly implement public opinion guidance — and I am personally responsible for the punishments we have received [as a result]. ...

My skateboard routine is fairly standard. I’ll pack my gear into my Adidas backpack, throw in a bottle and my keys, and head out the door grabbing my skateboard and the helmet. I don’t bother bringing my phone. I’ll get to my bike parked beneath my block, mount the skateboard onto the backseat, and cycle towards the direction of Bukit Batok MRT - passing by the station, passing by Princess Elizabeth Primary, over the green fields before reaching Yuhua, followed by Chinese Garden MRT, and then finally reaching the skatepark at Lakeside. The entire journey takes around 20mins or so, and I treat it as warm up. ...

I had a fascinating lecture earlier this week with Prateek Saxena as part of my Web Security course, where we learned about a core pillar of web security, the same origin policy. Briefly speaking, the same origin policy disallows different web apps from accessing each other’s content and resources through the browser. It’s a framework for access control. Just like how User A cannot access User B’s files on a Mac OS, website A cannot access website B’s HTML, Javascript, CSS, by default. ...

In my first lesson of the current course I am taking, BMD5302 - Financial Modelling for Fintech professionals, I was introduced to first to VBA, and then to a couple of popular functions in Excel. I mean, I was mindblown a little bit by VBA. I’ve heard of it before, but I’ve never known what it does. Now I know: it is a way for you to control excel programmatically. The syntax seems easy enough, and very quickly I can see how it can be really powerful. It’ll allow you to automate almost everything that you need in excel. ...

Warning: massive spoiler alert for Death’s End. Reader’s discretion advised. A brain gets shipped to outerspace, embeds itself into Trisolaris civilization, finds success as a fairy tale writer, and rendevous with his crush deep in space to send a cryptic warning back to human civilization in the form of a fairy tale. Four dimensions collapsing into three, and three dimension ultimately collapsing into two. A tale that spans billions of years, dimensions, and universes, yet tied together by a singular thread of human (the writer’s?) warmth. ...

My NextJs application had a problem: whenever it did CRUD operations, the server takes around 8 seconds to respond on first attempt, and then 2/3 seconds on subsequent attempts. This is the so called cold start problem that Vercel is known for. The source of the latency does not come from the DB as I’ve tested it in development mode against the same DB and there was no latency there. ...

Wow today I learned that 1 radian is actually the angle created by the length of the radius on a circle. And that 180degrees is simply 3.14 radians. Can’t believe I’ve never known this relationship.