End Semester Recap

I just finished all the exam and papers today. It has been a long day (wake up at 6) and I feel very exhausted. However, I want to do a quick recap of this semester before my judgment affected by my final grades.

Courses

CS 380D Distributed System

My first exam is a disaster. The exam is all about system design + understanding of RAFT. I didn’t get used to the system design in general. All I do is to remember every detail of some system implementations, which usually don’t matter from a design perspective. Vijay has been emphasized this point a lot but I didn’t get it until the second half of the course. The course is good and the biggest takeaway for me is two:

  • Can comfortably read distributed system paper. I cannot claim I can read all types of system paper but for distributed system paper, I begin to get the momentum and start to know where to focus on during the reading. Takes a lot of struggling to get this point but I’m happy overall after reading more than 30 papers.
  • Got intrigued by the distributed system and storage system. In the past, I have been struggling to find my research interests.  But, thanks to this course, I become more intrigued with the combination of distributed system and storage. Right now, I like storage more. I read tons of LSM-based storage paper to find a topic for my final course project. I really enjoy the moment to read LevelDB and PebblesDB’s code and enhance them in some way. That further makes me want to know more about SSDs and HDDs.

CS 388 Natural Language Processing

I trade this course with algorithm class. I have a mixed feeling right now. On one hand, unlike the NLP course that I take in the previous semester, which looks at NLP from models perspectives (HMM, CRF, different networks). this semester’s course is from more traditional linguistics + machine learning perspectives. I really like this part. Overall, I strongly believe linguistics domain knowledge should play the key role in NLP study not various deep learning manic.  First two homework, we look at language models and LSTM based on the intuition of prediction can be two ways. I really like Mooney’s view that you always think about intuition whether the model can work or not instead of mindlessly applying models.  Like last semester’s NLP class, my interests with class declines as the semester progresses partly due to the fact that the material is no longer relevant for homework and exam. That is my bad.

The final project is on VQA, which mostly done by my partner. I only gather the literature and survey the field plus some proofreading. I’m OK with that as I want to have more time working on my system project and my partner wants to work alone in the modeling.  This leads to my lesson learned from the class:

  • Graduate school is about research, not class. Pick the easiest courses and buy yourself time to work on the research problem that attracts you.

If I look back right now, I want to take algorithm class instead. My thoughts to NLP is that I want to start from the dumbass baseline and know the history of the field. If you think about NLP, the most basic technique is just regular expression pattern matching. But, how do we go from there to more complex statistical models is the most interesting point I want to learn.

LIN380M Semantics I

The course is taught by Hans Kamp, which I believe invents the Discourse Representation Theory (DRS). Really nice man. I learn the predicate logic, typed lambda calculus, Montague grammar and DRS. Very good course for the logic-based approach to derive the semantic meaning of a sentence. However, I do feel people in this field put a lot of efforts in handling rule-based exceptions like how do we handle type clash in Montague grammar. When I turn in the final exam, Hans is reading some research paper. He is still doing research and that inspires me a lot.

Other Lesson Learned

  • “Don’t be afraid to fail, be afraid not to try”. I learn a lot from my final system project partner. Reading complex code can be daunting but we can always start to play around even when we cannot understand the code fully. There is a great deal of psychological barrier to be overcome. My partner always starts with reading and then writing. Once bug happens, he is happy because the bug is an indicator of progress, which eventually leads to working code.
  • Work independently. When I got stuck for a while, I always want to seek help instead of counting on myself to solve the problem. It seems that I can never trust myself ability to solve the problem. By observing how my project partner solves the problem, I learn a lot. Start to trying and always seek for the root cause of the problem and situation changes as long as you start trying.
  • Some tips about system paper writing:
    • Use hatch on the bar graph. People may print out their paper in black and white. Use hatches on the bar graph help them to distinguish which bar is your system and which bar is the baseline system.
    • Add more descriptions to each figure and table below. I used to think that there should be only one line of description for each picture. But, as pointed out by my another project partner, people need instructions when read the graphs. People love the pictures and they hate to go to the paragraphs to search for the instructions to understand the graph. Thus, put instructions directly below the picture. Great insight!
  • I really want to know how to measure a system accurately.  From my system project, I realize that measuring the system performance is really hard. Numbers fluctuate crazily and you have no clue why is that because there are some many layers of abstraction  & factors in the experiment environment that can potentially impact the system measurement. I really want to know more about this area during my own study and summer internship.
  • System improvement without provable theoretical guarantees will be very unlikely successful. Overhead or the constant factor hidden in the big-O model usually dominate the actual improvement you might think you can get. For example, there are overhead in spawning threads. We need to compare how much we can get by having multiple threads running in parallel to do the subtask vs. having one single thread do the whole thing. PebblesDB’s paper on the guard and improvement to compaction ultimately prove that we really need to think more before getting our hands dirty. By reading the paper, I get the feeling that they know the system will work before even implementing one because they can clearly show that their functionality works before writing a single line of code. I need to develop more sense about this and taking more theory class.

 

Ok. Time to pack and catch the flight.

Advertisements

On Reading CS Papers – Thoughts & Reflections

Be forewarned:

  • This is not an advice post. There are tons of people out there who desperately want to give people advice on reading papers. Read theirs, please.
  • This post is a continuous reflection on the topic “how to read a CS paper” from my personal practice. I will list out my academic status before each point so that it may be interesting to myself on how my view on the matter has changed as time goes forward.

2018

The first year of my CS master program. Just get started on CS research.

  • It’s OK to not like a paper

In my first semester, I majorly read papers on Human Computation and Crowdsourcing.  Very occasionally, I read papers on NLP. Some papers on NLP are from extra readings in Greg’s course. Some are related to Greg’s final project, which deals with both code and language.  I don’t really like and want to read papers back then. In NLP class, I prefer to read textbooks (Jufrasky’s one) and tutorial posts that I can find online. One roadblock for me to read papers is that there is certain background knowledge gap I need to fill and I just simply don’t know how to read a paper. So, for Greg’s NLP course, I only read some papers related to my final project. This paper is the base paper for my final project. I got this paper from professors in linguistics and software engineering and they want me to try out the same idea but using neural network model instead. I read this paper several times and the more I read, the more I want to throw up.  I just think this paper hides many critical implementation details and the score 95% is just too high for me to believe. The authors open source their code but their code has some nasty maven dependencies, which won’t compile under my environment. Their evaluation metric is non-standard in NLP and many “junk words” wrap around their results. Of course, the result of my experiment is quite negative.  I often think it is just a waste of life to spend your precious time on some paper you dislike.  Here, I’m more of talking about paper writing style and the reproducibility of papers’ results. I probably want to count shunning from some background gap as a legitime reason not like a paper.

  • Try to get most of the paper and go from there

I got this message from Matt’s Crowdsourcing class. In the class, I have read a very mathematical heavy paper, which invokes some combinations of PGM and variational inference on the credibility of fake news. I’m worried back then about how should I approach a paper like this one, which I’m extremely lack of background and mathematics formula looks daunting.  I pose my doubts on Canvas and Matt responds in class and gives the message.  I think the message really gives me some courage on continuing read papers.

  • It’s OK to skip (most) parts of a paper.  Remember: paper is not a textbook!

This semester I’m taking a distributed system class. To be honest, distributed system paper can be extremely boring if they are from industry. Even worse, system paper can be quite long: usually around 15 pages, double column. So, if I read every word from beginning to end, I’ll be super tired and the goal is not feasible for a four-paper-per-week class. So, I have to skip. Some papers are quite useful maybe just for one or two paragraphs. Some papers are useful maybe just because of one figure. As long as your expectation about a paper gets met, you can stop wherever you want.

  • Multiple views of reading a paper

I didn’t get the point until very recently. I did quite terrible on the first midterm of my distributed system class. The exam is about how to design a system to meet a certain requirement. In the first half of the course, I focus on the knowledge part presented by the paper but that doesn’t work out well. Until then, I realize that I need to read those systems paper from a system design point of view: what problems they need to solve, what challenges they have, how they solve the challenges.  OF course, those papers are valuable from knowledge perspective: how consistent hashing works, for example. But, depends on the goal of reading paper, I can prioritize different angles of reading a paper. If I need to implement the system mentioned in the paper, I probably need to switch to a different paper reading style.

  • Get every bit of details of paper if you need to

It’s time again for the final course projects. Again, I need to generate some ideas and find some baseline papers. In this case, “skip parts” and “get most out of the paper and move on” strategy probably won’t work well. All in all, I need to understand the paper and those are rely on the details from the paper. In this case, I need to sit through the whole journey and remove any blockers that I may encounter.

Freedom of speech

Piazza is an online forum tool that is heavily used in the academia. It is used to help students ask questions and get feedback from both peers and instructors. It has a goal that is similar to Slack in the sense that they both try to cut the duplicate emails sent by several people for the same or similar type of request. It is a good tool but every tool that comes with power has its own consequence.

Instructors can perform the following configuration when they setup the forum for the course.

Screen Shot 2018-02-09 at 11.58.12 PM

Basically, this option means that when you make a post, whether you can choose to be “Anonymous” to both your peers and instructors or to your peers only (instructors can still see who makes the post).  The following picture shows what this option looks like from student’s perspective:

Screen Shot 2018-02-09 at 11.58.33 PM

The intention for this option I guess is that some students may feel embarrassed to ask questions. They might think their questions are dumb and will make them look bad in front of peers or instructors. I think this option is used as a way to encourage students to ask questions bravely.

However, this option may get abused. From my observation, Piazza is used as a way for instructors to show off their teaching quality. This is important for Assistant Professors because teaching still means something (if teaching quality doesn’t matter, why institution asks for the teaching statement at the very first place?). In addition, the teaching quality in some sense is an important indicator for students to evaluate you as a person. This is important for professors who are looking for graduate students because research publication is only part of the story and how those professors interact with students may be a crucial indicator to how good a professor as a human being is (evaluation may be a better indicator but it is confidential). Thus, if some potential students look at the piazza that his interested professor teaches gets a lot of complaints. The students may have a second thought on whether he should work with him for research (maybe he is a very bad person even he is doing a good research).

Thus, the instructors have a strong motivation to censor the posts on the piazza. This scares the students because they don’t have a secure way to provide feedback to the instructor. Let’s assume that the majority of students has a good heart: they won’t say bad stuff to the instructor who actually really cares about students. Thus, the time that something slightly negative appears on the Piazza may be a very important signal to the instructor that something wrong with his teaching. However, due to the strong motivation for instructors to show off their teaching quality through Piazza, the instructors may start to censor the speech on the Piazza by turning the option off.

I didn’t realize this thing last semester. Last semester, the instructor from one course sets this option off and I was thinking maybe he wants to know the students who are shy to ask questions and provide some individual attention. However, this semester, the instructor from one of my course initially turn the option on so that everyone can truly ask questions as “Anonymous”. Then, until one day, someone makes the below post and the option is turned off. Now, no students dare to make slightly negative posts.

Screen Shot 2018-02-10 at 12.28.57 AM

 

I fully understand the interests conflict between students and instructors on the use of Piazza: students may think Piazza is a secure way to provide anonymous feedback while instructor may think bad posts on the forum make them look bad. However, I still think there should be a better way to address this conflict to protect both students and instructors especially with the technology we have nowadays. But, (unintentional) censorship is not something we want to culture especially in the Academia. By the way, for this course, I still think the instructor is good but the material is quite challenging without laying down a solid theoretical foundation beforehand. He went through the material again after this post but too bad the truely “Anonymous” is gone.