By Philippe Kruchten
This talk is not science, nor engineering, but a story of his own experience. So, you’re working on some topic in software architecture or something alike. But why? How do you define success? Is it following the KPIs (H-index, job, pass, $$$, #users, etc.), or? Frameworks are there to conceptualize, and to filter your thoughts. His PhD filter is “how much science and how much engineering?”, “how valuable (impact) and valid is it, and how is it communicated?”. So, how to read a thesis / paper? This is his filter engine:
Main point: how are the claims in the conclusions supported in the body of the manuscript? What is the evidence to support the claims? Next, is the evidence supported? Is there a validation? How thorough is the validation? The related work helps in seeing whether the person looked around enough. This way, you understand the work communicated in the manuscript. Then, look again to the abstract and title: does it cover what I understood from it, and check the conclusions and references again.
Then, how is the balance between engineering and science? Where is Software architecture actually? What is a researcher in software architecture? On the science side, we want to create a better understanding, e.g. a model, (conceptually, mathematically, etc.). On the engineering side, we want to build tools and applications. The danger is on both sides: science can go bad (no grounding in evidence, not a new idea, etc.), but engineering as well: yet another mousetrap, does not solve a problem anyone has (anymore), or it does not scale to real-life problems. Where are you on the scale?
Next question: what is the best research method? According to Philippe, it does not exist! More important is explain what you do! You do not need to be fully compliant, but should explain very clearly what you do, and why!
Building a tool might be very nice engineering, but only little part of it is PhD material. Validation of a tool is very hard. Also, it solves a single problem, and is not integrated in the complete development kit (tool in isolation), so, is it going to be used at all? Most of the time, you can only build a prototype, and more important: how sustainable is your tool? Can you still support it after some years? In the science side there are traps as well: “everything is a graph”, or “everything can be reduced to second order logic”…
Next point is validity: how much is validated? Or, how much do I believe what you claim? Don’t just repeat the definitions, but actually show how you mitigated each of the elements. Be clear about your contribution. It is quite okay to change, don’t be a slave of your supervisor. It is your path to a PhD, not your supervisor’s. Be clear in your head, in motivation, impact and approach. And above all: behave ethically!
In the end, take the time to really achieve success the way you defined success. Optimize for that form. Communicate about your ideas and contributions. But also, be careful with what you share. Some personal gripes of Philippe are good reminders: not all knowledge is captuered in journal papers indexed by IEEExplore, (or web of sci, or scopus, or …), and “renaming old concepts is not innovation, just obfuscation”! Notice that software engineering students are not representives of the software development population. Important to realize is that in science, we are not at a battlefield. If somebody does similar things, don’t regard him as the enemy, but become close friends, and work together!