The Essence of "Controlled Experiment"

One of the hallmark of science is the presence of a control set in an experiment. Sometimes this modality is known as "randomized trial and error". Many science teachers in elementary schools are cognizant of that and always try to "teach" students what does "control set" really mean. 

Alas, the nature of this "controlled experiment" technique is very subtle and not easy to be grasped. I myself made the mistake to gloss over it for decades. Until very recently I had an epiphany on the real essence of "controlled experiment".

For example, if we want to know if water is a vital ingredient for photosynthesis, we build 2 sets of plants, one with water and another without (while keeping the remaining variables fixed for both plants). The corollary is that 1 of the 2 sets of plants must be able to grow. If it is the one with water hence it is self evidential that water is required, if it is the one without water then it is otherwise. 

It sounds very simple. But until recently I only grasped a somewhat gruesome corollary of "having 1 plant set with water + 1 plant set without water":

1 of the 2 sets MUST eventually be an utter waste

If it turns out that water is required, your meticulous act of setting up "a set which the plant doesn't have water access" would (with hindsight) appear as super half-witted.

"Bro, it's so obvious that plants need water la, why did you waste time do all this shit? Nothing more productive to do?" The know-it-all guys would tell you that.

That is the essence of "controlled experiment" and "trial and error" that I only truly grasped recently. The "error" part in "trial and error" is so paramount that there is no way you could dispense it, lest the whole truth searching process would crumble. I could extend this to all kinds of examples:

• MOBA video games: If you want to know which item fits best for a hero, you have to play 10 games using the same hero but with 10 different items each game. Some games will turn out to be phenomenal if the item is right, but it is also almost certain that at least in 1 or 2 games (or more) you would look so bad and lose so catastrophically
• Medicines or vaccines: If you administer a pill or vaccine, the best way to improve its efficacy is a nefarious one: Give a lot of different drugs to alot of different people, those who died eating pill X is a proof that pill X doesn't work (then you proceed with pills that didn't cause deaths). You gain enormous leap of knowledge via death of human beings
• Personal life: How do I know what should I do and what I do best in life? Easy. Spend 10 years to do 5 diametrically different things for 2 years each. After 10 years, look at the data:

• 20-21 y/o: Did A (Pretty fun but not profitable)

• 22-23 y/o: Did B (Pretty sordid)

• 24-25 y/o: Did C (Pretty profitable)

• 26-27 y/o: Did D (Pretty boring)

• 28-29 y/o: Did E (Pretty challenging but agonizing)

• Then you disqualify obviously appalling choices such as B and D. Then maybe C is the "right" thing to do. But there is a problem, not only other people but even you yourself would loathe those years you spent on B and D (which were complete farce!). That is the essence of "controlled experiment" i.e. there must be some sets of experiments turn out to be so abominably wasteful that you would regret so much.

That leads to my conclusion: How paradoxical the tug of war between "bottom up controlled experiment" and "top down knowledge and planning" is.

I have demonstrated above how "wasteful" controlled experiment actually is. But the most exhilarating part of this mode is that it is an incredibly rigorous way to find out truth. By discarding the obviously wrong, the remaining sets become more probable. The upfront cost of controlled experiment is high, but it also promises you a gigantic return in (possibly very) long run.

The opposite, "Top down knowledge and planning" is so damn alluring. You are promised that you could know which item is the best for your MOBA game's hero without having to lose a game, you are promised that we could cure an illness without people dying as white rat, you are promised that you could build a great career without having to waste 2 to 6 years on something appalling!

"Why try XXX to waste time? Just YYY, it's so obvious"

The above is the canonical ethos of the know-it-all people. This modality not only has no upfront costs (unlike that of the "randomized trial and error" mode), it also unequivocally guarantees you returns like how a multi-level marketing salesman promises that you will surely profit.