Best seen in steps.
fn(3) requires mystery_method to be evaluated first.
mystery_method(c(function(x) x + 1, function(x) x * x)) evaluates to...
function(z) Reduce(function(y, w) w(y), c(function(x) x + 1, function(x) x * x), z)
Now, we can see the 3 in fn(3) is supposed to be z, giving us...
Reduce(function(y, w) w(y), c(function(x) x + 1, function(x) x * x), 3)
This Reduce call is wonky, taking three arguments. A three argument Reduce call will initialize at the third argument, which is 3.
The inner function, function(y, w) w(y) is meant to take an argument and a function and apply that function to the argument. Luckily for us, we have some functions to apply.
That means we intialize at 3 and apply the first function, function(x) x + 1. 3 + 1 = 4.