You are right, measuring forecast accuracy should depend on what you are using the forecast for.
In our business (pharmaceuticals) we measure the actual demand vs the forecast at the time we had to use it. So our commercial colleagues send us a forecast 8 weeks out to allow us to buy components and raw materials. So when we get the final order 2 weeks out, we compare it to the forecast at 8 weeks and play the results back to them. For info we report both accuracy and bias, as we find different markets tend to have different bias. (Some over forecast, while other under forecast)
We are increasingly trying to be less and less reliant on forecasts, using them more to adjust inventory policy than to do the ordering (certainly for runners and repeaters). For our 'strangers' our purchase requirements tend to be well below MOQ for our suppliers anyway, so we work a re-order point process.
John Hardwick
FIOM