The spec doesn't specify how to compute the destination of a request, it only links to the fetch spec which contains this incomplete table and current implementations in Chrome and Firefox sometimes don't follow that table and also disagree with each other (for example they both set destination to "" for top-level navigations and Chrome sets it to "" for fonts).
Related: #698 - the initiator type can be used as another hint to distinguish the different kinds of requests.