StaticIndex: process tokens from include! expansions

[nicolas-guichard]

This updates the static index indexer to traverse macro expansions, but only actually records tokens from include! expansions.

The reason for not recording tokens from all expansions is twofold:

• I didn't find a way to reliably get the tokens' spelling location, original_node_file_range_rooted returns the macro call location.
• That would register a huge amount of tokens at macro call sites.

Marking as draft because it depends on #22530 (not functionally, but it would conflict otherwise).

For context this was motivated by https://bugzilla.mozilla.org/show_bug.cgi?id=2053847.

[ChayimFriedman2]

I didn't find a way to reliably get the tokens' spelling location, original_node_file_range_rooted returns the macro call location.

Only if it does not have a better span.

Generally macros are already descended, which means that tokens that have a direct mapping into tokens in the call will be included (with the semantics of the expansion). Tokens that are not we generally don't want to include (though I'm not an expert in SCIP and it might have different requirements). include!() might be special though because it doesn't have direct mapping but we might still want to include it, not sure. I also didn't look at the code yet.

[nicolas-guichard]

My test was a simple:

macro_rules! define_struct {
  () => {
    struct Struct;
  };
}

define_struct!();

In that case calling original_node_file_range on the Struct token from the macro expansion returns the define_struct call.

[ChayimFriedman2]

Indeed, because the span is associated with the macro definition and not anything inside the call. Such tokens we usually don't include in things, as I said.