One option: The CHOMP microfiber cleaning “smop” (sponge ... As long as you are scrubbing your gutters, you might as well scrub your siding, too.
Or you can use a cleaner sold for gutter stain removal, such as Chomp, Gutter Zap or Streak Getter. These break the bonds so the asphalt bits and other debris fall off or rinse away. These ...