Sami Lineage

Family lore has it that my Norwegian lineage is from the Sami. My great grandfather was labeled a "black Norwegian" due to his swarthy skin color. If I was to trace my ancestry back far enough, what would I look for to indicate a Sami connection? Is there a particular group of "farm names" that represent the Sami ? (I've presently traced my paternal side back to the late 1700's)