Halver Lee

I'm stuck in tracing my family heritage. Here is what I have so far. According to the 1880 US Federal Census Halver F. Lee (my great great grandfather) lived in Lisbon, Yellow Medicine, Minnesota. He was born in Norway in 1824 and came to America in about 1872. With him was his wife Ingebor, born in 1827 in Norway, and his children Andrew and Ole. Another son Elling, was born in Minnesota. I'm not sure where to go from here to trace my family back farther. Any suggestions? Thanks.