Heat oil in large pot over medium-high heat. Add leeks and fennel. Sauté until leeks are translucent, about 7 minutes. Add potatoes and sauté for a few minutes stirring constantly. Season with salt and pepper. Reduce heat to medium low and cover. Cook until potatoes are tender, stirring frequently. Add milk. Cook on medium low for 10 or 15 minutes stirring frequently. Ladle soup into bowls. Garnish with fennel fronds if desired.