Place potatoes (whole and unpeeled) in a large saucepan; cover with water. Add the rosemary, bay leaf and salt and bring to a boil. Reduce heat and simmer until potatoes are tender, about 25 minutes. Let cool in the cooking liquid. Once cool, cut into quarters lengthwise.
Combine mayonnaise, shallots, celery, capers, anchovies, garlic, mustard, vinegar and black pepper in a small bowl; mix well. In a large bowl, combine potatoes with mayonnaise mixture and parsley; toss until well coated. Season to taste with coarse salt and serve chilled.