Nephi specifies that it's the leaders that cause the people to err:

"O my people, they who lead thee cause thee to err and destroy the way of thy paths." (2 Nephi 13:12, quoting Isaiah 3:12)

"...they have all gone astray save it be a few, who are the humble followers of Christ; nevertheless, they are led, that in many instances they do err because they are taught by the precepts of men." (2 Nephi 28:14)