Deans from some of the nation’s top medical schools met Thursday — not to talk about training doctors or weathering economic challenges — but to size up the people who grade them.

The sit-down between editors at U.S. News & World Report and the top brass at Harvard, Yale, Columbia and several other schools showed how seriously those in medicine’s ivory tower take the magazine’s annual rankings.

The ratings count when it comes to marketing to prospective students, staff recruits and philanthropists, the deans acknowledged. But if their comments at the conference — marking the 20th anniversary of the rankings — are any guide, the deans take them less seriously as a scientific gauge of what actually goes on at their schools.

“It’s just not measuring how we educate our students,” said Dr. Dennis Charney, dean of Mount Sinai Medical School (No. 18 on the U.S. News list, for what it’s worth), which hosted the event in New York. But, he acknowledged, the schools need “to come up with better ways” of appraising educational performance.

He and others picked apart the methods U.S. News uses to score med schools. The magazine, for instance, polls the directors of various residency programs to see what they think of the programs from which they draw their medical residents.

“Good in theory, very bad in the way it comes out,” is how Dr. Lee Goldman, dean of Columbia University College of Physicians and Surgeons (No. 10 on the list), characterized the residency director survey. The response rate is only 17 percent.

Others pointed out that most medical residencies only draw from a small number of med schools — so residency directors are really only speculating about the broader universe of schools they’re being asked about.

Yet, Charney conceded, the U.S. News ratings drive the behavior of prospective students and the schools. Sometimes that’s not for the best.

Take, for instance, diversity. The ranking system values selectivity and that penalizes schools that accept “students who come from tough backgrounds and are inspirational in what they’ve achieved” but who don’t have stellar test scores, Charney said.

Rewarding diversity, Charney said, would motivate schools “to go out and recruit a diverse population of students who may not have the very best metrics.”

Amid the criticism, the deans also stuck some conciliatory notes. “You’re trying to measure what’s essentially immeasurable, and you’re doing as good a job as anyone,” said Dr. Jules Dienstag, of Harvard Medical School (ranked No. 1).

U.S. News editor Brian Kelly, said the publication repeatedly urges applicants to do their own research and not lean too heavily on the rankings. “We do view it as one tool and have cautionary language in there,” he said. “Maybe we need something like what’s on the back of a pack of cigarettes, something like ‘Use at Your Own Peril.’ ”

This post is part of a reporting partnership that includes WNYC, and Kaiser Health News.