ASHINGTON, Feb. 23 - The Supreme Court ruled on Wednesday that a California prison policy that temporarily segregates new or newly transferred inmates by race is constitutionally suspect and should be evaluated by the same searching judicial scrutiny that applies to other government policies that classify by race.

The 5-to-3 decision overturned an appeals court ruling that upheld the policy, defended by California officials as necessary to curb violence by gangs. In that ruling, the United States Court of Appeals for the Ninth Circuit examined the segregation under the relaxed standard of review the Supreme Court generally applies to prison policies.

That approach was erroneous, Justice Sandra Day O'Connor wrote for the majority on Wednesday.

"In the prison context, when the government's power is at its apex," Justice O'Connor said, "we think that searching judicial review of racial classifications is necessary to guard against invidious discrimination."

The court stopped short of declaring the policy unconstitutional, although that was the implication both of Justice O'Connor's opinion and of a concurring opinion by Justice Ruth Bader Ginsburg. Rather, the court returned the case to the Ninth Circuit with instructions to re-evaluate the policy under strict scrutiny, a legal standard under which the government must show that a challenged action is both necessary and "narrowly tailored" to achieve a "compelling" state interest.

While joining Justice O'Connor's opinion, Justice Ginsburg also wrote separately to preserve her position that not all racial classifications should be subject to strict scrutiny.

Citing her own concurring opinion in the case two years ago that upheld an affirmative-action admissions policy at the University of Michigan Law School, Justice Ginsburg said that "measures taken to hasten the day when entrenched discrimination and its aftereffects have been extirpated" should be judged under a more relaxed standard. But she said she agreed in the prison case "that the stereotypical classification at hand warrants rigorous scrutiny."

Justices David H. Souter and Stephen G. Breyer signed both Justice Ginsburg's opinion and Justice O'Connor's majority opinion. Justice Anthony M. Kennedy signed only the O'Connor opinion.

The three dissenters from the court's judgment - Chief Justice William H. Rehnquist did not participate in the case, which was heard in November while he was undergoing cancer treatment - divided into two opposing camps.

Justice John Paul Stevens said that because in his view the segregation policy was unconstitutional under any legal standard, the court should have gone on to strike it down rather than send the case back.

He noted that neither the federal prison system nor "the vast majority of states" have segregation policies, and observed that such policies could hurt rather than help the racial climate in prison by "lending credence to the view that members of other races are to be feared and that racial alliances are necessary."

On the other hand, Justice Clarence Thomas, joined by Justice Antonin Scalia, filed a lengthy dissent arguing that the court should defer to the judgment of prison officials on questions of race as on other questions of prison policy.

"The Constitution has always demanded less within the prison walls," Justice Thomas said.

Justice Thomas, who dissented two years ago in the Michigan case, noted with some asperity on Wednesday that the court had deferred then to the law school's judgment on its need for affirmative action.

"Deference would seem all the more warranted in the prison context," he said, "for whatever the court knows of administering educational institutions, it knows much less about administering penal ones."

"The potential consequences of second-guessing the judgments of prison administrators," Justice Thomas added, "are also much more severe."

In his 28-page dissenting opinion, nearly twice as long as the majority's 15 pages, Justice Thomas said the gang problem in California's prisons was worse than elsewhere in the country. The argument that interracial housing would lead to less racial hostility among inmates is "implausible," he said.

The California policy, which is more than 25 years old, requires that new or newly transferred inmates be placed in two-person cells, with a cellmate of the same racial and ethnic background (Japanese with Japanese, for example) during an evaluation period of 60 days. Permanent places are then assigned on a nonracial basis.

The policy was challenged by a black inmate, Garrison S. Johnson, who has been serving a sentence for murder since 1987 and has been transferred five times, each transfer followed by a period of segregation. He filed his own lawsuit, claiming racial discrimination, in 1995 and was eventually appointed a lawyer.

The Bush administration entered the case, Johnson v. California, No. 03-636, to oppose the California policy and, at the same time, emphasize its view that all government classifications by race, for whatever purpose, are subject to strict scrutiny. The multiplicity and the nuance of the views the justices expressed on Wednesday underscored the fragile compromises on which this area of the law has been constructed.