FILE: Aug. 9, 2014: President Obama speaks at the White House about the ongoing situation in Iraq.AP

Senate Democrats at risk of losing re-election in moderate or conservative-leaning states this fall are wary about President Obama using his executive powers to sidestep Congress and act alone on immigration reform.

Their biggest concerns appear to be that Obama will expand his 2012 executive action that deferred deportation for young illegal immigrants by extending it to as many as 5 million more people living illegally in the United States.

“This is an issue that I believe should be addressed legislatively and not through executive order,” said North Carolina Sen. Kay Hagan, one of the most vulnerable Democrats in Republicans’ bid to take control of the Senate.

The president is scheduled to return from a family vacation at some point this weekend to engage in high-level White House talks. Administration officials will not disclose what will be discussed but have said the president does not intend to take action at least on the current immigration crisis — the influx of unaccompanied Central American youths crossing the southern U.S. border — until after summer.

Sen. Mark Pryor, D-Ark., another vulnerable incumbent, said in a statement that he also is “frustrated with the partisanship in Washington. But that doesn’t give the president carte blanche authority to sidestep Congress when he doesn’t get his way.”

Such statements have immigration advocates on edge.

A coalition of advocacy groups, in a letter to congressional Democrats on Friday, said immigrant families should not have to wait until after the November elections for relief. The organizations said any attempts by Democrats to delay or dilute administrative changes “will be viewed as a betrayal of Latino and immigrant communities with serious and lasting consequences.”

The letter was released because of advocates’ concerns that leading Senate Democrats may be shifting their positions because of political considerations after previously urging Obama to act.

A spokesman for Sen. Chuck Schumer, D-N.Y., declined to say Friday whether Schumer still believes Obama should act by October, as Schumer had said before. A spokesman for Sen. Dick Durbin of Illinois, the second-ranking Democrat, said the timing of executive action on immigration was up to Obama. Senate Majority Leader Harry Reid’s spokesman had no comment on timing.

Still, Obama looks determined to move forward on his own despite the political risks for Democrats.

He is irritated by House Republicans’ inaction on immigration legislation passed last year by the Senate. The crisis over unaccompanied minors arriving in South Texas does not appear to have deterred him, and the slowdown of arrivals at the border may be shifting the issue away from the spotlight anyway.

The exact contours of Obama’s plans remain unclear.

Advocates and lawmakers who have talked with administration officials anticipate that he could expand a program that granted work permits and deferred deportation to more than 700,000 immigrants brought illegally to the U.S. as kids. It could be extended to include parents of those children, as well as parents of U.S. citizens, or potentially others — groups that could add up to perhaps 5 million people.

During a news conference this month, Obama was not specific on his immigration plans. He did say that in the absence of congressional action and in order to address the crisis involving unaccompanied youths, he had to shift resources on his own and exercise prosecutorial discretion.

“I promise you the American people don’t want me just standing around twiddling my thumbs and waiting for Congress to get something done,” Obama said.

Some GOP leaders worry that opposition to a comprehensive overhaul will harm their party in the 2016 presidential race, where Latino turnout is higher than in midterm elections. Hispanics are a fast-growing sector of the presidential electorate and backed Obama overwhelmingly in 2012.

   But Republicans also see a nearer-term chance to translate Obama’s potential executive actions into electoral success in November. Republicans need to win a net of six seats in order to take control of the Senate for the remainder of Obama’s term. The GOP already is all but assured of maintaining control of the House.

As Republicans meet with voters in their districts during the summer break, lawmakers have raised alarms about the scope of Obama’s potential plans. In some cases, they are hearing clamors for impeachment in return.

“It is up to Congress to actually go back and restrain this guy,” one voter told GOP Rep. Andy Harris of Maryland at a town hall meeting on the Eastern Shore. Harris had warned that Obama could expand an existing deportation relief program to 4 million or 5 million more people, “competing with Americans for work.”

Republicans have tagged Obama as an “imperial president” who goes around Congress rather than working with lawmakers, and House Republicans have moved to sue him over it. The prospect of the president making a unilateral move on a contentious issue such as immigration has Republican consultants salivating.

“President Obama’s executive amnesty would inject adrenaline into an electorate already eager to send him a message of disapproval,” said Brad Dayspring, communications director for the National Republican Senatorial Committee.

Also problematic for Obama: His apparent plans to act on his own authority come after years of saying that he did not have the legal justification to proceed without Congress.

“If, in fact, I could solve all these problems without passing them through Congress, I would do so. But we’re also a nation of laws,” he said in November. A heckler had interrupted a speech he was giving in San Francisco, prodding him to halt deportations, which have reached record highs on Obama’s watch.

Since then the White House has apparently concluded otherwise.

Democratic pollsters argue that any executive action by Obama could give a political boost to Democrats, not just from newly energized Latino voters but from an electorate at large that would welcome any action from gridlocked Washington.

“Voters are so sick of the do-nothing Congress they don’t mind if there’s an imperial president,” said Democratic pollster Celinda Lake. “They would just like someone to get something done about something.”

The Associated Press contributed to this story.