Lobby group Fairness in Religions in Schools said at least 55 schools had axed the program this year following changes introduced by the Andrews government, which they support.
The state's largest provider of special religious instruction, Access Ministries, said they expected the figure would rise.
In August, the Andrews government announced that special religious instruction would be axed from the Victorian curriculum and replaced with respectful relationship, global cultures, ethics and faith education.
The 30-minute religion classes can now only be delivered during lunchtime or before or after school.
Access Ministries spokesman Rob Ward said "bureaucratic obstacles" had made it challenging for some schools to continue offering the program.
Teachers must give up their lunch break, start work early, or stay back at school to supervise the classes, he said.
"We are very disappointed that the government made this decision. We don't think it is in the best interests of children," Mr Ward said.
He said Access Ministries had made its program more interactive to attract students during lunchtime.
Fairness in Religions spokeswoman Lara Wood said schools had ditched the program because it was unpopular and had created a lot of extra work for teachers.
She said religious instruction had no place in secular state schools. "If parents want this program they can easily send their children to church for Sunday school."
Kew MP Tim Smith raised concerns about the Andrews government's changes to the program at a public accounts and estimates hearing on Friday.