I'm far from being knowledgeable about this but I have had the opportunity of serving as a "consumer reviewer" on the Department of Defense Prostate Cancer Research Program and got to see some of the innards of the research process. We have people in our group who have, themselves, done medical research and can probably give corrections or additional information. However we're all interested in new treatments so here are my impressions of the process for developing them.
To begin with, some very intelligent and hard working researcher, she might even be a novice in her first research job, gets a new idea about some disease process. It might have to do with how some particular chemical plays a vital role in a particular "pathway" (chain of biochemical reactions) in cancer cells. She has an idea for how to interfere with it by introducing some other chemical into the pathway.
The first thing she has to do is convince someone that this is promising enough to spend some scarce research funds on it. She's paid to work on other projects that are currently funded and she's got to get time and funding from someone to enable her to do the experiments that are needed. She also needs advice and criticism from colleagues who can possibly spot holes in her ideas or offer suggestions for improvement.
Time passes.
When she's got the money she assembles her tissue cultures, specially designed mice, chemicals, lab equipment, and whatever else she needs - a process that also takes some time. Then she begins her experiments. If mice are involved then one to two years are likely to pass before all the results are in. If it's just tissue cultures, the work goes faster but then the mice still have to be tried before humans can be.
Time passes.
The experiments turn out to be more or less successful. Now it is necessary to figure out what comes next. It may require more analysis of the existing data or more mouse studies to be sure that the results really are what they appear to be. Money has to be procured to continue the process.
Time passes.
Everything looks promising. It's time to design a clinical trial. A careful design has to be done including figuring out how to test, what end points will be measured, what side effects may occur and how they can be mitigated, how many subjects are needed for statistical significance, how patients will be recruited, and what institutions can be recruited to participate. This has to be done right. Lives are at stake. A budget has to be calculated and it has to be as tight as possible in order to get funding. A carefully written and convincing proposal has to be written and sent out, with variations for the funding source requirements, to each of the usual funding sources and put in line for them to consider. The funding sources typically get a new budget each year and go through one grand process per year to decide on all of the projects they will fund. They can't just fund something that comes in, no matter how good it looks. It has to be a fair process in which all researchers and proposals get a fair chance to put their proposals in the pot and no one gets any money until all proposals have been considered. And by the way, this was probably done for the mouse studies too.
At some point in this process, the drug companies start to get interested and put their fingers and dollars into the pie - with time spent by the researchers to interact with them.
Time passes.
The trial is approved. The process begins for each participating site to get the materials and training they need to handle the new treatment. Patients have to be recruited, a process that can go on for a year or multiple years. The patients are treated and the results start coming in, a process that requires months to years, especially if long term effects are what we're looking for.
Time passes.
All the results are in. The principal investigator and her assistants write a report of what they have discovered. The write up has to be done right. No one wants to submit a paper with a stupid mistake in it for the entire world to see. It goes through levels of review by local colleagues, other colleagues, and finally peer reviewers for the journals. If a journal accepts it the article goes somewhere into the queue for publication. Reports are also filed with the Food and Drug Administration. The FDA has its own reviewers who look at the reports and render judgments about whether a new drug should be approved, or perhaps more questions asked or more research done.
Time passes.
Then VOILA! a new drug becomes available.
To patients, it seems like a ridiculously long time has passed. Many years went by. But no one has found a better way to do it. Most of the decisions to approve a new drug result in actually helping people - something that would not be true if the process were shortened too much.
The process costs more and more at each stage of the research. Skipping a step or abbreviating it too much is likely, not to save money and lives, but to cost them. Bringing bad drugs into clinical trials wastes tons of money that could have funded better projects, and may result in patients needlessly suffering or dying.
It's a hard slog. My hat is off to the incredible people who make it happen.
Alan