Setting The Legend Color On A Stacked Bar Chart

I am trying to develop a report that will show a 100% stacked bar graph for many teachers. Each teacher will have there own graph. The series in the report are the percent of scores given on a test, with each test in a group. Scores are given on a scale of 1-4. Most of the time a teacher will have all fours scores in there data; however it is possible to only have 2 or 3 of the scores. How can I set the legend up so that all scores across all the graphs show the same color for the same data series? Right now I get something like this.
First Graph
4: Lime
3: Purple
2: Blue
1: Green
Second Graph (no values for 1 or 2)
4: Blue
3: Green

I need the second graph to be
4: Lime
3: Purple