Swing/AWT/SWT :: Multiple Action Listeners?

So im making a calculator just basic 4 function with a clear button. as i am working into this project i am running into an issue of assigning which click sets what. For instance i click the two button how do i tell it to assign it to firstNum, versus secondNum?

private class theHandler implements ActionListener{
public void actionPerformed(ActionEvent event){
if(event.getSource()==one){
firstNum=1;
}
if(event.getSource()==two){
secondNum =2;
}
if(event.getSource()==three){
secondNum =3;
}

So this for instance allows me to set one to the firstNum and then add it to two or three but im not sure what to do next. I have in my mind i want to do something like make three listener classes, that operate in sequential order like you pick the first numbers, click an operation, and then select the second numbers then hit the operation(equal) button to display answer. Am i on the right track or what?