Why Does Removal Of Boolean Test Result In Different Answers

I'm trying to find all of the prime factors for a given number. The following code achieves this:

private static ArrayList<Long> findPrimesOf(long number) {
ArrayList<Long> primes = new ArrayList<>();
long highestDivisor = (long)sqrt(number);
for (long divisor = 2; divisor <= highestDivisor; divisor++) {
if (number % divisor == 0) {
boolean isPrime = true;
for (long i = 2L; i < divisor; i++) {
if (divisor % i == 0) {
isPrime = false;
break;

[code]....

However, if I remove the boolean test isPrime, as below, I get additional numbers after the highest prime factor.

private static ArrayList<Long> findPrimesOf(long number) {
ArrayList<Long> primes = new ArrayList<>();
long highestDivisor = (long)sqrt(number);
for (long divisor = 2; divisor <= highestDivisor; divisor++) {
if (number % divisor == 0) {
for (long i = 2L; i < divisor; i++) {
if (divisor % i == 0) {
break;

[code]....

I can't work out why this should be. From my understanding, the divisor only gets added to the list of primes if the loop doesn't break (which, with the boolean test would be true anyway).