Why is the time difference between flushes so big?

[Yujie-Cui]

Thanks a lot for your source code. When I run histogram/ff/calibration, I find that the time difference is more than 100. However, the time difference stated in the original paper is only about 10 cycles.

[Yujie-Cui]

Oh my god. I found that flushing data that is not in the cache takes 100 more cycles than flushing data that is in the cache.
I think this should be unreasonable, but when I run decrypt AES, it gives correct result (when flush time delay is less than threshold, think in cache). Why is this?

Below is my key code for making the measurements.

int flush_cached(size_t* addr)
{
//   size_t time = rdtsc();
//   flush(addr);
//   size_t delta = rdtsc() - time;

  maccess(addr);
  maccess(addr);
  asm volatile("lfence\nmfence\n");
  int delta = time_flush(addr); 


  return delta;
}

int flush_uncached(size_t* addr)
{

  flush(addr);
  flush(addr);
  asm volatile("lfence\nmfence\n");
  int delta = time_flush(addr);
  return delta;
}

inline
int
time_flush(void *adrs) {

  uint64_t t = rdtsc();
  clflush(adrs);
  t = rdtsc() - t;
  return (int)t;

}

I use the following method to measure the half keys of AES. I found that when using delta < FF_THRESHOLD, I successfully restored half keys. But here it should be >. It's so weird.

  	    flush(probe);
  	    AES_encrypt(plaintext, ciphertext, &key_struct);
  	    sched_yield();
           size_t delta = time_flush(probe);
        // printf("delta=%ld\n",delta);
  	    sched_yield();
  	    if (delta < FF_THRESHOLD)
  	    { 
  		  	count++;
  	    }   

[dgruss]

Normally we observe differences like 3-15 cycles on different CPUs... 100 cycles is a bit extreme.... what CPU is that? Think about why that timing difference is there: What does flush have to do when the cache line is in the cache (to make it not be in the cache anymore)? What does flush have to do when the cache line is already not in the cache (to make it not be in the cache)?

[Yujie-Cui]

The model number of the processor is: Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70GHz.
Are you measuring the delay difference directly with the histogram/ff/calibration.c/calibration.c program?

When I run it directly, and output the delay corresponding to the highest frequency, it is bizarrely high (about 220 and 320 respectively).

I re-tested. On an Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz cloud server, run histogram/ff/calibration.c/calibration.c. And the threshold with the highest output frequency is:

Flush uncached  :    234
Flush   cached  :    366

On an Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70GHz cloud server, run histogram/ff/calibration.c/calibration.c. And the threshold with the highest output frequency is:

Flush uncached  :    156
Flush   cached  :    272

This is a time difference of 120 cycles, provided I haven't modified any code.
Flush the data stored in the cache with a larger time difference.

If measured by the function described in my last comment, it would be:

Flush uncached  :    304
Flush   cached  :    188