Performance Comparison of Wikibench on ‘m3’ Instances

application_benchmarking_of_m3_instances

In our last post about wikibench benchmarking, we discussed the benchmarking methodology and gave a detailed analysis of the results for the m3.xlarge instance. In this post, we will compare the wikibench performance on all ‘m3’ instance types.

We ran experiments on all the ‘m3’ machines using the same methodology as in the m3.xlarge instance. We compared the performance in each of the following categories:

  • Missed Deadlines

  • Request Timeouts

  • Average Response Times

  • CPU Usage

  • Memory Usage

Missed Deadlines

Due to the delayed response time from the server for lower sampling rates, we see that wikibench is not able to replay all the requests in the tracefile. This results in missed deadlines. Here is a table of missed deadlines for each of the instance types.

Table 1: Number of Missed Deadlines

Sampling Factor

Total Number of Requests  

m3.medium

m3.large

m3.xlarge

m3.2xlarge

980

46239

31726

10668

952

628

981

43847

28367

5774

267

62

982

41589

25683

4524

191

35

983

39319

23366

3277

108

15

984

36901

20888

2240

74

15

985

34725

18939

1672

43

8

986

32370

16848

1069

28

4

987

30069

14681

659

19

1

988

27720

12028

391

6

3

989

25497

9966

195

3

0

990

23271

8146

98

0

0

991

20767

5463

28

1

0

992

18340

3408

4

0

0

993

15862

1668

0

0

0

994

13520

670

0

0

0

995

11326

230

0

0

0

996

9053

46

0

0

0

997

6797

2

0

0

0

998

4505

1

0

0

0

999

2193

0

0

0

0


Request Timeouts

We configured the benchmark to have request timeouts at 10000 milliseconds. When the requests rates are high, we did see quite a few request timeouts. Here is a table showing the number of requests which timed out.

Table 2: Number of Request Timeouts

Number of Requests

m3.medium

m3.large

m3.xlarge

m3.2xlarge

46239

200

86

6

12

43847

49

0

0

0

41589

23

0

0

0

39319

14

0

0

0

36901

4

0

0

0

34725

3

0

0

0

32370

3

0

0

0

30069

1

0

0

0

27720

2

0

0

0

25497

4

0

0

0

23271

2

0

0

0

20767

0

0

0

0

18340

0

0

0

0

15862

0

0

0

0

13520

4

0

0

0

11326

3

0

0

0

9053

0

0

0

0

6797

0

0

0

0

4505

0

0

0

0

2193

0

0

0

0

Average Response Times

We calculated average response time by first calculating the averages over a period of one minute and then calculating the average of the resulting values. Here is a table of average response times.

Table 3: Number of Response Times

Number of Requests

m3.medium

m3.large

m3.xlarge

m3.2xlarge

46239

1406.770492

439.016393

132.573770

115.803279

43847

1290.200000

298.508197

102.426230

84.934426

41589

1232.573770

279.016393

100.426230

86.983607

39319

1219.163934

260.278689

96.688525

82.114754

36901

1198.737705

237.770492

93.770492

80.721311

34725

1202.016393

221.426230

92.344262

80.393443

32370

1211.672131

204.573770

90.721311

80.213115

30069

1172.409836

193.901639

89.704918

80.721311

27720

1144.098361

178.032787

88.704918

81.114754

25497

1111.622951

164.573770

87.344262

80.311475

23271

1095.754098

148.934426

86.704918

82.065574

20767

973.311475

137.508197

83.786885

79.672131

18340

875.114754

119.442623

81.360656

79.065574

15862

714.918033

109.278689

79.655738

75.081967

13520

571.233333

104.433333

79.700000

74.850000

11326

448.533333

101.166667

79.633333

76.350000

9053

338.083333

94.033333

78.133333

74.383333

6797

275.816667

91.650000

77.966667

75.733333

4505

239.783333

91.016667

79.983333

77.500000

2193

191.716667

84.250000

76.316667

73.283333

The chart below shows the inflection points for when the average latency starts to increase. It gives you a better idea of available throughput.

CPU Usage

From these graphs, it is quite clear that the performance of m3.xlarge is very close to that of m3.2xlarge. Another interesting factor we noticed was in CPU usage.

Here are the details of typical CPU usage during the experiment. We ran the experiments starting from a sampling factor of 980 (~46K requests per hour). We gradually increased the sampling rate to 999 (~2k requests). The trace file is designed to run for 60 minutes for each sampling factor. In total, there were 20 samples which took approximately 20 hours to run.

Table 4: CPU Usage

 

m3.medium

m3.large

m3.xlarge

m3.2xlarge

idle

20.211905

51.849000

78.020200

89.794100

interrupt

0.000053

0.000080

0.000103

0.000054

nice

0.000000

0.008555

0.010855

0.009163

softirq

0.068475

0.079080

0.076420

0.014414

steal

40.155423

0.013410

0.021688

0.014548

system

18.229266

14.334300

6.868448

3.159783

user

38.685834

32.652000

14.585000

6.554873

wait

0.142068

0.944665

0.294283

0.325501


As you can see from the CPU usage graph, there is significant steal time in m3.medium. This effectively reduces the CPU-‘user’ time, and it’s a major reason for its poor performance. For the other instance types, we do not see CPU ‘steal’.

Memory Usage

The following table provides the details for memory usage.

Table 5: Memory Usage

(units in MB)

buffered

cached

free

used

m3.medium

127.5357291

899.240325

2106.372271

617.2891745

m3.large

141.5943501

1138.495956

5546.004756

624.545563

m3.xlarge

139.3173739

989.264207

13165.54742

685.9725626

m3.2xlarge

143.2133517

1187.562002

27862.95623

845.2801315


As you can see, the application isn’t bound by memory consumption.

Conclusion

These experiments give you an idea of the number of requests per minute the various m3 instances can handle. It is important to keep in mind that performance varies significantly by use case.

It’s also interesting to note that none of the resources that we traditionally treat as the bottlenecks for throughput are obvious bottlenecks here. This is because the complete system has many interactions.

View all Benchmarking Posts here

August 26, 2014 / Benchmarking

About the Author

Flux7 Labs
Find me on:

Join Us

Join thousands of technology enthusiasts, subscribe and get expert perspective in your inbox.

Connect With Us