Selenium grid hardware and system requirements. Optimizing for just one GRID or many GRIDs in paralell

By neokrates, written on June 22, 2010

article

  • Join date: 11-30-99
  • Posts: 224
View Counter:
  • 3,652 views
Rate it
Ad
Poll
  • Which Selenium configurations do you use to test your web front end?

    View Results

    Loading ... Loading ...
Feeds:
  • bodytext bodytext bodytext

Selenium is very sensitive to system configuration. Particularly memory requirements are huge. Of course, less remote controls means lesser memory drain. But that will slow down test suites. Here we explore how to configure optimal Selenium Grid system. We start with one particular example.

Software:

✔ Firefox 3.6.5

✔ Debian GNU/Linux 5.0.3

Grid:

  • Selenium grid remote control standalone – 1.0.4
  • Selenium server – 1.0.1
  • Selenium grid hub standalone – 1.0.4
  • Selenium grid tools standalone 1.0.4

Hardware:

✔ 4x Dual-Core AMD Opteron

✔ Total Mem 4048 Mb

General

In our firm, we currently use 20 Selenium RC’s per test suite. We also have particular hardware configuration with Debian Lenny. Given our current hardware we must find out how much parallel builds can be run on it. Also we want to understand the limitations and what to do if we need more GRID builds.

❓ Specific question: How much selenium grids can be run in parallel on a given server without slowing any single grid too much or even causing test failure?

In this article I would describe how to find the correct configuration for the Selinum Grid’s you need to run.

Preconditions

It is often a problem that many test suites use the same users, sessions or some other resource. On the same server, port conflict must be solved before two hubs can run. Any conflicting resource will result in concurrency problem. Precondition is, that no concurrency problem can take place, all resources can be used in parallel (no conflict on "Selenium level").

1

Validating the server: method and criteria

How to measure that system is Ok for grid builds?

We will use free basic criteria:

1. Memory usage

This result is extracted using the output of

free -o -m -s 10|grep Mem

during the parallel run.

2. Build speed

How slow will the test suite be when two, three or many selenium grids run in parallel. We use Hudson to run the builds, so we just see how much particular Hudson build took.

3. Errors during build

An out of memory can occur. That means, hardware is not sufficient or settings should be tuned. I prefer to say “hardware not sufficient” because limiting the resource usage (maybe JVM memory etc…) normally results in slower selenium builds (selenium is very greedy and it needs all it requests to run fast).

We will use:

less /var/log/syslog

to watch out for system problems

and

build log to see what selenium says.

Other problems, which are not caused by configuration or hardware limitations

There are also ‘stability problems’ which can’t be considered during general system optimization for selenium tests. One particular type is the browser stability. It basically has nothing to do with selenium. The browser stability depends on combination of OS, browser type and version, plugins and page you browse.

Configuration 1. Running one Selenium Grid – OK

One of the problems with selenium Grid seems to be the large span between Min and Max memory usage. It is most likely the consequence of one single fact:

‘Resource usage = Number RC * Browser Requirements * Page Content Requirements’

depending on what you are testing and with which browser, you can get very different results. In our case we have highly optimized pages with lots of graphic. Some pages have flash.

Thats what we got:

Free memory:

⭐ before the run: 3610 Mb

⭐ minimum during run: 1050 Mb

⭐ maximum during run: 2605 Mb

Build speed:

⭐ 100% (1 build, reference speed)

Fatal errors

⭐ No errors in build log

Conclusion: The system with given configuration is more then enough to run one Selenium Grid.

Configuration 2. Running two Selenium Grids in parallel – OK

Already in the case of single grid, there is a large span between Min and Max memory usage. For two agents it may be twice as big.

In our particular case:

Free memory:

⭐ before the run: 3161 Mb

⭐ minimum during run: 115 Mb

⭐ maximum during run: 730 Mb

Build speed:

⭐ 95%-100% !! (Might be because we use quad-core system, one core pro build… )

Fatal errors

⭐ No errors in build log

💡 In rare extreme cases /var/log/syslog would show that one of build was taken out. Once again (your case may be different) depends on what your tests are made of.

Conclusion: Selenium Grid with 20 RC’s on 4 Gb can support a maximum of 2 parallel executions. The fact that it has 4 Processor cores seems to result in near-to-none build speed loss.

Configuration 3. Running three and more Selenium Grids in parallel – PROBLEM

With 4 Gb memory successful run of more then two parallel builds becomes a question of pure luck. There is any second a chance that one of the builds eats the remains of the memory and Linux core shuts it down. (See OOM killer, etc…)

So, the problem is that selenium build becomes kind of unpredictable. Linux core doesn’t kill the process completely and that results in wide messages. For example, we often got:

FINE: sun.net.www.MessageHeader@1ff177c7 pairs: {POST/selenium-server/driver/ HTTP/1.1: null}{Content-Type:application/x-www-form-urlencoded; charset=utf-8}{User-Agent:
Java/1.6.0_14}{Host: localhost:48141}{Accept: text/html, image/gif,image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}{Content-Length: 59}
WARN Current location couldn't be retrieved by getLocation():com.thoughtworks.selenium.SeleniumException:  ERROR Server Exception:sessionId afe3ab19a90143e2aecb1575996316ff doesn't exist; perhaps this session was already stopped?
WARN There is no valid image to be saved.
WARN There is no valid screenshot to be thumbnailed.
WARN There is no valid source to be saved.

 
We found out what is the cause by watching the moment in which system took out a part of selenium. (The second cause was that browser was just unstable)

Conclusion: Three builds will not run good (memory limit).

2

Preconditions for more parallel Selenium grid builds

Memory

Without enough memory it will simply fail with different error messages or slow down every system precess causing multiple build malfunctions.

In my experience the minimal step is 1.5 Gb pro additional Grid:

Number of parallel Grids Memory (Gb)
1 2.5
2 4
3 5.5
4 6

💡 If you are running rather large test suite, I would recommend 4 Gb step. Then:
 

Number of parallel Grids Memory (Gb)
1 4
2 8
3 12
4 16

 

Number of CPU’s

Optimally, 1 CPU pro build.
If you have less, there are two risk factors:

➖ Build speed will get a major hit.

➖ Builds will be have differently because of too slow system response (have you tried to open 50 tabs in Firefox!? The system will react like in “time loop” .
You can see that thats the case if you just login per ssh and try some basic commands. In that state, system can’t handle the load.

Other system resources

Might influence build speed but are not that critical.

Conclusion: 1 Processor pro Grid and 1.5 Gb – 4 Gb of memory are needed for one additional Selenium Grid.

3

Additional system configuration considerations

Virtualization

Can work, provided you can accept 10-20% slower builds. If all the same resources are provided, virtual system (I used VM-Ware) would run slower. All else is the same. Virtual system tends to behave differently under heavy load. We had multiple process kills by the linux under VMWare and decided not to use it for any time-critical tests.

32 bit or 64 bit

Both works, but I guess to use more memory and get some more speed (64bit) is better. Just make sure all packages your use can 64 bit. With 32 bit system it is also possible to use 4 gb and more. There is a package under linux which “enables” that memory for programs.

Linux Swap

Might be a bad idea. Yes, you can make sure that no system kill occur because swap is used. But the time loss may be very large, essentially system might freeze if it tries to run selenium with swap.

👉 Conclusion

Properly configured server can run 4-8 selenium grid builds without slowing them down too much. Hence, one or two good servers can provide test coverage for pretty big software project.

Thats it, have fun :)

Be Sociable, Share!
 

LEARN MORE (amazon bookstore)

TAGS

SOCIAL
Be Sociable, Share!


Leave a Reply