Tuning FotoWeb and Index Manager for performance
What's in this document?
According to surveys by Akamai and Gomez.com (http://blog.kissmetrics.com/loading-time) nearly half of web users expect a site to load in two seconds or less, and they tend to abandon a site that isn't loaded within three seconds. Google has also done extensive testing on user behavior relative to page load times that show how easily users navigate away if a site becomes less responsive.
What this means in a FotoWare DAM system is that system administrators must be wary of hardware requirements, configuration, regular maintenance tasks and absolute do's and don'ts in terms of how the system is configured. This document aims to give some pointers on these matters.
What does a FotoWare system require, hardware-wise?
Different types of applications have different requirements: Some are CPU-intensive, some are memory-intensive and others are highly dependent on I/O.
A FotoWare system requires no less than all three. The table below explains why.
| Resource | Why? |
|---|---|
| CPU | Calculations, especially rendition and transcoding tasks |
| Memory |
Image and video subsampling (large files, large calculations). Caching of immediate objects Keeping database indexes readily available |
| IO |
Network bandwidth - Speed of delivery to clients and number of concurrent users Disk bandwidth - time to load and serve data (renditions, metadata etc) and speed of database |
Reducing latency
Latency is the time interval that passes from a user performs an operation until he sees the response of that action. It is not to be confused with interface sluggishness, although such sluggishness can be a symptom of high network latency. Bandwidth and latency are not always connected; a high-bandwidth network may be able to push enormous amounts of data quickly, but with considerable latency a system built on such a network can still be perceived as slow by users for its sheer response time.
Thus, here are some key points for reducing latency.
- Faster networks
- Is fiber an option?
- In extreme cases, shorter network cables?
- Reduction of middleware
- Can firewalls, proxies, routers and/or switches be bypassed? Such components will inevitably induce a level of network latency. With only a few connector points, this may not be perceivable, but in large network environments, they can have a very notable effect on performance.
- Faster CPUs
- SSD Drives
- Prefer unencrypted over encrypted protocols
A real-life FotoWeb performance example
Disclaimer: The test results below are meant as an example only, and reflects the performance of this specific system. Actual performance will rely on several factors, including but not limited to the system's hardware and software configuration.
FotoWare has made some real-life performance testing using the Apache benchmarking tool to measure the delivery speed of uncached and cached content from FotoWeb.
TEST IMAGE
File size: 7552 Kb
Dimensions: 5184 x 3456 px (18 Megapixels)
Resolution: 240.00 ppi
Benchmarking hardware: Lenovo T430 portable - Intel i7 2.9 GHz, 16GB memory, Dual SSD
Uncached loading results
The below screenshot shows the time it takes to load the uncached image (emphasized in red; click for larger version).
How does this scale?
Browsers typically open 8 connections to load content, and users may have to wait when content is uncached.
| 1 Core | 4 Cores | 8 Cores | 16 Cores | |
|---|---|---|---|---|
| 1 User | 0.8 | 0.2 | 0.1 | 0.1 |
| 10 Users | 8 | 2 | 0.5 | 0.25 |
| 100 Users | 80 | 40 | 20 | 10 |
| 1000 Users | 800 | 400 | 200 | 100 |
Time is in seconds
Results when content is already cached
These are the results of cached content delivery using the same Apache benchmarking utility to deliver a single file. (Click to view larger version)
How does that scale?
| 1 Core | 4 Cores | 8 Cores | 16 Cores | |
|---|---|---|---|---|
| 1 User | 0.024 | 0.006 | 0.003 | 0.0015 |
| 10 Users | 0.24 | 0.06 | 0.03 | 0.015 |
| 100 Users | 2.4 | 0.6 | 0.3 | 0.15 |
| 1000 Users | 24 | 6 | 3 | 1.5 |
Time is in seconds
What to do if there is lots of uncached content in the archives?
In certain environments, such as where large image feeds are received, there may be large portions of uncached content at any one time. To compensate for such content, the solution is to add more hardware:
- Faster disks
- Faster CPUs
- More CPUs
Another option is to add more Index Manager servers to help the system more easily keep up with the incoming flow of files.
The FotoWare 8.0 architecture - do's and don'ts
The below chart show the system architecture in FotoWare version 8.0. Click for a larger version.
The key points to note here is that FotoWeb needs low-latency access to both MongoDB and Index Manager. They should be on the same LAN and not be constrained by any firewalls. Often a firewall can slow performance considerably and performance can vary a lot depending on a number of factors - at time it may work just fine, at other times performance may be sluggish. Because troubleshooting such issues can be a challenge, we recommend running these components on the same LAN and making sure there's minimal latency between components.
Strongly discouraged configurations
The below chart shows a system where a firewall filters communications between FotoWeb/MongoDB and Index Manager and the storage system on the other. Index Manager relies on being able to push metadata to MongoDB (the FotoWeb metadata database) and introcuing latency at that level can have a negative effect on performance. Also the client communication from FotoWeb, which runs on port 7000/7001 by default, will also have to be funneled through the firewall, which can have a negative effect on how users perceive client performance.
Finally, the last chart shows a configuration where a firewall has been placed between FotoWeb and the MongoDB instance. This configuration should be avoided at all costs, since it adds latency to so many of the core functions in FotoWeb that it is positively guaranteed to generate performance issues.
Recommendations
Ideally, any firewall on the system will limit incoming traffic from the Internet to the FotoWeb server, for instance by allowing only ports 80 and/or 443. Performance-wise, it is best to place the entire system - FotoWeb, Index Manager, and file storage, in the DMZ zone to allow the servers to communicate freely without firewall restrictions.
Configuring Index Manager for use with FotoWeb
Setting a search threshold
When configuring Index Manager for use with a FotoWeb, it's good practice to reduce the maximum number of hits a search can produce by setting an abort threshold. The will cut off a search when it produces more than the preset number of hits and deliver that search result to the client. This is a good, practical measure to prevent bogging down the server with heavy queries that will produce more hits than a user can anyhow evaluate. Using search thresholds has some implications that you can read more about by following this link.
Remove users' possibility to search for *searchword
To reduce the possibility that users execute queries that generate a lot of server processing and/or extremely large results, it is possible to remove their possibility to use wildcards at the beginning of the search string. To do so, tick Disable leading search characters in the index configuration.
Configuring FotoWeb for optimal searching
Reduce the maximum number of hits
A system administrator can configure FotoWeb to only allow searches to return a certain number of hits. This prevents users from performing very wide queries that take a lot of time to process and also reduces the server traffic that's generated when users scroll through their results in the thumbnail grid.
Also, its good practice to set as low a search timeout as possible. That way, if Index Manager fails to deliver a search result within the set time frame, you know that the performance problem is Index Manager related.
Note that if the search timeout period is too long, the server's capacity to serve results to a large number of concurrent users will be reduced.
Example: Index Manager performance benchmarking
| SATA Instance | SSD instance | |
|---|---|---|
| CPU | 8 vCPUs, AMD Opteron 2,1 GHz | 8 vCPUs, AMD Opteron 2,1 GHz |
| RAM | 30GB | 30GB |
| Disks | RAID 10-protected SATA | RAID 10-protected SSD |
| Index size | 100,475 JPEGs | 100,475 JPEGs |
Indexing speed
This table shows the time it took to index the roughly 100,000 files on the two systems. Cache warming is the process of creating cached lowres thumbnail representations of indexed assets that can be delivered to clients.
| SATA Instance | SSD instance | SSD Gain | |
|---|---|---|---|
| Indexing | 18m 50s | 11m 11s | -38% |
| Cache warming | 13m 20s | 5m 31s | -58% |
As the table shows, reading metadata (the indexing process) requires much IO, and generating thumbnails during cache warming is even more IO-intensive.
Query speed
For these tests the Index Manager cache was disabled to show actual disk IO performance measures.
| SATA Instance | SSD instance | SSD Gain | |
|---|---|---|---|
| Single result | 10 ms | 5 ms | -50% |
| ~4k results | 100 ms | 50 ms | -50% |
| ~40k results | 750 ms | 450 ms | -40% |
| ~100k results | 1800 ms | 1200 ms | -22% |
As the table shows, searching is also IO intensive. The servers were equally equipped apart from the storage system. From a user's perspective anything below 100 ms will be perceived as instant. It should be noted that the times reported here are the search times reported by Index Manager, so any network latency will affect the time it takes before the resulting thumbnails actually show up on the client's screen.
Naturally, the size of the result set alo affects the performance of the query, so educating users on how to create meaningful queries is important. Using taxonomies to assist users in filtering the archive content is also valuable in this regard.
Tools for analyzing Index Manager performance
Index Manager 8.0 has tools that make it possible to measure its performance using Windows utilities. Learn more about using Index Manager performance counters by following this link.