Setup

Installation and configuration are necessary only if you intend to run VisiOmatic locally or as a web server. If you are only interested in using the default web client as a remote user, you may head directly to the web interface section.

Installation

The easiest way to install VisiOmatic is from the shell, through pip:

$ pip install visiomatic

If the command above does not work and your Operating System (OS) still comes with Python 2, try pip3 instead of pip.

On recent systems, the vanilla pip install command may trigger an "externally managed environment" error, due to a change in policy to avoid conflicts between the Python package manager and that of your own operating system. In this case, you should first set a virtual environment to run VisiOmatic, or even better, use the pipx package that will automatically do it for you:

$ pipx install visiomatic

Checking installation

Once installation is complete, you may do a simple check of the server component by starting VisiOmatic with the default settings:

$ visiomatic

and pointing your browser to http://localhost:8009/api. You should land on the page shown below.

If you have a FITS image in hand (say, image.fits), you can quickly check that VisiOmatic works on your machine by typing in a shell window

$ visiomatic image.fits

A browser window should pop up, and after some time (used for caching the data), the image should appear in the web interface. Type ^C in the shell window to quit.

Configuration

VisiOmatic can operate in a wide variety of environments and is highly configurable.

The command

$ visiomatic --help

returns a list of all the available options and their default values. There are three successive levels at which the settings are applied (and overridden) :

1. Built-in default configuration.

2. The configuration file.

3. Command line options.

Default configuration

The default configuration is defined within the code itself. The default values are those returned by the visiomatic --help command. They are generally appropriate for local use on a "regular" machine.

The configuration file

A configuration file may be used to store settings that will be applied anytime VisiOmatic is started. The default location of the configuration file depends on the platform and is defined by the platformdirs library: ~/.config/visiomatic/visiomatic.conf (Linux), ~/Library/Application Support/visiomatic/visiomatic.conf (macOS), or C:\Documents and Settings\<User>\Application Data\Local Settings\visiomatic\visiomatic.conf (Windows). The path to the configuration file may be changed using the -c / --config command line argument, for example:

$ visiomatic --config /etc/visiomatic.conf

Editing an existing configuration file is easier than writing one from scratch! The following command may be used to write (or replace) a default configuration file at the default location:

$ visiomatic -s

Command line options

All configuration file settings may also be changed using arguments to the visiomatic command; however the command line syntax differs slightly from that of the configuration file. For instance, to set the port number to 8010 from the config file one would use the line port = 8010, whereas from the command line one could type

$ visiomatic --port=8010

or

$ visiomatic --port 8010

or even

$ visiomatic -p 8010

and boolean command line options do not require to be followed by the True value. For example, the following command runs VisiOmatic with the cache cleared at startup:

$ visiomatic --clear_cache

The same option may be abbreviated

$ visiomatic -C

Configuration settings

Configuration settings are grouped into five sections: host, image, server, engine, and cache. The full list of options is listed below; jump to the setup guide for use cases.

Host options

The Host options allow you to specify how and where the Visiomatic service runs.

host host

This option sets the host name or IP address where the service will run. If not specified, it defaults to localhost.

port integer

Define the port number for the service (<65536). The default port number is 8009. Note that without administrator priviledges, only values equal to or above 1024 are allowed.

root_path path

Configure the ASGI root path, which is an advanced setting for specific deployment environments. The default value is the empty string.

access_log

Use this option to display the access log, which can be useful for monitoring requests to the server.

reload

This option enables auto-reloading of the server when code changes are detected, which can be convenient for development purposes. Note that enabling this option turns off multiple workers.

workers integer

Specify the number of worker processes to handle requests. By default, this is set to 4.

Image options

Image options control the default settings for image processing and presentation.

brightness float

Set the default brightness for images (between -100.0 and +100.0). The default value is 0.0, indicating no change from the original brightness.

contrast float

Adjust the default contrast of images (between 0.01 and 10.0). The default value is 1.0, meaning no change from the original contrast.

color_saturation float

Define the default color saturation level (between 0.0 and 5.0). The default is 1.5, which increases the saturation.

gamma float

Set the default gamma correction value (between 0.1 and 5.0). The default is 2.2, which is appropriate for sRGB displays.

quality integer

Determine the default compression quality for images, expressed as a percentage. The default value is 97%, balancing quality and bandwidth requirements.

tile_size shape

Specify the shape of image tiles used for processing, in pixels (between 1 and 4096 per axis). Note that the vertical size comes first, following the Python convention for image arrays. The default tile size is 256,256.

Server options

Server options configure various aspects of the web server and its endpoints.

api_path path

Set the endpoint URL for the web service API. The default path is /api.

banner_template filename

Specify the HTML template file for the service banner. The default template is banner.html.

base_template filename

Define the HTML template file for the web client interface. The default is base.html.

client_dir directory

Point to the directory containing the web client code, including styles and media files. By default, this is set to <install_dir>/client.

data_dir directory

Set the root directory for data storage. The default is the current directory (.).

doc_dir directory

Specify the directory where HTML documentation is built and stored. The default is <install_dir>/doc/build/html.

doc_path path

Set the endpoint URL for accessing HTML documentation. The default path is /manual.

extra_dir directory

Provide an additional directory for storing extra data. The default is the current directory (.).

max_region_tile_count integer

Set the maximum number of image tiles per snapshot. The default is 1024.

no_browser

Use this option to prevent the automatic opening of a web browser when an image filename is provided.

template_dir directory

Define the directory containing HTML templates. The default is <install_dir>/templates.

userdoc_url URL

Set the URL endpoint for user documentation. The default URL is /manual/interface.html.

Engine options

Engine options configure the internal processing engine.

thread_count integer

Specify the number of threads the engine uses. The default is 10, allowing for parallel processing of 10 tasks.

Cache options

Cache options manage how image data is cached for improved performance.

cache_dir directory

Set the directory where image cache files are stored. The default depends on the platform; on Linux, it is ~/.cache/visiomatic.

clear_cache

Enable this option to clear the image cache on startup, ensuring no cached data from previous runs is used.

max_cache_image_count integer

Define the maximum number of images to store in the disk cache. The default is 100.

max_cache_tile_count integer

Set the maximum number of image tiles to keep in the memory cache. The default is 1000.

max_open_files integer:

Specify the maximum number of files that can be opened simultaneously (between 100 and 1,000,000). The default is 10000.

ultradict_cache_file filename

Define the file name for the pickled cache dictionary shared across processes. The default depends on the platform; on Linux it is /dev/shm/visiomatic_cache_dict.pkl.

Setup guide

This section will help you configure VisiOmatic for best performance in typical situations. The default settings are generally appropriate for basic, local usage, but may not be the best compromise when it comes to running VisiOmatic on a big server or on specific hardware.

Accessing VisiOmatic server from another machine

There are basically two ways to make your VisiOmatic server instance accessible from another machine.

The first is to set host to 0.0.0.0 to allow connections from all interfaces, and make sure that your machine firewall has the VisiOmatic TCP port (8009 by default) open. This can be convenient (although insecure) if you don't have administrator's rights but still want to browse your images from a different machine on a local network.

However the recommended way is to run VisiOmatic (or a containerized version of it) behind a web server acting as a reverse proxy (Fig. 1), such as nginx or Apache (httpd daemon). For information, here is a possible example of a reverse-proxy configuration for Apache running on a Linux machine with IP address myserver.com, allowing web-clients to communicate remotely and securely with a local VisiOmatic server instance:

<VirtualHost *:443>

  ServerName myserver.com
  ServerAlias anyalias.com

  # configure SSL
  SSLEngine on

  SSLProxyEngine On
  SSLProxyVerify none
  SSLProxyCheckPeerCN off
  SSLProxyCheckPeerName off
  SSLProxyCheckPeerExpire off
  ProxyPreserveHost Off

  # Use RewriteEngine to handle websocket connection upgrades
  RewriteEngine On
  RedirectMatch permanent ^/vis$ /vis/
  RewriteCond %{HTTP:Connection} Upgrade [NC]
  RewriteCond %{HTTP:Upgrade} =websocket [NC]
  RewriteRule /vis/ws/(.*) ws://127.0.0.1:8009/ws/$1 [NE,P,L]
  RewriteRule /vis/(.*) http://127.0.0.1:8009/$1 [NE,P,L]
  # Proxy to VisiOmatic
  ProxyPass /vis http://127.0.0.1:8009
  ProxyPassReverse /vis http://127.0.0.1:8009

  SSLCertificateFile /etc/letsencrypt/live/myserver.com/fullchain.pem
  SSLCertificateKeyFile /etc/letsencrypt/live/myserver.com/privkey.pem
  Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

The HTTPS certificate is obtained from the LetsEncrypt authority. In this example, the URL of the VisiOmatic client is https://myserver.com/vis/, which means that VisiOmatic 's root_path option should be set to /vis.

Tuning for performance

The VisiOmatic server component can be resource-intensive. Using the proper setup will help getting the most out of the computer serving your images.

Parallel processing

Parallel computing is one way to improve the tile-serving performance of the server. VisiOmatic uses both multiprocessing and multithreading to parallelize the processing of images.

Attention

Multiprocessing and multiple workers are currently available only on Linux, because of compatibility issues with some low-level libraries on macOS and Windows.

The multiprocessing aspect involves several worker processes which are handled the task of processing the requests and serving the images. The number of workers can be set with the workers option (4 per default on Linux). The higher the number, the more processes are available to process requests in parallel, but the higher the memory usage. In the most extreme situation (as many images being cached as workers at a given time), the total memory usage in bytes may temporarily grow as high as \(4 \times \sum_{i=1}^{i \le n_W} n_{\rm pix}^{(i)}\), where \(n_W\) is the number of workers and \(n_{\rm pix}^{(i)}\) is the number of pixels in all channels in the \(i^{\rm th}\) largest image to be served. For example, for a machine with 256GB or physical memory, serving gigapixel-sized images, the maximum recommended number of workers would be about 60 (64 minus some margin for the system and for memory caching). Note that having more workers than CPU cores on the machine will not improve serving performance, hence if the machine in the example above has 48 CPU cores in total, then there is a priori no reason for the number of workers to exceed 48. Finally, it is worth stressing that all systems but those equipped with the fastest storage hardware can become I/O-bound when too many workers are operating in parallel. Limited I/O performance may somewhat be mitigated by the page cache that comes with the OS, although this depends on the number of different images being served at a given time. Hence it is recommended to act with caution when increasing the number of workers to large values.

Multithreading is used more sporadically at lower levels, for instance when generating cache data from MEF files. It is set by default to half the number of "CPUs" reported by the operating system (which is often the number of physical CPUs). It is recommended to leave it unchanged, unless one intends to run several instances of VisiOmatic on the same machine.

Image caches

Caching of image data is essential to the good performance of VisiOmatic. Caching of the image tiles computed by the server component occurs at several levels, all of which following the LRU ("Least-Recently-Used") policy.

Browser cache

The first level is that of the web browser, which has its own cache on the client side. It is particularly useful when displaying animated sequences. The browser cache does not require specific tuning when using the VisiOmatic web client, however it may sometimes interfere with the expected behavior of the visualisation engine when rapidly updating image parameters, and lead to inconsistencies between the displayed tiles. In this case, clearing the cache of the browser and reloading the page will solve the issue.

Server cache

The second cache level, which may or may not be present, is that of the (optional) web server acting as a reverse proxy to the VisiOmatic server component. Please check your web server documentation for configuration tips (e.g. nginx caching or Apache caching).

The VisiOmatic server code itself provides the next two caches, in the form of a memory cache and a disk cache.

Memory tile cache

The memory cache deals with the JPEG-encoded tiles. By default, up to 1,000 encoded tiles, or about 20-30 megabytes (per worker) are cached in memory. This limit may be increased or decreased using the max_cache_tile_count option.

Disk image cache

The client requests managed by the server component do not deal directly with FITS image arrays. Instead, they get their data from tiled, multi-resolution image files stored in the VisiOmatic LRU disk cache. The cache data are generated on-the-fly and memory-mapped by the server component. By default, up to 100 image files can be cached on disk. This limit may be increased or decreased using the max_cache_image_count option.

The directory path where the cached data are written and read can be specified with the cache_dir option. The default path depends on the platform; on Linux, it is ~/.cache/visiomatic. One should ensure that enough disk space is available for the cached data, knowing that an image with \(n_{\rm pix}\) pixels uses about \(10\,n_{\rm pix}\) bytes of cache disk space. The read/write performance of the device that carries the cache directory largely defines the performance and responsiveness of the VisiOmatic server component. Hence make sure that the cache directory is located on the fastest block device the server machine has access too, ideally a local, recent solid state drive. Another possibility, if the machine has enough memory, is to use a RAM drive (e.g., /dev/shm on Linux). A RAM drive provides the best possible cache performance [1], however the cache will not survive a reboot. In all cases, care must be taken to ensure that the filesystem does not get in the way of memory-mapping operations; for instance ZFS comes with its own caching mechanism that can negatively impact memory-mapping performance.

[1]

Such an option may seem somewhat counterproductive in terms of performance, because memory space dedicated to the RAM drive competes with that of the built-in page cache the OS uses to cache memory-mapped data on disk. However, measurements show that the RAM drive option is indeed the fastest in practice.