installSynApps Documentation

A python3 module for cloning and building EPICS and synApps

Author: Jakub Wlodek

Corresponding author: Kazimierz Gofron

Introduction:

installSynApps is a python3 module that allows users to clone and build EPICS and synApps with a modular configuration system. Configuration options for building are automatically handled by the module, greatly simplifying the process for users unfamiliar with the EPICS build environment. The module is capable of injecting configuration options into target files as well, which is useful when external modules are being compiled that do not have the appropriate Makefile options within EPICS. installSynApps also allows for auto setting of build flags, which further simplifies the process of compiling EPICS base, synApps, and areaDetector.

Installation:

To install the installSynApps module, you may either clone the git repository, or download a release. From there, make sure that it is unpacked if it was downloaded as a tarfile or zip archive.

installSynApps requires that several external programs be installed and in the system path in order to run. Firstly, it requires python3, and if using the GUI version, Tkinter. On windows, python3 (which should include Tkinter) can be downloaded from the python website. On linux, you may also use the website, or simply use the package manager, running the sudo apt install python3 python3-tk command. The minimum supported python version is 3.4+, and on linux distributions that have an older version in their package repositories it is required to download one from an external source and use it in a virtual env.

Next, installSynApps requires git, wget, and tar in order to download and unpack the entirety of EPICS and synApps. On linux, simply install these packages through the package manager: sudo apt install git wget tar. On windows, you may find wget here, and git and tar can be downloaded from the git website. When installing git, you will have the option to place git in your system path, which you should enable, and an option to install "UNIX tools", which will include tar. Finally, installSynApps requires make, perl, and a C/C++ compiler in order to actually build EPICS. On linux, install these by simply running sudo apt install make gcc g++ perl. On windows, it is recommended to use the MinGW version of make, which can be found here, Strawberry perl can be found here, and the recommended C/C++ compiler is a recent version of the MSVC/MSVC++ compiler included with Visual Studio. Install all of these packages, and installSynApps is now ready for use.

Windows Setup:

On Windows, prior to running installSynApps, it is required to set up your build environment. First, all of the above described required packages must be in your system path. You may test this by opening a command prompt and typing make, perl, git, tar, and wget. If the output describes the usage of each module, it is correctly placed in the system path. If the message states that it is not a recognized program or batch file, check to make sure it was added to the path. Then, to initialize a Visual Studio build environment, in the command prompt, navigate to the Visual Studio intall directory (usually C:\Program Files (x86)), and follow this path: "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\" replacing 2017 for your version of Visual Studio. Once in this directory, run the following command: vcvarsall.bat x86_amd64. This will set up your command prompt for buildng C/C++ applications with MSVC++ for 64bit windows. Next, run set EPICS_HOST_ARCH=windows-x64-static, and set MAKEFLAGS=-w. This will set some environment variables needed for the build. Once this is done, navigate to the installSynApps directory, and proceed as described below.

Install Configuration Setup

The most important file for configuring your installation of EPICS, synApps, and areaDetector is the configure/INSTALL_CONFIG file:
In this file, you will find defined your INSTALL location. INSTALL represents where installSynApps will clone all of its modules. Below this are the modules themselves, along with GIT_URL and WGET_URL definitions. Each module has a name, version, relative path, repository name, and options to clone/build/package it. installSynApps will locate the module at the most recently listed URL appended with the repository, and will use git clone if it is a GIT_URL, or wget if it is a WGET_URL. Simply edit the version numbers and the CLONE/BUILD options as demonstrated in the image above to edit your install configuration. In addition, you may edit the INSTALL variable above to customize the install location. Note that all of these options may be customized through the GUI if using installGUI rather than installCLI.

Usage - installCLI

One way to run installSynApps is to use the installCLI.py script. On linux this is done by running ./installCLI.py, and on windows: python installCLI.py.
Once the script is run, it will begin execution. Simply follow the instructions as they come up. You may be prompted to enter your sudo password if running on linux to install dependency packages. Simply enter 'y' when prompted to continue, or 'n' to skip the next step. The script will ask before starting each stage of the build process so if you wish to skip certain portions, simply select 'n'.

installCLI also supports several optional flags as arguments. The '-h' flag will explain the remaining flags. The -y flag will simply automatically answer 'y' to each of the queries, and the '-c CONFIG PATH' flag allows for selecting a different configure directory than the default configure/. The default configure folder is tailored to linux installations, and thus when building for windows it is recommended to use the '-c' flag and point to addtlConfDirs/configureWindows.

Usage - installGUI

The recommended way to use installSynApps is to use installGUI. To start it, in the terminal that has been configured with the proper build environment, run ./installGUI.py on linux, and python installGUI.py on windows. Note that it must be started from the terminal with the proper build environment, as installSynApps uses terminal utilities via the python 'subprocess' module to compile EPICS and synApps. Once it is run, you will see the following screen:

On the bottom left, you will see the currently loaded install configuration. On the right you will see the log which will show information regarding the clone and build. By default info, warning, and error messages are only written to the log, though popups can be Toggled by entering Edit -> Toggle Popups. On the upper-right, you will see the 8 main control buttons.

In order to load a new Install configuration, click the 'Load Config' button, or select 'File -> Open'. Then select the directory containing a valid install configuration. Note that if it is not valid, the currently loaded install config will remain the same. Once a config is loaded, you may edit it by entering the Edit menu and selecting an appropriate option. 'Edit -> Edit Config' will open a window that allows for selecting the install location, which module versions are selected, and whether or not the modules should be cloned and or built.

Simply edit whatever you wish and select either apply or apply and exit. You can see the panel displaying the currently loaded install config has been updated to reflect your changes. The 'Edit -> Edit Injector Files' option will open a screen that displays the injector files that will be applied to your build process. Selecting each one from the dropdown will show which file the text will be injected into. You may simply edit these in the given window, and press apply changes. If you wish to reload what it had been previously, provided you have not applied your changes, you may select 'Reload'.

Finally, selecting 'Edit -> Edit Build Flags' will open a window that will allow you to set macro values that will be applied to the areaDetector bulid. Simply follow the MACRO=VALUE standard as demonstrated.

Once you are satisfied with your loaded configuration it may be wise to save it with 'File -> Save As', and then it can be applied using either 'Autorun' to run all processes sequentially, or individually by pressing each of the appropriate buttons. The log will output status messages, and any further instructions. buttons

installSynApps Releases:

R2-7 (7-December-2020)

• Features Added

  • Migrate CI to github actions
  • Buildable with pip, creates utility epics-install
  • Default config creation overhaul
  • Remove dependency on make release
  • Use requests module if possible

• Bug Fixes/Improvements

  • Resolve bug with recursive cloning of EPICS base
  • Resolve bugs with building on windows
  • Resolve bug with bundle generation with softlinks
  • Resolve minor unit test issues
  • Resolve issue with module ordering
  • Fix some minor module names
  • Fixes for minor GUI based issues
  • Remove some redundant folders
  • Fix issues with building Xspress3 from APS
  • Improved logging

R2-6 (22-May-2020)

• Features Added

  • Allow for generating Debug versions of bundles that include sources
  • Generation of dummy IOCs that are included with the bundles
  • Integrate certain initIOC features in new modules.

• Bug Fixes/Improvements

  • Fixed issue with certain relative paths being incorrectly represented
  • Fix minor bugs in installCLI build script
  • Fix minor issue in envPaths generation
  • Documentation cleanup - redundant docstrings removed

R2-5 (26-Febuary-2020)

• Features Added

  • Utilize improved README file format for source bundles
  • New Screen for creating new configurations.
  • setup.py file worked on. Can install both scripts, but need external configuration
  • regenerateSources.sh script is now generated during build. Can be used to recreate sources.

• Bug Fixes/Improvements

  • Fixed several minor issues causing failed module building with default configure directory
  • New configs should now have populated injectionFiles and macroFiles directories.
  • Fix issue with building QuadEM on certain distributions.
  • Remove some unnecessary prompts from installCLI
  • Fix minor issues found by linter in installGUI
  • Refactor installCLI for more modularity
  • Fix sync-tags auto overwrite in GUI
  • Add injection and macro files to resources (were missing from new configs)
  • More robust debug options
  • Renamed submodules for consistency

• Future Plans

  • Improve documentation - Use sphinx for auto-building docs from source code
  • Include better handling of module by module dependency packages
  • Work on improving support for auto-building motion IOCs.
  • Add more support for running installSynApps in an automated environment (ansible)

R2-4 (10-January-2020)

• Features Added

  • OPI directory generation - Creates OPI bundle from all modules in configuration
  • Add-On Generation. Create an add on package to add new modules to existing bundles.
  • Parser now can update module path macros in macro files prior to configure update step.

• Bug Fixes/Improvements

  • Removed pygithub dependancy. Versions are now updated using git ls-remote
  • Improved version detection. Tags are now better analyzed to avoid false positives.
  • Fix bug on version sync in installCLI where updated versions wouldn't save
  • Moved printing of commands to logging module
  • Reorganize packager to remove redundant code
  • In-code documentation improved for installGUI

• Future Plans

  • Improve automation for version updates (So Docker-Builder can perform these itself)
  • Add error handling for an expanded set of edge-cases

R2-3 (12-November-2019)

• Features Added

  • Automatic dependency detection. Modules are now reordered prior to building to account for dependencies
  • Improved logging. Dedicated module that gives for several degrees of granularity
  • Ability to select between flat/non-flat binaries
  • Ability to print bash/batch commands as they are printed

• Bug Fixes/Improvements

  • Greatly simplified build_driver module code.
  • Simplified packager. Packager now pulls from module.version for wget modules
  • Improved failure conditions. installSynApps should now better respond to non-critical errors.

• Future Plans

  • Improve documentation
  • Integrate with database for install configuration storage.

R2-2 (21-August-2019)

• Features Added

  • Integration with the github api via PyGithub to allow for version sync
  • Custom build script support - can now specify a custom build script for each non-core module
  • Option to manually set package output name
  • Option to manually specify install location in installCLI.py
  • Custom dependency script is now run before clone if selected.

• Bug Fixes/Improvements

  • Install location changes automatically whether on windows or linux
  • import guards now allow the program to run even if certain optional dependencies are missing
  • Improved error conditions should allow build to run smoother if issues are encountered.

• Future Plans

  • Improve log messages
  • Module reordering.

R2-1 (09-August-2019)

• Features Added

  • Integrated Packager - Can now create AD binary bundles directly from installSynApps
  • Metadata - Settings are now saved accross uses, so you don't need to keep loading configs
  • Core count regulation - Regulate maximum core count used by installSynApps
    • Useful for lower power devices
  • ConfigWriter - dedicated module for saving install configurations
  • installCLI automation - installCLI can now run fully automated without any user input
    • Can Clone, Update, Compile, and Package with one command
  • TravisCI automated testing added

• Bug Fixes

  • Removed unneccessary print statements in installCLI
  • Fixed issue where editing the configuration would not refresh any modules aside from builder
  • Expanded error checking - should fail more gracefully

• Future Plans

  • Add support for custom build scripts for each module
  • Add improved support for custom dependency scripts

R2-0 (10-July-2019)

• Features Added

  • Rework of installSynApps with code readability and cross-platform support in mind.
  • Legacy scripts still included, with slight edits to support new configure format
  • New installSynApps python module - allows for importing into other code.
  • Modular solution - Easy port to CLI and GUI versions
    • DataModel/ViewModel
  • New GUI version, with ability to edit almost all config options.
  • Tested an working on windows systems as well as linux - bash dependencies removed.
  • Changed License
  • Added unit testing and CI integration
  • Split initIOCs to separate module

• Bug Fixes

  • Numerous edge case error checking fixes
  • Thread cleanup fixes
  • Improved reliablility of auto-configuration
  • Fixes bugs that were caused from unexpected formats in Install configuration

• Future Plans

  • Integrate packager from ioc_deploy
  • Add support for metadata and saving settings from session to session

R1-0 (30-May-2019)

• Features Added

  • Initial release of installSynApps
  • Based around a series of scripts
  • Included scripts
    • auto_build.sh - guided process through running remaining scripts
    • installSynApps.sh - top level script that ran the remaining ones sequentially
    • dependencyInstall.sh - Script for installing dependency packages
    • read_install_config.py - Script for reading provided configure folder
    • clone_and_checkout.py - script for cloning and checking out all modules
    • ad_config_setup.py - port of adConfigSetup module - allowed for auto-config of areaDetector files
    • update_release_file.py - top level script for updating configuration options
    • buildEPICS.py - script for calling build commands
    • script_generator.py - autogenerates some helpful information about build

• Limitations:

  • Reliant on bash - only ran on Linux-based systems
  • Required constant edits of configure/INSTALL_CONFIG file.
  • No support for multiple configure directories
  • Code readability issues - arrays used to represent installation modules
  • Limited customizability
  • Issues when running scripts individually

Copyright and Notes

Copyright (c): Brookhaven National Laboratory as a part of Brookhaven Science Associates 2018 - 2019

Important Links:

• installSynApps on Github
• synApps on Github
• EPICS on Github
• areaDetector on Github
• synAppsRelease on Github