DeepLenstronomy Configuration Files

An overview of how to structure the main configuration file for deeplenstronomy.

Introduction

The purpose of deeplenstronomy is to wrap the strong-lensing image simulation utilities of lenstronomy into a convenient and reproducible framework that enables various science analyses. This is achieved in deeplenstronomy by allowing the user to generate an entire large-scale dataset from a single, human-readable configuration file.

This tutorial will help you to create the main configuration file for problem you are working on.

Contents

Sections in the Config File

The main configuration file for deeplenstronomy is a yaml-style document. The top-level keywords in this document will tell deeplenstronomy everything it needs to know to build you a dataset. These keywords are mandatory for deeplenstronomy to function properly.

Note: You must use spaces (not tabs) for indentation in deeplenstronomy configuration files.

The DATASET Section

This section will contain information regarding the dataset as a whole, rather than the individual images or systems you want to generate. Here you must specify the NAME and SIZE of the dataset. If you plan on saving the dataset, you must also specify the OUTDIR where you want the simulated images and associated metadata to be located. This is done with a block like this

DATASET:
    NAME: MyDeeplenstronomyDataset
    PARAMETERS:
        SIZE: 100
        OUTDIR: MySimulationResults
    SEED: 6     # optional

This block will set the name of the dataset to MyDeeplenstronomyDataset, specify that the dataset will contain 100 images, set a random seed for the simulation, and specify that the simulated images and associated metadata will be saved in a directory named MySimulationResults. If this directory does not already exist on your system, deeplenstronomy will create it automatically.

The COSMOLOGY Section

This section will specify the cosmology with which lenstronomy will perform all calculations. In the config file, the bare-minimum cosmology section would look like this

COSMOLOGY:
    PARAMETERS:
        H0: 70
        Om0: 0.3

Internally, all that happens within deeplenstronomy is the instantiation of an astropy.cosmology.FlatLambdaCDM object. Thus, you can specify the value of H0, Om0, Tcmb0, Neff, m_nu, and Ob0. You can also view the astropy documentation for more information on these parameters.

The IMAGE Section

This section will specify all the information needed to make simulated images look realistic. The following parameters are required for deeplenstronomy to generate images:

A typical IMAGE block will look like this

IMAGE:
    PARAMETERS:
        exposure_time: 90
        numPix: 100
        pixel_scale: 0.263
        psf_type: 'GAUSSIAN'
        read_noise: 7
        ccd_gain: 6.083

The SURVEY Section

This section specifies the properties of the survey that collected the dataset you are seeking to simulate. This could be the characteristics of DES, LSST, SDSS, ZTF, etc. The parameters deeplenstronomy needs in this section are:

Thus, an example SURVEY block could look like this

SURVEY:
    PARAMETERS:
        BANDS: g,r,i,z,Y
        seeing: 0.9
        magnitude_zero_point: 30.0
        sky_brightness: 23.5
        num_exposures: 10

Often the parameters in the SURVEY section will have a different value for each band. This relationship is technically a correlation between the band and the parameter, and therefore is covered in the Advanced Features section of this document.

Half-Way-There Summary

Just to recap what we've talked about, let's put everything together. So far, we've covered the top-level aspects of the simulation: the properties of the dataset, the cosmological model for calculations, the image properties, and the survey conditions. Our in-progress config file currently looks like this:

DATASET:
    NAME: MyDeeplenstronomyDataset
    PARAMETERS:
        SIZE: 100
        OUTDIR: MySimulationResults
    SEED: 6

COSMOLOGY:
    PARAMETERS:
        H0: 70
        Om0: 0.3

IMAGE:
    PARAMETERS:
        exposure_time: 90
        numPix: 100
        pixel_scale: 0.263
        psf_type: 'GAUSSIAN'
        read_noise: 7
        ccd_gain: 6.083

SURVEY:
    PARAMETERS:
        BANDS: g,r,i,z,Y
        seeing: 0.9
        magnitude_zero_point: 30.0
        sky_brightness: 23.5
        num_exposures: 10

Next, we'll look at how to specify the properties of the physical objects in the images as well as the geometry of the strong-lensing systems you're interested in.

The SPECIES Section

This section specifies the properties of each object or noise source that could potentially be in one of your images. There are currently three species of object supported in deeplenstronomy: GALAXY, POINTSOURCE, and NOISE. The most information-rich species is GALAXY, so let's start there.

GALAXY. This species has up to 5 attributes for which you can provide information. The first of these is the NAME keyword, which deeplenstronomy will use internally to group the properties of the object. Thus, the NAME attribute is mandatory and must be unique. The next three attributes (light profiles, mass profiles, and shear profiles) are the core of the lenstronomy simulation.

Light profiles are specified with the LIGHT_PROFILE_X keyword and describe how light is distributed around the object. The X represents an index to differentiate multiple light profiles in the case where you would like to utilize a superposition of profiles and must be indexed incrementally starting at 1. The available profiles are any that are available in lenstronomy, and you can look at the lenstronomy documentation to see all the options and associated parameters.

As an example, let's create a GALAXY species that we'll name "LENS" with a "SERSIC_ELLIPSE" light profile.

SPECIES:
    GALAXY_1:
        NAME: LENS
        LIGHT_PROFILE_1:
            NAME: SERSIC_ELLIPSE
            PARAMETERS:
                magnitude: 19.5
                center_x: 0.0
                center_y: 0.0
                R_sersic: 10
                n_sersic: 4
                e1: 0.2
                e2: -0.1

The parameters you need to specify will depend on the lenstronomy light profile you select, so check the lenstronomy documentation for what parameters you need. deeplenstronomy will warn you if you supply an incorrect parameter. deeplenstronomy light profiles use magnitude as the defualt way of specifying the brightness; in lenstronomy you also have the amp parameter available, but that it not utilized in deeplenstronomy. If we wanted to add a second light profile to this GALAXY, perhaps to add more light in the center, we can do so by adding a second section like this:

SPECIES:
    GALAXY_1:
        NAME: LENS
        LIGHT_PROFILE_1:
            NAME: SERSIC_ELLIPSE
            PARAMETERS:
                magnitude: 19.5
                center_x: 0.0
                center_y: 0.0
                R_sersic: 10
                n_sersic: 4
                e1: 0.2
                e2: -0.1
        LIGHT_PROFILE_2:
            NAME: SERSIC_ELLIPSE
            PARAMETERS:
                magnitude: 18.0
                center_x: 0.0
                center_y: 0.0
                R_sersic: 3
                n_sersic: 8
                e1: 0.05
                e2: -0.05

The next bit of information lenstronomy needs to calculate how light is lensed is the mass distibution. Since we have conveniently named this GALAXY "LENS", it's not too farfetched to image that it will act as the lensing galaxy when this is all said and done, so we'll give it what deeplenstronomy calls a mass profile.

Mass profiles will also draw directly from the lenstronomy allowed models, so check the lenstronomy documentation for all your options. In deeplenstronomy, you will adopt the same indexing scheme as the light profiles. Let's add a mass profile to "LENS".

SPECIES:
    GALAXY_1:
        NAME: LENS
        LIGHT_PROFILE_1:
            NAME: SERSIC_ELLIPSE
            PARAMETERS:
                magnitude: 19.5
                center_x: 0.0
                center_y: 0.0
                R_sersic: 10
                n_sersic: 4
                e1: 0.2
                e2: -0.1
        LIGHT_PROFILE_2:
            NAME: SERSIC_ELLIPSE
            PARAMETERS:
                magnitude: 18.0
                center_x: 0.0
                center_y: 0.0
                R_sersic: 3
                n_sersic: 8
                e1: 0.05
                e2: -0.05
        MASS_PROFILE_1:
            NAME: SIE 
            PARAMETERS:
                theta_E: 2.0
                e1: 0.1
                e2: -0.1
                center_x: 0.0
                center_y: 0.0

Here we've used lenstronomy's "SIE" model and specified the necessary parameters for it. Let's finish up describing "LENS" by giving it a shear profile. A shear profile describes how all the light in the system is lensed by an external source, perhaps a dark matter halo. You can add this type of profile to "LENS" as follows:

SPECIES:
    GALAXY_1:
        NAME: LENS
        LIGHT_PROFILE_1:
            NAME: SERSIC_ELLIPSE
            PARAMETERS:
                magnitude: 19.5
                center_x: 0.0
                center_y: 0.0
                R_sersic: 10
                n_sersic: 4
                e1: 0.2
                e2: -0.1
        LIGHT_PROFILE_2:
            NAME: SERSIC_ELLIPSE
            PARAMETERS:
                magnitude: 18.0
                center_x: 0.0
                center_y: 0.0
                R_sersic: 3
                n_sersic: 8
                e1: 0.05
                e2: -0.05
        MASS_PROFILE_1:
            NAME: SIE 
            PARAMETERS:
                theta_E: 2.0
                e1: 0.1
                e2: -0.1
                center_x: 0.0
                center_y: 0.0
        SHEAR_PROFILE_1:
            NAME: SHEAR
            PARAMETERS:
                gamma1: 0.08
                gamma2: 0.01

Here we've used lenstronomy's "SHEAR" model and specified the necessary parameters for it to work.

To specify GALAXY objects with different properties, just create a new entry in the SPECIES section. Since we will want to do some lensing eventually, lets create another galaxy that will act the light that is actually being lensed. Fittingly, we might choose to name this GALAXY object "SOURCE". The SPECIES section would then look like this

SPECIES:
    GALAXY_1:
        NAME: LENS
        LIGHT_PROFILE_1:
            NAME: SERSIC_ELLIPSE
            PARAMETERS:
                magnitude: 19.5
                center_x: 0.0
                center_y: 0.0
                R_sersic: 10
                n_sersic: 4
                e1: 0.2
                e2: -0.1
        LIGHT_PROFILE_2:
            NAME: SERSIC_ELLIPSE
            PARAMETERS:
                magnitude: 18.0
                center_x: 0.0
                center_y: 0.0
                R_sersic: 3
                n_sersic: 8
                e1: 0.05
                e2: -0.05
        MASS_PROFILE_1:
            NAME: SIE 
            PARAMETERS:
                theta_E: 2.0
                e1: 0.1
                e2: -0.1
                center_x: 0.0
                center_y: 0.0
        SHEAR_PROFILE_1:
            NAME: SHEAR
            PARAMETERS:
                gamma1: 0.08
                gamma2: 0.01
    GALAXY_2:
        NAME: SOURCE
        LIGHT_PROFILE_1:
            NAME: SERSIC_ELLIPSE
            PARAMETERS:
                magnitude: 21.5
                center_x: 0.0
                center_y: 0.0
                R_sersic: 6
                n_sersic: 5
                e1: 0.2
                e2: -0.1 
        SHEAR_PROFILE_1:
            NAME: SHEAR
            PARAMETERS:
                gamma1: 0.08
                gamma2: 0.01

Since we'll be using this galaxy as the so-called source in our lensing system, we don't need to worry about specifying a mass profile. We have also given it the same shear profile as the lens, since they will be part of the same system.

While the GALAXY objects will perhaps require the most thought when constructing the config file, the POINTSOURCE and NOISE objects will be much more straightforward.

POINTSOURCE. This object will allow you to add a point-like light source to the images. You may wish to utilize this feature to include foreground stars, to make a galaxy look like an AGN, or to include a supernova or other transient in the image. I'll show all three of those possibilities here.

Let's start with an AGN. In similar fashion to the way we specified the different GALAXY objects, we'll describe a point source object as follows:

<Still inside SPECIES section>
    POINTSOURCE_1:
        NAME: AGN
        HOST: SOURCE
        PARAMETERS:
            magnitude: 16

Each POINTSOURCE object is characterized by a NAME, a HOST, and magnitude, sep, sep_unit, and angle parameters. The NAME attribute will be used to track all properties of this object, so make sure all names are unique. The HOST attribute will tell deeplenstronomy where your object will be in the image.

The additional parameters determine how the POINTSOURCE will appear. The sep parameter will specify how far to distance the POINTSOURCE from its host. If this parameter is not specified, the POINTSOURCE will be placed exactly on the center of the HOST. If you specify sep, you also need to sepcify the sep_unit (either kpc or arcsec) to indicate the physical units you wish to use. The angle parameter can also be specified if you wish to place the POINTSOURCE in a certain orientation with respect to its host. The units of angle are radians and the value must be between 0 and 2$\pi$. In this case, we have selected to place this point source on the center of the GALAXY object that we named "SOURCE". Lastly, the magnitude is the apparent magnitude of the point source.

If we wanted, we could also add a supernova-like point source by specifying a nonzero sep parameter. The config entry would then look like this:

<Still inside SPECIES section>
    POINTSOURCE_2:
        NAME: SUPERNOVA
        HOST: SOURCE
        PARAMETERS:
            magnitude: 21.0
            sep: 2.0
        sep_unit: arcsec

Specifying this second POINTSOURCE for the GALAXY we named "SOURCE" will not by itself put both POINTSOURCES in "SOURCE" at the same time. That aspect is not determined until the GEOMETRY section of the config file. Here, you are just preparing deeplenstronomy to put your POINTSOURCEs in "SOURCE" later on.

The final type way you may choose to use the POINTSOURCE object is to include foreground stars. This is done by setting the HOST keyword to 'Foreground' as follows:

<Still inside SPECIES section>
    POINTSOURCE_3:
        NAME: STAR
        HOST: Foreground
        PARAMETERS:
            magnitude: 14.0

This setting will place the point source randomly in your image, and it will not contribute to or be affected by any of the lensing calculations.

NOISE. The final type of object that deeplenstronomy allows you to include in your images is the NOISE object. Currently, only POISSON_NOISE is built in to deeplenstronomy, but other types of noise are on the way. If you would like a specific type of noise, just ping me and I'll make it happen.

You can define a NOISE species like this:

<Still inside SPECIES section>
    NOISE_1:
        NAME: POISSON_NOISE
        PARAMETERS:
            mean: 2.0

In this implementation, each pixel in the image will have a noise value added to it, and that noise value will be drawn from a Poisson distribution with mean 2.0. The values assigned to different pixels are assumed to be independent.

Summary of the SPECIES Section

That certainly was a lot of material. The SPECIES section carries all the properties of all the objects in your entire dataset, so it is important to create it with care. To summarize this component of the config file, let's look at the SEPCIES section in our example as a whole.

SPECIES:
    GALAXY_1:
        NAME: LENS
        LIGHT_PROFILE_1:
            NAME: SERSIC_ELLIPSE
            PARAMETERS:
                magnitude: 19.5
                center_x: 0.0
                center_y: 0.0
                R_sersic: 10
                n_sersic: 4
                e1: 0.2
                e2: -0.1
        LIGHT_PROFILE_2:
            NAME: SERSIC_ELLIPSE
            PARAMETERS:
                magnitude: 18.0
                center_x: 0.0
                center_y: 0.0
                R_sersic: 3
                n_sersic: 8
                e1: 0.05
                e2: -0.05
        MASS_PROFILE_1:
            NAME: SIE 
            PARAMETERS:
                theta_E: 2.0
                e1: 0.1
                e2: -0.1
                center_x: 0.0
                center_y: 0.0
        SHEAR_PROFILE_1:
            NAME: SHEAR
            PARAMETERS:
                gamma1: 0.08
                gamma2: 0.01
    GALAXY_2:
        NAME: SOURCE
        LIGHT_PROFILE_1:
            NAME: SERSIC_ELLIPSE
            PARAMETERS:
                magnitude: 21.5
                center_x: 0.0
                center_y: 0.0
                R_sersic: 6
                n_sersic: 5
                e1: 0.2
                e2: -0.1 
        SHEAR_PROFILE_1:
            NAME: SHEAR
            PARAMETERS:
                gamma1: 0.08
                gamma2: 0.01               
    POINTSOURCE_1:
        NAME: AGN
        HOST: SOURCE
        PARAMETERS:
            magnitude: 16
    POINTSOURCE_2:
        NAME: SUPERNOVA
        HOST: SOURCE
        PARAMETERS:
            magnitude: 21.0
            sep: 2.0
            sep_unit: arcsec
    POINTSOURCE_3:
        NAME: STAR
        HOST: Foreground
        PARAMETERS:
            magnitude: 14.0
    NOISE_1:
        NAME: POISSON_NOISE
        PARAMETERS:
            mean: 2.0

In making the dataset, this is the collection of objects you will have at your disposal. We've created two different GALAXYs, three different POINTSOURCEs, and a single NOISE source. Now we'll see how to specify the geometry of the systems that these objects will constitute.

The GEOMETRY Section

This is the final required section in the config file, and will contain information about the relative orientations of the objects in the SPECIES section. The geometry section is organized by the user specifying different CONFIGURATIONs, which will follow the same indexing rules as the objects in the SPECIES section.

Each configuration will have a NAME, a FRACTION, and then informataion about the physical systems. The NAME will be used to track the association of properties within deeplenstronomy. The FRACTION is how much of the dataset you would like to be in the specific configuration. Internally, deeplenstronomy just multiplies the FRACTION by the SIZE in the DATASET section to obtain a number of images to simulate, but it is necessary that the FRACTION values across your CONFIGURATIONs sum to one.

Next, you will defind the PLANEs in your physical systems. Let's look at an example to work through this.

GEOMETRY:
    CONFIGURATION_1:
        NAME: GALAXY_AGN
        FRACTION: 0.25
        PLANE_1:
            OBJECT_1: LENS
            PARAMETERS:
                REDSHIFT: 0.2                    
        PLANE_2:
            OBJECT_1: SOURCE
            OBJECT_2: AGN
            PARAMETERS:
                REDSHIFT: 0.7                  
        NOISE_SOURCE_1: POISSON_NOISE

After the NAME and FRACTION attributes, we can see a PLANE_1 and a PLANE_2 have been defined. PLANEs require you to list the objects you want in the plane and to define the redshift as a parameter. The defining characteristic of a PLANE in deeplenstronomy is that all objects place in that plane are assigned the same redshift.

Here, we've chosen to put a single object in PLANE_1, which is the plane closest to Earth. We've specified which SPECIES to serve as OBJECT_1 by using the name "LENS", which recall was the name of GALAXY_1 in the SPECIES section. We've also set this plane to have a redshift of 0.2.

In PLANE_2, we've place two objects: "SOURCE" which refers to GALAXY_2 and "AGN" which refers to POINTSOURCE_1. We've set this plane to have redshift 0.7 so that it will be behind the objects in PLANE_1.

Lastly, we specify that we want to simulate extra noise on top of the image by specifying a single noise source: NOISE_SOURCE_1. We tell deeplenstronomy that we want "POISSON_NOISE" to be used which will reference NOISE_1 in the SPECIES section since it was named "POISSON_NOISE".

Now. That seems like a lot, but let's say you also want to simulate a set of images with no noise to evaluate the performance of some neural network on high quality images. All you need to change to the config file is to add another configuration:

<Still inside GEOMETRY section>
    CONFIGURATION_2:
        NAME: GALAXY_AGN_NOISELESS
        FRACTION: 0.25
        PLANE_1:
            OBJECT_1: LENS
            PARAMETERS:
                REDSHIFT: 0.2                    
        PLANE_2:
            OBJECT_1: SOURCE
            OBJECT_2: AGN
            PARAMETERS:
                REDSHIFT: 0.7

What if we want to simulate lensed supernovae instead of lensed AGN? Just add another configuration:

<Still inside GEOMETRY section>
    CONFIGURATION_3:
        NAME: LENSED_SNE
        FRACTION: 0.25
        PLANE_1:
            OBJECT_1: LENS
            PARAMETERS:
                REDSHIFT: 0.2                    
        PLANE_2:
            OBJECT_1: SOURCE
            OBJECT_2: SUPERNOVA
            PARAMETERS:
                REDSHIFT: 0.7      
        NOISE_SOURCE_1: POISSON_NOISE

What if you're looking to add some spice to your life, and you want to include in this dataset a double source plane system, with a supernova and AGN in the source galaxy, two foreground stars, and Poisson noise? Just add another configuration:

<Still inside GEOMETRY section>
    CONFIGURATION_4:
        NAME: SPICY_LIFE
        FRACTION: 0.25
        PLANE_1:
            OBJECT_1: LENS
            OBJECT_2: STAR
            OBJECT_3: STAR
            PARAMETERS:
                REDSHIFT: 0.2                    
        PLANE_2:
            OBJECT_1: LENS
            PARAMETERS:
                REDSHIFT: 0.7
        PLANE_3:
            OBJECT_1: SOURCE
            OBJECT_2: SUPERNOVA
            OBJECT_3: AGN
            PARAMETERS:
                REDSHIFT: 1.3      
        NOISE_SOURCE_1: POISSON_NOISE

Whatever you want to do, the answer will likely be "Just add another configuration."

One of the main ideas behind deeplenstronomy is that once you have defined all the properties of the objects, survey, and images, you are free to simulate any physical system you want.

Summary

That's the basics of how to organize the main config file. You will need DATASET, COSMOLOGY, IMAGE, SURVEY, SPECIES, and GEOMETRY sections. These sections make it possible to separate different aspects of the simulation, hopefully in a way that seems logical to astronomers.

Next we'll look at how to draw parameter values from distributions instead of hard-coding fixed values into the main config file.

Drawing from Probability Distributions

It doesn't do much good to have to specify all the light profile, mass profile, redshift, survey, image (and many more) parameters directly in the config file. You can, but often you will want to draw those parameters from a probability distribution. This action is communicated to deeplenstronomy by using the DISTRIBUTION keyword in place of the actual parameter value.

As an example, maybe you want to have a range of magnitudes for an object. Furthermore, let's say you want this range to be a uniform distribution from 16.0 to 23.0. In the main config file, replace

magnitude: 20.0

with

magnitude:
    DISTRIBUTION:
        NAME: uniform
        PARAMETERS:
            minimum: 16.0
            maximum: 23.0

and deeplenstronomy will do the rest. This approach will work for any parameter.

Available Distributions

The name "uniform" targets a specific function uniform() in the file deeplenstronomy/distributions.py. This file contains several distributions that can also be utilized by specifying their name. The main functions are:

You are welcome to edit the source code to add any distribution you need, or open a pull request if you think it will be useful for others.

Advanced Features

Above is all the information you need to begin constructing configuration files and building deeplenstronomy datasets. To sculpt your simulated datasets with a specific science case in mind, you may find some of these advanced features useful.

Forcing Correlations

Here we have looked at how to draw parameter values from one-dimensional distributions. In practice, you may find it useful to enforce physically-motivated correlations, say between an object's redshift and magnitude, or have color-dependent properties of an object. In either case, this can be accomplished using "User-Specified Distributions". In the configuration file, you can add an entry (at the same level as IMAGE, SURVEY, GEOMETRY, etc.) like this:

DISTRIBUTIONS:
    USERDIST_1:
        FILENAME: distribution_file.txt
        MODE: interpolate

The properties of the file distribution_file.txt are described in the "UserDistributions" Notebook.

Including Personal Images

You may find it useful to include real astronomical images in your simulations, either to have real-data like noise or to use actual object pictures in your dataset. In the configuration file, you can add an entry (at the same level as IMAGE, SURVEY, GEOMETRY, etc.) like this:

BACKGROUNDS: 
    PATH: <path/to/image_directory_name> #(no trailing '/')
    CONFIGURATIONS: <configuration list, e.g. ['CONFIGURATION_1'] or ['CONFIGURATION_1', 'CONFIGURATION_3']>

The PATH and CONFIGURATIONS keys are described in detail in the "UserImages" Notebook.

Supplemental Input Files

As you may have noticed by now, this main config file can get pretty long. To make staying organized when working in this framework a little easier, try out the INPUT keyword.

As an example, let's clean up the SURVEY section. This section currently looks like this:

SURVEY:
    PARAMETERS:
        BANDS: g,r,i,z,Y
        seeing: 0.9
        magnitude_zero_point: 30.0
        sky_brightness: 27.5
        num_exposures: 10

Copy the parameters subsection into a new file, which can be named whatever you want, but we'll name the file survey.yaml. The file will look like this:

PARAMETERS:
    BANDS: g,r,i,z,Y
    seeing: 0.9
    magnitude_zero_point: 30.0
    sky_brightness: 27.5
    num_exposures: 10

Now in the main config file, change the SURVEY section to this:

SURVEY:
    INPUT: survey.yaml

Thus, the INPUT keyword will substitue the contents of whatever file it is used on in place of itself. Hopefully, you will find this tool useful in modularizing and organizing your deeplenstronomy workflow.

Troubleshooting

This software is new and certainly breakable. In an ideal world, the only time you will encounter an error will be typos in the configuration file. deeplenstronomy automatically parses your configuration file for areas that are entered incorrectly and will alert you before simulating your dataset. However, the checks it performs are by no means exhaustive. If you do encounter an error, please open an issue so that it can be addressed in the list of checks deeplenstronomy performs at runtime.