About the DeLINEATE Toolbox   Download   Documentation   Contact/Contribute 

 

 

PREFACE TO ALL DOCUMENTATION: We have tried to be as comprehensive, helpful, and accurate as we can in all of these documents, but providing good documentation is always an uphill climb. Our code, and the underlying backend code we rely on, is always changing, which means things can easily go out of date; and doing these kinds of analyses is intrinsically a complicated process, which makes it hard to write documentation that works well for people at all different levels of technical proficiency and familiarity with the underlying concepts and technologies. We really don't want the learning curve to be a barrier to people using this toolbox, so we highly recommend -- especially while the number of users is relatively small and manageable -- getting in touch with the developers if you're confused, don't know where to start, etc., etc. And, of course, if you find any errors, omissions, or inconsistencies! Seriously, don't be a stranger... we are happy to add features, flesh out documentation, walk through setup, and so on, to help this project serve as many users as possible, but that requires hearing from you to help let us know what areas need the most attention. Please see our website (http://delineate.it/) and click on the contact page to get a link to our Bitbucket repo and an email address for the project devs.

DTModel (class)

This is the class that is primarily responsible for storing your "model," i.e., the thing that gets trained during the training process and run during the testing phase. This could theoretically be any kind of machine-learning model if we implemented the corresponding backends, but right now we only support certain PyMVPA and Keras models. (It is relatively straightforward to add support for some of the additional features of those backends, so if you want to run something that isn't yet supported, drop us a note.) As much as possible, we try to abstract away the details of the backend implementation from the user, to make it easier to run the "same" analysis using, for example, a support vector machine in PyMVPA and then a multilayer perceptron in Keras, with minimal changes to the user's code or JSON job files. Your master DTAnalysis object will have one DTModel object representing the data for that analysis.

Like the other main classes in this toolbox, a DTModel object is typically created by DTJob from a specification contained in a JSON-format job file, although you can also create one manually in code if you know what you're doing.



Attributes

As we note above, we try in DTModel (and in the toolbox overall) to abstract away as many implementation details as possible so that the user doesn't have to change much when going from PyMVPA to Keras or vice versa. With that said, though, there is only so much we can strip away, so several of the attributes below are denoted as Keras-specific or PyMVPA-specific. Caveat emptor.

model_func (Python function object, or None): Keras-specific. If you're doing a Keras-type analysis, you obviously need to describe your neural network architecture (aka, your model) somehow. In most cases, and especially if you're using a JSON job file to create your analysis, the easiest way to do so would generally be via the layers attribute (see below).

However, if you're writing your own code, and you want to do a neural network model that is more complex than what the layers attribute can accommodate, model_func (and its companion attribute, model_args, described below) give you a less user-friendly but more flexible way to specify your neural network architecture.

All you need to do is write your own Python function somewhere that returns a Keras model object (either using the Sequential class or the Model class in Keras; see the Keras docs for details), and pass that function (as a Python function object) into the DTModel initializer, along with any relevant arguments in the model_args attribute (see below). When the initializer runs, it basically just runs your model_func function and passes the contents of the model_args attribute into it, takes whatever model object is returned, and sticks that into the model attribute (see below).

At this point you might be reasonable to ask yourself, "But couldn't I just create a dummy DTModel object, run my model-creating function myself, and manually assign that model to the DTModel.model attribute afterwards?"

Yes. You could also do that. But we like you to have options. (Also, at least for now, we enforce a nanny state where you can't leave both layers and model_func empty at initialization time; so if you DID want to swap in your own model attribute manually after the fact, you'd have to put in some kind of dummy data for either layers or model_func to keep the initializer from complaining and quitting.)

model_args (dictionary or None): Keras-specific. The companion attribute to model_func (described above); if you specify a model_func, you will also need to provide model_args. Basically, it's just whatever arguments you want to pass into your model-creating function.

model_args takes the form of a Python dictionary, where the keys in the dictionary are names of arguments taken by your model_func and the values in the dictionary are the values of those arguments. It's complicated to describe in English but if you are familiar with the relevant Python syntax, the line in the DTModel code is just this:

self.model = self.model_func(**self.model_args)

layers (Python list of dictionaries, or None): Keras-specific. This is the other way of specifying your neural network architecture if you aren't using model_func and model_args as described above; and currently, layers is the only way of implementing a Keras deep learning model using JSON job files. Using job files and the layers attribute is probably the easiest way to go for most users unless they need something fancier than a feedforward network in Keras's Sequential class. (If those words just sound like crazy made-up nonsense to you, you probably do not need anything fancier.)

Properly using the layers attribute requires a bit more background knowledge on neural nets and Keras's implementation of them than we can run down here, so you may want to check out our tutorials, documentation on JSON job files, general online tutorials on feedforward neural network architectures, and/or the Keras docs to get an overall lay of the land. But we'll give a rough outline and keep the jargon to as much of a minimum as we can.

Briefly, layers should be a list of dictionaries, where each dictionary in the list is a description of one layer in the network. The ordering of the list determines the order in the network; the first layer specified is the input layer, and after that come all the hidden layers, and the last dictionary in the list specifies the output layer. When designing your architecture, keep in mind that each layer's output becomes the next layer's input. (And also remember that your input layer shape has to match your data shape, and that your output layer should reflect the shape of the output you desire, i.e., if you're doing a run-of-the-mill classification, your output layer should have as many units as your dataset has classes.)

Now, what goes in each of those dictionaries? Boy, are we sorry you asked. OK, so, each dictionary can have a bunch of different keys/values in it. These keys/values generally represent the names and the values (respectively) of arguments taken by Keras functions that create different types of layers. What exact keys and values you can/will specify depends a lot on what kind of layer you're making at the time; for the most part, we just pass those arguments along to Keras and let it deal with the details. We highly recommend you peruse our JSON job file docs, tutorial videos, etc., to get a sense of this if you aren't already a Keras whiz.

However, in addition to those keys/values, there are also a limited number of "special" keys and values that are used by our toolbox and not passed along to Keras. Those "special" keys are as follows, with the data type of their corresponding values (in italics and parentheses):

  • layer_type (string): Probably the most special of those special keys; this is the name of the type of layer you want to create. These are names of Keras layer classes, such as Dense or Dropout or Flatten or Reshape or Permute or Conv2D or Conv3D or LSTM or Sleepy or Sneezy or Doc, just to name a few we use a lot. Wait, forget those last three.

    Anyway, in principle any of Keras's layer classes can be specified here, although we have certainly used/tested some more than others; the main constraint is that that if the layer requires an argument to create it that can't be described with JSON syntax (i.e., anything besides the numbers, strings, Boolean values, or lists/dictionaries of those basic data types that can be written in JSON), such as a function or a more complex type of Python object, you're out of luck. (There are some limited exceptions to this "you're out of luck" rule that will get described in the next bullet point.) Anyway, if you want to use one of the weirder layer types in Keras, feel free to try it, as most of them will work fine; but if it doesn't, get in touch with the devs and we are happy to add explicit support for any specific layer types that might require weird arguments.

    A minor note: You shouldn't have to specify the specific module the layer type is in (e.g. keras.layers.convolutional.Conv2D); just give the bare name (e.g., Conv2D). We automatically scan all the sub-modules within keras.layers (at least, all the ones we're aware of at the time of writing) to find the matching layer type.

  • kernel_regularizer, bias_regularizer, and activity_regularizer (string): We'll lump these all together because they all work the same way in principle, but they are three separate keys; you can specify all, some, or none of them for a given layer, as you desire.

    These all allow you to specify regularization functions (e.g., L1 and/or L2; for more info, see the Keras docs on regularizers and/or general documentation on L1/L2/etc. regularization) that can be applied to different aspects of the layer's functionality. Note that not all layer types support these different regularizers; see the Keras docs on regularizers and the Keras docs on the individual layer types for more details on which layers can take which regularizers.

    Regardless of which regularizer(s) you're specifying, the value should be a string that contains a small snippet of Python code. That Python code should be a function call that creates a Keras regularizer object with a given set of parameters, e.g. something like keras.regularizers.l1(0.01) or keras.regularizers.l1_l2(l1=0.01, l2=0.01) (you may want to see our sample JSON job files for examples of these snippets being used in the wild). This snippet will be executed when the layer is created and the return value will be attached to the layer's kernel, bias, or activity regularizer attribute.

    We realize that this is 100% totally gross, but unfortunately Keras requires you to attach a regularizer as an object, and this was the most straightforward way to implement that. In theory, it should also be possible for users to write their own regularization functions and attach those (see the Keras regularization docs for details), but we have not been brave enough to try it ourselves. If you really want to do that, get in touch with the devs and we'll see if we can make it work.

  • wreg: This provided another way to attach a regularizer the way regularizers were implemented in Keras 1. But we aren't particularly recommending that anyone try to use Keras 1 these days (although in most cases it should work), so although wreg is still in the code base for now, we can't officially condone its usage. In fact, forget we ever said anything. There's no wreg here, who said anything about wreg? Hey look, is that the Goodyear blimp?! [runs away]

At any rate, layers is in many ways the heart of a Keras-based DTModel specification, but it's also really hard to describe in prose without giving a full course on neural nets and Keras. So, again, we highly recommend that if you have not worked with these ideas before, you check out the sample JSON job files and compare them to the Keras documentation for the various layer types you find described in the layers list, just to get a sense of how all the layer-creating arguments work.

backend (string): Whew, after all that layers nonsense, finally we're back to an easy one. This is just a string value telling the toolbox which general backend library to use. Choices are either pymvpa or keras. If you don't specify anything, the default is keras. In the future, we might add additional backends, but nothing else is supported right now. Could we have inferred the intended backend from the other attributes and left this one out? Yeah, probably, but in general it's likely better to make things explicit, just to help people from getting confused.

One minor implementation note: We only try to import Keras or PyMVPA when the corresponding backend's functionality is used. So, if you are only ever running Keras-based analyses, you don't have to have PyMVPA installed, and vice versa.

compile_options (dictionary or None): Keras-specific. Ah, geez, Rick. Why'd you have to go and ask about compile_options? Boy, are we in trouble now!

In all seriousness, compile_options isn't too bad; it's just another thing (like layers, above) that is harder to describe in prose than it is to actually use. If you are doing a Keras-based analysis via JSON job file, you will almost certainly want to provide some compile options. See our docs on JSON job files and/or our tutorial videos for some examples that should make things clearer.

This is a catch-all attribute for stuff that gets passed along to the Keras model's compile() function (see Keras's docs for the Sequential class for details). If you're not a Keras whiz yet, basically once the neural network model is fully described, it has to be "compiled" (similar to how you'd compile code in language like C into a runnable application) before it can be used. Really, the compilation part itself is not something users typically have to worry about (except insofar as explaining why they have to wait so long before their analysis starts running sometimes); the more salient point is that the compile_options contain a bunch of general settings for an entire Keras model that are outside the settings for the individual layer. (At an even broader level are the settings contained in the backend_options attribute described further below, which for Keras analyses means meta-options about how the training is run, but not ones that need to be baked into the model itself; and at a still broader level are the settings in the DTAnalysis object, which are settings for the overall analysis that are so general, they don't even depend on whether the user is using the Keras or PyMVPA backend.)

In principle, the keys and values in compile_options are the same as the argument names and values, respectively, that get passed into Keras's Sequential.compile() function. Mostly, we just unpack these options and pass them right along to that compile() function, so again, see the Keras docs for the full scoop). However, as we do for layers, there are a few ways in which we slightly transmogrify some of the information in compile_options for the ease of the user before passing the rest along to Keras. Thus, we will now run down some of the most likely compile_options users will want to specify (not an exhaustive list, just the ones we typically use and think others would use), including any notes on options where we apply our own special Szechwan sauce. As for layers above, each bullet point is the name of a key that can exist in the compile_options dictionary, with the data type of its corresponding value (in italics and parentheses):

  • loss (string): The name of a loss function (aka objective function, aka various other things) that computes the value that Keras tries to optimize over. This is a mandatory option (please ignore the oxymoron), so you have to specify something for loss. If you don't know much about loss functions, you might want to look them up in the Keras docs or more generally on the Internet. Keras provides a bunch, so valid values include things like mean_squared_error, categorical_crossentropy, and many more (see the Keras "losses" page for all of them). In general, if you are doing a run-of-the-mill classification and not trying any funny business, you might want to stick with categorical_crossentropy since the toolbox generally assumes people will want to represent their output in a "one-hot" manner and thus converts category labels to such one-hot encoding schemes (see DTData docs for more details). This means that under normal usage, certain loss functions like sparse_categorical_crossentropy will NOT work at all. So, basically, if you provide something besides categorical_crossentropy here, hopefully you know what you're doing.

  • metrics: This one is listed in the Keras docs, but you don't actually have to specify anything for it; the only one we currently support is accuracy, so we just default to that, and anything you did provide under metrics would get overwritten.

  • optimizer (string): The name of an optimizer function that does the actual updating of the model during each round of training. In other words, basically the learning function; the optimizer considers the value given by the loss function and tries to update the weights (aka parameters) of the neural network to minimize the loss on future rounds (in other words, make the model's predictions as accurate as possible).

    In principle, the choice of an optimizer function is very important, since it's the thing that is actually making your network learn things. But VERY LOOSELY SPEAKING, in our experience several of the common optimizers Keras provides should all work reasonably well on many datasets. As always, your mileage may vary, so you may want to check out the Keras docs on optimizers or read up on optimizers in general to help inform your choice. Reasonable options might include SGD, RMSprop, or Adam... Keras provides several more and those might work perfectly well too, but those three are the ones we use most often.

    If this stuff wasn't too complicated for you already, note that each optimizer can ALSO be used either in a default configuration, or it can be deployed with several configurable options that can alter how it works (e.g., one pretty important option is the learning rate -- basically how aggressively the optimizer changes the network). These options are different for each optimizer, so if you plan to change the defaults, you will probably want to check the Keras documentation to see which options are supported by which optimizers.

    If you were using Keras directly like a common Python programmer (ew), you'd deploy those options by creating the optimizer object manually, and then passing that object along to the compile() function instead of just the name of the optimizer. But you are no commoner. You use our toolbox, and our cool JSON job file stuff. So to make it easier to specify optimizer options in a JSON file, we made this little guy in the next bullet point that we like to call:

  • optimizer_options (dictionary): Further down the rabbit-hole we go! Now we have a dictionary within a dictionary! (Which, if you're keeping track, would typically be further ensconced inside another dictionary or two, wrapped in a mystery, inside an enigma, surrounded by a pancake, smothered in maple syrup.) Seriously, we know these nested data structures get complicated, but if you follow the examples in our sample JSON job files, it's not too bad.

    Note that you won't find optimizer_options anywhere in the Keras documentation; again, it's our hacked-in way of letting you specify options for your optimizer function (described in the previous bullet). If you are OK using the optimizer with its default options, you don't have to specify optimizer_options at all.

    If you do want to create an optimizer with customized options, basically the keys and values of this optimizer_options dictionary will be the names and values of arguments taken by the initializer of your chosen optimizer function. These arguments are different for different optimizers, so back to the Keras docs you go to see which options your chosen optimizer takes! As an example, though, if you wanted to use the Adam optimizer with some custom parameters, your JSON for the whole compile_options dictionary might look like this:

    "compile_options": { "loss": "categorical_crossentropy", "optimizer": "Adam", "optimizer_options": { "lr":0.005, "beta_1":0.9, "beta_2":0.999, "epsilon": 0.1, "decay":0 } } or, equivalently in Python code: compile_options={} compile_options["loss"] = "categorical_crossentropy" compile_options["optimizer"] = "Adam" optimizer_options={} optimizer_options["lr"] = .005 optimizer_options["beta_1"] = .9 optimizer_options["beta_2"] = .999 optimizer_options["epsilon"] = .1 optimizer_options["decay"] = 0 compile_options["optimizer_options"] = optimizer_options

classifier (dictionary, or None): PyMVPA-specific. If you're using the PyMVPA backend, there's good news and bad news. The good news is that the specification won't require as much detail as Keras models, what with all their layers and such, although there are still a bunch of things you'll need to spell out. At the present moment, all of your options that are specific to a PyMVPA analysis are in this dictionary. The backend_options attribute (see below) is named as if it is used for all backends, and in future we might provide some general PyMVPA options in that, but for now backend_options is only actually used with Keras. So all your PyMVPA settings end up right here in classifier. If you're doing a PyMVPA analysis, this attribute is mandatory.

Here's the bad news: Since PyMVPA implements numerous classifiers and they all have their own quirks, trying to figure out how to specify all of their variations can be all over the place and, frankly, kind of a nightmare. To simplify things, for now, we have only implemented support for two types of PyMVPA classifiers, support vector machines (SVMs) and sparse multinomial linear regression (SMLR). If you are interested in using other types of PyMVPA classifiers, get in touch with the devs; it's relatively easy to add new types, it just requires more spaghetti code than we'd due to the convoluted way that PyMVPA constructs its classifiers, so we're adding on an as-needed basis.

Also due to PyMVPA's weird convoluted architecture, we have decided (for better or for worse) to try to abstract away some of that weirdness by simplifying their options. The result is that some of the contents of the classifier attribute get passed along straight to PyMVPA, and others we handle ourselves. This also means that not every sub-feature of a particular classifier will be supported, but we have tried to handle the most common use cases. Again, if you want to do something we can't handle yet, get in touch.

As always, an example is worth 1000 lines of documentation, so see our sample JSON job files for that.

If you want to do an SVM analysis, the keys and values of the classifier dictionary should be as follows:

  • classifier_type (string): Should be SVM to do an SVM analysis; this is what tells us what kind of classifier you're making. If that seemed too easy, hold on -- it's going to get much weirder.

  • svm_impl (string): Which of PyMVPA's several possible variations (implementations) of the basic SVM concept to use. At the moment, the only thing we formally support/endorse you to use for this is C_SVC. There are other choices (see the PyMVPA docs for mvpa2.clfs.svm.SVM for details), but you probably shouldn't try them unless you know what you're doing, and even then they aren't guaranteed to work. If you DO know what you're doing and the other options DON'T work, get in touch and we can add explicit support for the implementation you wanted to use.

  • params (dictionary): This is where we stick all the gross stuff. Technically, we do a step where anything that we don't handle ourselves gets passed along to PyMVPA to do what it wants with, and if you love living on the ragged edge of disaster you could theoretically look at the code, figure out what we do, and use that to specify additional parameters besides the ones we explicitly handle -- but we don't really suggest anyone try that. Better to get in touch with the devs if you want specific parameters supported. So, in the subsequent sub-bullets are the keys and values of the params sub-dictionary that we presently do support:

    • C (number): The main (hyper)parameter associated with SVMs. See more general resources on SVMs if you are not familiar with what this is, but very loosely speaking, larger C values try to get more examples classified correctly during training (at the expense of a narrower separation between the classes, potentially leading to overfitting), whereas smaller C values maximize the separation between the classes, potentially at the expense of underfitting during training and/or test. This should be a positive number, but it does not need to be an integer or greater than 1. Unfortunately it's hard to suggest a good default; for whatever it's worth, PyMVPA uses a default of 1, but that may or may not be a good default for your dataset. You may want to try various values in a range like .001, .01, .1, 1, 10, 100, 1000, etc. to get a sense of how different values pan out for you. We don't technically force you to specify something for this (and if you don't put anything in, you get PyMVPA's default), but you should probably put something in here.

    • kernel (string): String name of the type of kernel you want to use. If you just want a linear SVM (the most common option in many applications), you don't have to specify this, as a linear kernel is the default. These names are defined and interpreted by us, not just passed along to PyMVPA; currently possible kernel types include linear, rbf (radial basis function), polynomial, and sigmoid. There correspond to instances of the PyMVPA classes mvpa2.kernels.libsvm.LinearLSKernel, mvpa2.kernels.libsvm.RbfLSKernel, mvpa2.kernels.libsvm.PolyLSKernel, and mvpa2.kernels.libsvm.SigmoidLSKernel, respectively. If you specify anything other than linear, you can/should also specify kernel_params (see immediately below).

    • kernel_params (dictionary): Yet another level of dictionary-ness. Yeesh. The keys and values of this dictionary get passed directly into the constructor of PyMVPA's kernel class for whatever kernel option you specify in kernel. For example, if you are specifying these options in a JSON job file and your entry for the whole params dictionary looks like this: "params": {"C": 10, "kernel": "rbf", "kernel_params": {"gamma": 0.0001} }, that would be equivalent to calling mvpa2.kernels.libsvm.RbfLSKernel(gamma=.0001) in PyMVPA code.

      At the moment we have not experimented much with kernels other than linear and rbf in our own work, so we will not extensively document the panoply of options available for all the kernel types; see the PyMVPA documentation if you want to delve into that madness. (The example above is also cribbed directly from one of our sample JSON job files, so you might also want to check those out for a simple example of an SVM job that uses a non-linear kernel.)

If you want to do a SMLR analysis, the keys and values of the classifier dictionary should be as follows:

  • classifier_type (string): Should be SMLR to do a SMLR analysis; this is what tells us what kind of classifier you're making.

  • implementation (string): Which of PyMVPA's possible variations (implementations) of the basic SMLR concept to use. (Not that because the PyMVPA devs want us to suffer as much as possible, this has a different name than the equivalent option for SVMs). At the moment, the only thing we formally support/endorse you to use for this is C. Currently, the only other choice is Python (see the PyMVPA docs for mvpa2.clfs.smlr.SMLR for details), but you probably shouldn't try that unless you have issues with C, as C should work much faster than Python.

  • params (dictionary): This is where we stick all the gross stuff. (You may also want to see what we say about params above under the entry for SVMs, as we do things similarly for SVMs and SMLR.) Technically, we do a step where anything that we don't handle ourselves gets passed along to PyMVPA to do what it wants with, and if you love living on the ragged edge of disaster you could theoretically look at the code, figure out what we do, and use that to specify additional parameters besides the ones we explicitly handle -- but we don't really suggest anyone try that. Better to get in touch with the devs if you want specific parameters supported. So, in the subsequent sub-bullet is the only key/value combo of the params sub-dictionary that we presently do support explicitly:

    • lm (number): The main (hyper)parameter associated with SMLR. See more general resources on SMLR if you are not familiar with what this is, but per the PyMVPA docs, "Larger values will give rise to more sparsification." Like the C parameter of SVM, this should be a positive number, but it does not need to be an integer or greater than 1. Unfortunately it's hard to suggest a good default; for whatever it's worth, PyMVPA uses a default of .1, but that may or may not be a good default for your dataset. You may want to try various values in a range like .001, .01, .1, 1, 10, 100, 1000, etc. to get a sense of how different values pan out for you. We don't technically force you to specify something for this (and if you don't put anything in, you get PyMVPA's default), but you should probably put something in here.

      At the moment we have not experimented much with any of the other SMLR options, so experiment for yourself or get in touch with the devs if you want to do something fancier that requires specifying more than just the lambda value.

backend_options (dictionary, or None): OK! Finally back to the main level of the DTModel attributes. This attribute a set of options that are essentially passed straight along to whatever backend you're using, in theory. In practice, at the moment only the Keras backend has any options that use this parameter -- if you are using the PyMVPA backend then backend_options just gets ignored. And we have kind of renamed a bunch of them because they're used in different places within Keras that we're trying to abstract away from you, dear user, by processing them ourselves at appropriate junctures, so don't go looking for things named exactly like these options in the Keras source code. If you do want to look up details in the Keras documentation/code, we have tried to note what the underlying Keras versions are called in our docs.

We think these more-or-less global options make more sense as part of DTAnalysis than any other class, so we typically specify them there -- although our dirty little secret is that they basically just get passed along to DTModel for most of the heavy lifting. So see the DTAnalysis docs for how to actually use backend_options (and in your JSON job files or Python scripts, you should generally specify backend_options as part of DTAnalysis, not DTModel). But if you're curious about how we parse these options and send them to Keras under the hood, look at DTModel.py instead of DTAnalysis.py for the down-and-dirty.

job_file_hash (string or None): Keras-specific, in DTModel. Another attribute that is mainly set in a different class, so documentation here is minimal. Usually, the user will not have to set or access this attribute. It is currently used in two places: By DTModel, to determine the name of a tempfile that will be created during Keras analyses, and by DTOutput, to provide a default output filename if the user has not specified their own. Essentially, if the user loads a JSON job file, this gets a hashed version of the contents of that file; if no job file is provided, it gets a generic default value.

If you are doing PyMVPA analyses, you don't need to worry about job_file_hash as far as DTModel is concerned (although it still may have relevance for output files; see DTJob and DTOutput documentation). If you are doing Keras analyses: Essentially, during training, Keras occasionally outputs a partially trained version of the neural network to a temp file, and that file should ideally have a somewhat-unique name, especially if you are running multiple analyses concurrently inside the same folder, so you don't get filename collisions. Hence, for users of JSON job files, we use a hashed version of the job file contents to ensure an almost-certainly-unique name. For the Keras tempfiles, that hash value is prepended with the name delineate_checkpoint_weights_tempfile_. If analyses terminate normally, we will clean up those files automatically and you don't have to worry about them, although if an analysis crashes or is terminated manually by the user, they might hang around. They don't harm anything (and will get overwritten if you re-run the same analysis), but if they bug you, you can delete them manually. Just be sure that you only delete the tempfile if you're sure the analysis in question is not still running, or you'll probably crash the analysis.

If you are using JSON job files, you don't need to worry about this attribute at all in most cases (see DTJob docs for possible exceptions); we take care of everything for you. However, if you are writing your own scripts, you may need to make up something yourself for this yourself and give it to some subset of DTJob, DTModel, and/or DTOutput, depending on exactly what you're trying to do. (See the docs and/or code for those other classes for more info on how job_file_hash is used in them.)

For DTModel, if you don't specify anything for job_file_hash, the tempfiles get a default name of delineate_checkpoint_weights_tempfile (with nothing after that). This is fine as long as you aren't running more than one analysis concurrently in the same directory, but if you fear filename collisions, you should specify something for job_file_hash. It doesn't actually have to be a hash or anything like that; it can be any string as long as it is unique to the analysis you are running.

remove_checkpoint_weights (True or False): Keras-specific. Minor exception to the discussion about Keras tempfiles immediately above; if for some reason you don't want us to auto-delete those temporary files when the analysis finishes, you can specify False for this (default value is True). Note that depending on what you are doing, they may still get overwritten in the course of the analysis (e.g., by subsequent subjects/iterations of the analysis). So most users probably won't care what is in those tempfiles and can leave this option at the default behavior.

If you are doing PyMVPA analyses, you don't have to specify anything for remove_checkpoint_weights, as it isn't used and is silently ignored.

model (PyMVPA classifier or Keras neural network): This is the actual model at the heart of the DTModel wrapper class. Depending on whether you're using PyMVPA or Keras, it stores the object representing the classifier or the neural network, respectively.

Users don't have to worry about this attribute if using JSON job files, as everything is taken care of for them. If you're writing your own code in Python, you probably don't either; under most normal conditions, you would still set up the design of your model using the other attributes of DTModel and then let DTModel take care of creating the actual model attribute via the build() method (see below).

However, if you really want to, you can create your own Keras network or PyMVPA classifier object manually and stick it inside the model attribute. That should be a relatively rare use case, but if that's what you're doing, you should also be fine to ignore most of the other attributes that are normally used to set up the model design for the build() method.

others: Other attributes get created on-the-fly that aren't provided at initialization (or provided by the user, ever)... for now, we don't document those extensively as they aren't necessary for creating your analysis and they aren't really meant to be user-accessible in most cases, so if you are using JSON job files to create and run your analysis, you don't really have to worry about them. We may document these more extensively in the future, for people who are using this module by writing their own code. For now, we will just mention that ones you might be interested in are training_history and training_history_raw, which contain the results of the training after that is run. Our toolbox normally handles processing of the training results in DTAnalysis and DTOutput, so if you're doing anything vaguely normal you should probably look there, but if you really want the results in a raw-er format, you'll find it in those attributes. Just be forewarned that things could get a bit hairy, as what is actually in those attributes is backend-specific.



Methods

Note that most users won't need to invoke these directly if they are creating their analyses via JSON job files, but some brief descriptions are provided for the brave few considering writing their own scripts. As always, if you are considering writing your own scripts, you might want to contact the devs for inside details.


__init__( self, model_func=None, model_args=None, layers=None, backend='keras', build_now=False, compile_options=None, classifier=None, backend_options=None, job_file_hash=None, remove_checkpoint_weights=True )

(no return value) Initializer function for creating a new DTModel object. Pretty much just assigns all the object's attributes. All of the arguments are optional at this point in time, but most of them will need to get assigned one way or another before you can actually use the object. Note that build() (see below) does also get called during initialization if the build_now argument is set to True. And a little other argument checking and basic backend-specific setup (such as importing Keras, if using the Kears backend) happens here too.


build( self )

(no return value) Somewhat complicated in execution (if you want to verify, feel free to examine the code) but simple in concept: This method builds the model attribute of DTModel according to the various options specified in the other attributes. You normally would not have to call this manually even if writing your own code, as it gets called either during DTModel initialization (if you request it; see __init__() above) or right before training, if you call train() or one of the train_X() variant methods when the model attribute is still None. However, it is conceivable that you might want to use the build() method manually if you are doing something like slightly tweaking the model configuration and then re-running an analysis, in which case you could just do the tweak, call build(), and proceed from there without creating a whole new DTModel object.


train( self, train_data=None, val_data=None, train_labels=None, val_labels=None )

(no return value) Non-backend-specific method to train the model according to the training (and, potentially, validation) dataset(s) passed in. Basically just does a little checking to see which backend you're using and then passes the buck to train_keras() or train_pymvpa() accordingly, so if you are writing your own code, you might just want to use those directly... especially because what exact format the training/validation data take does depend on backend. Normally DTAnalysis takes care of all of the details for you and we generally recommend using its higher-level run_X() methods for that reason, but if you really want to use the lower-level training and/or test methods here in DTModel, read on.


train_keras( self, train_data=None, val_data=None, train_labels=None, val_labels=None )

(no return value) The model-training method specific to the Keras backend. The center of this is calling the fit() method of the Keras Sequential object that is stored in DTModel's model attribute when running Keras-based analyses. Around that is wrapped a bit of setup and bookkeeping, such as setting up the Keras callbacks for early stopping, model checkpoints, and learning rate reduction when the learning rate plateaus. Note that at this point in the development of our toolbox, we assume all Keras users will want to use these features; get in touch with the devs if you are interested in implementing options to disable them. For more on what these features entail (and what options you can choose for them), see the Keras documentation and/or our discussion of the backend_options attribute in the DTAnalysis docs. At the end of training the temp file storing the model checkpoints is removed (assuming you leave that option on; see remove_checkpoint_weights attribute above).

There is also a little legacy code in here for older versions of Keras (before version 2), but we won't say any more about that because we don't really recommend/support anyone using those old versions; that code is only in there from very early development versions of this toolbox and will be taken out without warning in the future, whenever we're sure we won't need it anymore for replicating some of our truly ancient-est work.

As noted above in the train() method documentation, even if you are writing your own Python code, it is generally easier/better to let DTAnalysis take care of all of the details for you via its higher-level run_X() methods, but if you are using this train_keras() method for your own reasons, note that all of the input arguments are mandatory and should be in the format that Keras expects (NumPy arrays for your training/validation data, in a shape that depends on the shape of the input layer of your neural network; the labels passed in should already have been converted to "one-hot" coding via the train_val_test_to_categorical() or to_categorical() methods of DTData). You'll also need to make sure that there is a valid Keras neural network model in the model attribute (or at least have configured all the other attributes of DTModel that will allow it to generate that model via the build() method, as detailed earlier in this document), and that any other relevant DTModel attributes have reasonable values in them (specifically, remove_checkpoint_weights and backend_options, if you don't just want to use the defaults for both of those).


train_pymvpa( self, train_data=None, train_labels=None ):

(no return value) The model-training method specific to the PyMVPA backend. The center of this is calling the train() method of the PyMVPA classifier object that is stored in DTModel's model attribute when running PyMVPA-based analyses. Around that is wrapped a bit of setup and bookkeeping, such as wrapping the input data in the format that PyMVPA expects (using DTData.pymvpa_wrap(), so see that documentation for specifics), and then doing some post-processing on the results of training to squeeze it into the training_history attribute in a form that more-or-less mirrors the structure of Keras's training results.

As noted above in the train() method documentation, even if you are writing your own Python code, it is generally easier/better to let DTAnalysis take care of all of the details for you via its higher-level run_X() methods, but if you are using this train_pymvpa() method for your own reasons, note that all of the input arguments are mandatory and should be in the format that the method expects (2-D trials x features NumPy array for your training data; the labels passed in, as a 1-D NumPy array, should already have been converted to "zero-based" integer coding via the slice_train_val_test() method of DTData, or manually by you). You'll also need to make sure that there is a valid PyMVPA classifier object in the model attribute (or at least have configured all the other attributes of DTModel that will allow it to generate that model via the build() method, as detailed earlier in this document).


keras_options( self )

(returns a bunch of options for Keras stuff, packaged up into a single dictionary for convenience and forward-compatibility) This method is not meant to be accessed directly by users in most cases, so we will keep things brief, but basically this is where any Keras-specific options specified in the backend_options attribute (see elsewhere in this document and the backend_options entry in DTAnalysis for details) get merged with the default values we provide for any options the user hasn't given us values for. We have generally noted what those default values are wherever the corresponding options are documented, but if you do want a one-stop shop for all of our Keras-backend defaults, look at the code for keras_options(). (In future updates, we may provide functionality for a global preferences file that will then get taken care of in this method.)


test( self, test_data, test_labels )

(returns three outputs: mean accuracy, mean loss function value, and a list/array of raw prediction scores) Non-backend-specific method to test the model on a previously held-out test dataset and return output on how it performed. Basically just does a little checking to see which backend you're using and then passes the buck to test_keras() or test_pymvpa() accordingly, so if you are writing your own code, you might just want to use those directly. Normally DTAnalysis takes care of all of the details for you and we generally recommend using its higher-level run_X() methods for that reason, but if you really want to use the lower-level training and/or test methods here in DTModel, read on.

Note that if you do use the plain test() method with a PyMVPA backend, the middle return value (loss function value) will just be a single number, -1, because that particular output value is more of a Keras thing. Hence, any code handling the output should not expect a meaningful number for the loss function value when using PyMVPA.


test_keras( self, test_data, test_labels )

(returns three outputs: mean accuracy, mean loss function value, and an array of raw prediction scores) Keras-backend-specific method to test the model on a previously held-out test dataset and return output on how it performed. Essentially just runs the model currently contained in the model attribute on whatever data/labels are passed in. May display some output to console depending on the verbosity level returned by keras_options() (which in turn relies on the backend_options attribute).

As noted above in the test() method documentation, even if you are writing your own Python code, it is generally easier/better to let DTAnalysis take care of all of the details for you via its higher-level run_X() methods, but if you are using this test_keras() method for your own reasons, note that both of the input arguments are mandatory and should be in the format that Keras expects (NumPy array for your test data, in a shape that depends on the shape of the input layer of your neural network; the labels passed in should already have been converted to "one-hot" coding via the train_val_test_to_categorical() or to_categorical() methods of DTData). You'll also need to make sure that there is a valid, trained-up Keras neural network model in the model attribute, and that any other relevant DTModel attributes have reasonable values in them (mainly backend_options, if you don't just want to use the defaults).


test_pymvpa( self, test_data, test_labels )

(returns two outputs: mean accuracy and a list/array of raw prediction scores) PyMVPA-backend-specific method to test the model on a previously held-out test dataset and return output on how it performed. Essentially just runs the model currently contained in the model attribute on whatever data/labels are passed in.

As noted above in the test() method documentation, even if you are writing your own Python code, it is generally easier/better to let DTAnalysis take care of all of the details for you via its higher-level run_X() methods, but if you are using this test_pymvpa() method for your own reasons, note that both of the input arguments are mandatory and should be in the format that the method expects (2-D trials x features NumPy array for your test data; the labels passed in, as a 1-D NumPy array, should already have been converted to "zero-based" integer coding via the slice_train_val_test() method of DTData, or manually by you). You'll also need to make sure that there is a valid, trained-up PyMVPA classifier in the model attribute.

If you're actually writing your own code, another possibly important note: Because (to paraphrase a Benjamin Franklin misquote) PyMVPA is proof that God hates us and wants us to be eternally confused, the list you get back in the second return value will vary depending on what kind of classifier you're running (SVM vs SMLR vs potential other options down the road). This is because the PyMVPA sub-functions for those classifiers return output in different forms, and we have opted not to heavily process those output values at this stage. If you're using DTOutput to spit out your results (which we highly recommend), rest assured that it will take take of that headache for you. If you really must access the raw results, see the code for this function and relevant parts of DTOutput to see how you can unpack them, or get in touch with the devs; just be aware that you're entering a world of pain, Smokey. A world of pain.


get_SVM_kernel( self, svm_params )

(returns a PyMVPA SVM kernel object) Users probably won't need to interact with this method much even if writing their own code; it is only used by the build() method to turn the stuff in the params field of the classifier attribute into a real, live object representing the SVM kernel in question. The only reason we could see you wanting to slum around in here is if you wanted to hack in support for an SVM kernel we don't currently implement, but if you are intending to do that, you might want to just get in touch with the devs instead and we can add your special snowflake kernel to the toolbox at large so everyone else can enjoy it too, thus rendering you un-special once again.



CLASS METHOD

validate_arguments( cls, args )

(returns True or False) Validates the various input arguments used to initialize a DTModel object; returns True if they are all OK and False if something is wrong with them (e.g. missing required attributes, wrong values or data types). Typically used to check the format of a JSON job file, and as such would be called by DTJob when the job file is read in (rather than a user calling this method directly).

Note that currently this method works in the laziest way possible, namely it just tries to create a temporary DTModel object with the arguments given. If that object is created successfully, then it returns True; if some kind of error occurs, it returns False. In the future, hopefully we will make this method a bit smarter so it can actually inspect the arguments and give more useful feedback on exactly what is wrong with them.

Note also that if a Python global variable named dt_debug_mode is defined and set to True, a failed validation will cause an error rather than just making this method return False. Right now dt_debug_mode does default to True, but in the future we intend to some day change this behavior to the more graceful validation failure behavior of simply giving an informative warning.



JSON job file options

Generally what you specify in a JSON job file will be some subset of the attributes listed in the Attributes section above; however, not all attributes will need to be included in a typical job file. So here is a quick recap of the attributes that would typically be a good idea to include in a job file, and what data type they should be. For details on how they behave, see the Attributes section. As always, we recommend that you check out the sample_jobfiles directory and/or our video tutorials for some usage hints.

layers (list of dictionaries): Keras-specific. Data structure defining all the layers of the neural network model. Can get a little complicated until you're used to the format. We highly recommend checking out examples for this one.

backend (string): One of pymvpa or keras. If you omit, defaults to Keras. Other backend options are possible in the future.

compile_options (dictionary): Keras-specific. Data structure defining a number of general options for the overall neural network model, outside the scope of any one layer. Like layers, can get a little complicated until you're used to the format. We also highly recommend checking out examples for this one.

classifier (dictionary): PyMVPA-specific. Data structure defining a number of general options for a PyMVPA classifier. Once more, can get a little complicated until you're used to the format, and we highly recommend checking out examples for this one.