DMG logo PMML 4.4 - Neural Network Models
PMML4.4 Menu

Home

Changes

XML Schema

Conformance

Interoperability

General Structure

Field Scope

Header

Data
Dictionary


Mining
Schema


Transformations

Statistics

Taxomony

Targets

Output

Functions

Built-in Functions

Model Verification

Model Explanation

Multiple Models

Anomaly Detection
Models


Association Rules

Baseline Models

Bayesian Network

Cluster
Models


Gaussian
Process


General
Regression


k-Nearest
Neighbors


Naive
Bayes


Neural
Network


Regression

Ruleset

Scorecard

Sequences

Text Models

Time Series

Trees

Vector Machine

PMML 4.4 - Neural Network Models

Neural Network Models for Backpropagation

The description of neural network models assumes that the reader has a general knowledge of artificial neural network technology. A neural network has one or more input nodes and one or more neurons. Some neurons' outputs are the output of the network. The network is defined by the neurons and their connections, aka weights. All neurons are organized into layers; the sequence of layers defines the order in which the activations are computed. All output activations for neurons in some layer L are evaluated before computation proceeds to the next layer L+1. Note that this allows for recurrent networks where outputs of neurons in layer L+i can be used as input in layer L where L+i > L. The model does not define a specific evaluation order for neurons within a layer.

Each neuron receives one or more input values, each coming via a network connection, and sends only one output value. All incoming connections for a certain neuron are contained in the corresponding Neuron element. Each connection Con of the element Neuron stores the ID of a node it comes from and the weight. A bias weight coefficient or a width of a radial basis function unit may be stored as an attribute of Neuron element.

All neurons in the network are assumed to have the same (default) activation function, although each individual layer may have its own activation and threshold that override the default. Given a fixed neuron j, and Wi representing the weight on the connection from neuron i, the activation for neuron j is computed using up to three steps as follows

  1. Compute a linear combination or euclidean distance using input activations and weights Wi. The input activations to the current neuron are the outputs of the connected neurons.
    Z = see below
  2. The activation function is applied to the result of step 1:
    output(j) = activation( Z )
  3. A normalization method softmax ( pj = exp(yj) / Sumi(exp(yi) ) ) or simplemax ( pj = yj / Sumi(yi) ) can be applied to the computed activation values. The attribute normalizationMethod is defined for the network with default value none ( pj = yj ), but can be specified for each layer as well. Softmax normalization is most often applied to the output layer of a classification network to get the probabilities of all answers. Simplemax normalization is often applied to the hidden layer consisting of elements with radial basis activation function to get a "normalized RBF" activation.

There are two groups of activation functions.

  1. Group 1 uses a linear combination of weights and input activations.
    Z = Sum( Wi * output(i) ) + bias
    Activation functions are:
    threshold:
    activation(Z) = 1 if Z > threshold else 0
    logistic:
    activation(Z) = 1 / (1 + exp(-Z))
    tanh:
    activation(Z) = (1-exp(-2Z)/(1+exp(-2Z))
    identity:
    activation(Z) = Z
    exponential:
    activation(Z) = exp(Z)
    reciprocal:
    activation(Z) = 1/Z
    square:
    activation(Z) = Z*Z
    Gauss:
    activation(Z) = exp(-(Z*Z))
    sine:
    activation(Z) = sin(Z)
    cosine:
    activation(Z) = cos(Z)
    Elliott:
    activation(Z) = Z/(1+|Z|)
    arctan:
    activation(Z) = 2 * arctan(Z)/Pi
    rectifier:
    activation(Z) = max(0,Z)

  2. Group 2 computes a euclidean distance between weights and input activations (= outputs of other neurons)
    Z = (Sumi (output(i)-Wi)2 )/(2*width2)
    where the sum is taken over all input units, Wi are the coordinates of the center stored in Con elements in place of the weights, width is a positive number describing the width for the radial basis function unit stored either in Neuron element or in NeuralLayer or even in NeuralNetwork.
    The only activation function in this group is 'radialBasis'.
    radialBasis:
    activation = exp( f * log(altitude) - Z )

    where f is the fan-in of each unit in the layer, that is the number of other units feeding into that unit, excluding bias, and the altitude is a positive number stored in Neuron or NeuralLayer or NeuralNetwork. The default is altitude="1.0", for that value the activation function reduces to the simple exp(-Z).

XSD

<xs:element name="NeuralNetwork">
  <xs:complexType>
    <xs:sequence>
      <xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
      <xs:element ref="MiningSchema"/>
      <xs:element ref="Output" minOccurs="0"/>
      <xs:element ref="ModelStats" minOccurs="0"/>
      <xs:element ref="ModelExplanation" minOccurs="0"/>
      <xs:element ref="Targets" minOccurs="0"/>
      <xs:element ref="LocalTransformations" minOccurs="0"/>
      <xs:element ref="NeuralInputs"/>
      <xs:element maxOccurs="unbounded" ref="NeuralLayer"/>
      <xs:element minOccurs="0" ref="NeuralOutputs"/>
      <xs:element ref="ModelVerification" minOccurs="0"/>
      <xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
    <xs:attribute name="modelName" type="xs:string"/>
    <xs:attribute name="functionName" type="MINING-FUNCTION" use="required"/>
    <xs:attribute name="algorithmName" type="xs:string"/>
    <xs:attribute name="activationFunction" type="ACTIVATION-FUNCTION" use="required"/>
    <xs:attribute name="normalizationMethod" type="NN-NORMALIZATION-METHOD" default="none"/>
    <xs:attribute name="threshold" type="REAL-NUMBER" default="0"/>
    <xs:attribute name="width" type="REAL-NUMBER"/>
    <xs:attribute name="altitude" type="REAL-NUMBER" default="1.0"/>
    <xs:attribute name="numberOfLayers" type="xs:nonNegativeInteger"/>
    <xs:attribute name="isScorable" type="xs:boolean" default="true"/>
  </xs:complexType>
</xs:element>

<xs:element name="NeuralInputs">
  <xs:complexType>
    <xs:sequence>
      <xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
      <xs:element maxOccurs="unbounded" ref="NeuralInput"/>
    </xs:sequence>
    <xs:attribute name="numberOfInputs" type="xs:nonNegativeInteger"/>
  </xs:complexType>
</xs:element>

<xs:element name="NeuralLayer">
  <xs:complexType>
    <xs:sequence>
      <xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
      <xs:element maxOccurs="unbounded" ref="Neuron"/>
    </xs:sequence>
    <xs:attribute name="numberOfNeurons" type="xs:nonNegativeInteger"/>
    <xs:attribute name="activationFunction" type="ACTIVATION-FUNCTION"/>
    <xs:attribute name="threshold" type="REAL-NUMBER"/>
    <xs:attribute name="width" type="REAL-NUMBER"/>
    <xs:attribute name="altitude" type="REAL-NUMBER"/>
    <xs:attribute name="normalizationMethod" type="NN-NORMALIZATION-METHOD"/>
  </xs:complexType>
</xs:element>

<xs:element name="NeuralOutputs">
  <xs:complexType>
    <xs:sequence>
      <xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
      <xs:element maxOccurs="unbounded" ref="NeuralOutput"/>
    </xs:sequence>
    <xs:attribute name="numberOfOutputs" type="xs:nonNegativeInteger"/>
  </xs:complexType>
</xs:element>

The isScorable attribute indicates whether the model is valid for scoring. If this attribute is true or if it is missing, then the model should be processed normally. However, if the attribute is false, then the model producer has indicated that this model is intended for information purposes only and should not be used to generate results. In order to be valid PMML, all required elements and attributes must be present, even for non-scoring models. For more details, see General Structure.

NeuralInput defines how input fields are normalized so that the values can be processed in the neural network. For example, string values must be encoded as numeric values.

NeuralOutput defines how the output of the neural network must be interpreted.

<xs:simpleType name="ACTIVATION-FUNCTION">
  <xs:restriction base="xs:string">
    <xs:enumeration value="threshold"/>
    <xs:enumeration value="logistic"/>
    <xs:enumeration value="tanh"/>
    <xs:enumeration value="identity"/>
    <xs:enumeration value="exponential"/>
    <xs:enumeration value="reciprocal"/>
    <xs:enumeration value="square"/>
    <xs:enumeration value="Gauss"/>
    <xs:enumeration value="sine"/>
    <xs:enumeration value="cosine"/>
    <xs:enumeration value="Elliott"/>
    <xs:enumeration value="arctan"/>
    <xs:enumeration value="rectifier"/>
    <xs:enumeration value="radialBasis"/>
  </xs:restriction>
</xs:simpleType>

<xs:simpleType name="NN-NORMALIZATION-METHOD">
  <xs:restriction base="xs:string">
    <xs:enumeration value="none"/>
    <xs:enumeration value="simplemax"/>
    <xs:enumeration value="softmax"/>
  </xs:restriction>
</xs:simpleType>

<xs:simpleType name="NN-NEURON-ID">
  <xs:restriction base="xs:string"/>
</xs:simpleType>

<xs:simpleType name="NN-NEURON-IDREF">
  <xs:restriction base="xs:string"/>
</xs:simpleType>

NN-NEURON-ID is just a string which identifies a neuron. The string is not necessarily an XML ID because a PMML document may contain multiple network models where neurons in different models can have the same identifier. Within a model, though, all neurons (elements of NeuralInput and Neuron) must have a unique identifier.

Neural Network Input Neurons

An input neuron represents the normalized value for an input field. A numeric input field is usually mapped to a single input neuron while a categorical input field is usually mapped to a set of input neurons using some fan-out function. The normalization is defined using the elements NormContinuous and NormDiscrete defined in the Transformation Dictionary. The element DerivedField is the general container for these transformations.

<xs:element name="NeuralInput">
  <xs:complexType>
    <xs:sequence>
      <xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
      <xs:element ref="DerivedField"/>
    </xs:sequence>
    <xs:attribute name="id" type="NN-NEURON-ID" use="required"/>
  </xs:complexType>
</xs:element>

Restrictions: A numeric input field must not appear more than once in the input layer. Similarly, a pair of categorical input field together with an input value must not appear more than once in the input layer.

Neural Network Neurons

<xs:element name="Neuron">
  <xs:complexType>
    <xs:sequence>
      <xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
      <xs:element maxOccurs="unbounded" ref="Con"/>
    </xs:sequence>
    <xs:attribute name="id" type="NN-NEURON-ID" use="required"/>
    <xs:attribute name="bias" type="REAL-NUMBER"/>
    <xs:attribute name="width" type="REAL-NUMBER"/>
    <xs:attribute name="altitude" type="REAL-NUMBER"/>
  </xs:complexType>
</xs:element>

Neuron contains an identifier id which must be unique in all layers. The attribute bias implicitly defines a connection to a bias unit where the unit's value is 1.0 and the weight is the value of bias. The activation function and normalization method for Neuron can be defined in NeuralLayer. If either one is not defined for the layer then the default one specified for NeuralNetwork applies. If the activation function is radialBasis, the attribute width must be specified either in Neuron, NeuralLayer or NeuralNetwork. Again, width specified in Neuron will override a respective value from NeuralLayer, and in turn will override a value given in NeuralNetwork.

Weighted connections between neural net nodes are represented by Con elements.

<xs:element name="Con">
  <xs:complexType>
    <xs:sequence>
      <xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
    <xs:attribute name="from" type="NN-NEURON-IDREF" use="required"/>
    <xs:attribute name="weight" type="REAL-NUMBER" use="required"/>
  </xs:complexType>
</xs:element>

Con elements are always part of a Neuron. They define the connections coming into that parent element. The neuron identified by from may be part of any layer.

NN-NEURON-IDs of all nodes must be unique across the combined set of NeuralInput and Neuron nodes. The from attributes of connections and NeuralOutputs refer to these identifiers.

Neural Network Output Neurons

In parallel to input neurons, there are output neurons which are connected to input fields via some normalization. While the activation of an input neuron is defined by the value of the corresponding input field, the activation of an output neuron is computed by the activation function. Therefore, an output neuron is defined by a Neuron. In networks with supervised learning the computed activation of the output neurons is compared with the normalized values of the corresponding target fields; these values are often called teach values. The difference between the neuron's activation and the normalized target field determines the prediction error. For scoring the normalization for the target field is used to denormalize the predicted value in the output neuron. Therefore, each instance of Neuron which represent an output neuron, is additionally connected to a normalized field. Note that the scoring procedure must apply the inverse of the normalization in order to map the neuron activation to a value in the original domain.

Connect a neuron's output to the output of the network.

<xs:element name="NeuralOutput">
  <xs:complexType>
    <xs:sequence>
      <xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
      <xs:element ref="DerivedField"/>
    </xs:sequence>
    <xs:attribute name="outputNeuron" type="NN-NEURON-IDREF" use="required"/>
  </xs:complexType>
</xs:element>

For neural value prediction with back propagation, the output layer contains a single neuron, this is denormalized giving the predicted value.

For neural classification with backpropagation, the output layer contains one or more neurons. The neuron with maximal activation determines the predicted class label. If there is no unique neuron with maximal activation then the predicted value is the first output neuron with maximal activation.

Example model

<PMML xmlns="https://www.dmg.org/PMML-4_4" version="4.4">
  <Header copyright="DMG.org"/>
  <DataDictionary numberOfFields="5">
    <DataField name="gender" optype="categorical" dataType="string">
      <Value value="  female"/>
      <Value value="    male"/>
    </DataField>
    <DataField name="no of claims" optype="categorical" dataType="string">
      <Value value="       0"/>
      <Value value="       1"/>
      <Value value="       3"/>
      <Value value="     &amp;gt; 3"/>
      <Value value="       2"/>
    </DataField>
    <DataField name="domicile" optype="categorical" dataType="string">
      <Value value="suburban"/>
      <Value value="   urban"/>
      <Value value="   rural"/>
    </DataField>
    <DataField name="age of car" optype="continuous" dataType="double"/>
    <DataField name="amount of claims" optype="continuous" dataType="integer"/>
  </DataDictionary>
  <NeuralNetwork modelName="Neural Insurance" functionName="regression" activationFunction="logistic" numberOfLayers="2">
    <MiningSchema>
      <MiningField name="gender"/>
      <MiningField name="no of claims"/>
      <MiningField name="domicile"/>
      <MiningField name="age of car"/>
      <MiningField name="amount of claims" usageType="target"/>
    </MiningSchema>
    <NeuralInputs numberOfInputs="10">
      <NeuralInput id="0">
        <DerivedField optype="continuous" dataType="double">
          <NormContinuous field="age of car">
            <LinearNorm orig="0.01" norm="0"/>
            <LinearNorm orig="3.07897" norm="0.5"/>
            <LinearNorm orig="11.44" norm="1"/>
          </NormContinuous>
        </DerivedField>
      </NeuralInput>
      <NeuralInput id="1">
        <DerivedField optype="continuous" dataType="double">
          <NormDiscrete field="gender" value="    male"/>
        </DerivedField>
      </NeuralInput>
      <NeuralInput id="2">
        <DerivedField optype="continuous" dataType="double">
          <NormDiscrete field="no of claims" value="       0"/>
        </DerivedField>
      </NeuralInput>
      <NeuralInput id="3">
        <DerivedField optype="continuous" dataType="double">
          <NormDiscrete field="no of claims" value="       1"/>
        </DerivedField>
      </NeuralInput>
      <NeuralInput id="4">
        <DerivedField optype="continuous" dataType="double">
          <NormDiscrete field="no of claims" value="       3"/>
        </DerivedField>
      </NeuralInput>
      <NeuralInput id="5">
        <DerivedField optype="continuous" dataType="double">
          <NormDiscrete field="no of claims" value="     &amp;gt; 3"/>
        </DerivedField>
      </NeuralInput>
      <NeuralInput id="6">
        <DerivedField optype="continuous" dataType="double">
          <NormDiscrete field="no of claims" value="       2"/>
        </DerivedField>
      </NeuralInput>
      <NeuralInput id="7">
        <DerivedField optype="continuous" dataType="double">
          <NormDiscrete field="domicile" value="suburban"/>
        </DerivedField>
      </NeuralInput>
      <NeuralInput id="8">
        <DerivedField optype="continuous" dataType="double">
          <NormDiscrete field="domicile" value="   urban"/>
        </DerivedField>
      </NeuralInput>
      <NeuralInput id="9">
        <DerivedField optype="continuous" dataType="double">
          <NormDiscrete field="domicile" value="   rural"/>
        </DerivedField>
      </NeuralInput>
    </NeuralInputs>
    <NeuralLayer numberOfNeurons="3">
      <Neuron id="10">
        <Con from="0" weight="-2.08148"/>
        <Con from="1" weight="3.69657"/>
        <Con from="2" weight="-1.89986"/>
        <Con from="3" weight="5.61779"/>
        <Con from="4" weight="0.427558"/>
        <Con from="5" weight="-1.25971"/>
        <Con from="6" weight="-6.55549"/>
        <Con from="7" weight="-4.62773"/>
        <Con from="8" weight="1.97525"/>
        <Con from="9" weight="-1.0962"/>
      </Neuron>
      <Neuron id="11">
        <Con from="0" weight="-0.698997"/>
        <Con from="1" weight="-3.54943"/>
        <Con from="2" weight="-3.29632"/>
        <Con from="3" weight="-1.20931"/>
        <Con from="4" weight="1.00497"/>
        <Con from="5" weight="0.033502"/>
        <Con from="6" weight="1.12016"/>
        <Con from="7" weight="0.523197"/>
        <Con from="8" weight="-2.96135"/>
        <Con from="9" weight="-0.398626"/>
      </Neuron>
      <Neuron id="12">
        <Con from="0" weight="0.904057"/>
        <Con from="1" weight="1.75084"/>
        <Con from="2" weight="2.51658"/>
        <Con from="3" weight="-0.151895"/>
        <Con from="4" weight="-2.88008"/>
        <Con from="5" weight="0.920063"/>
        <Con from="6" weight="-3.30742"/>
        <Con from="7" weight="-1.72251"/>
        <Con from="8" weight="-1.13156"/>
        <Con from="9" weight="-0.758563"/>
      </Neuron>
    </NeuralLayer>
    <NeuralLayer numberOfNeurons="1">
      <Neuron id="13">
        <Con from="10" weight="0.76617"/>
        <Con from="11" weight="-1.5065"/>
        <Con from="12" weight="0.999797"/>
      </Neuron>
    </NeuralLayer>
    <NeuralOutputs numberOfOutputs="1">
      <NeuralOutput outputNeuron="13">
        <DerivedField optype="continuous" dataType="double">
          <NormContinuous field="amount of claims">
            <LinearNorm orig="0" norm="0.1"/>
            <LinearNorm orig="1291.68" norm="0.5"/>
            <LinearNorm orig="5327.26" norm="0.9"/>
          </NormContinuous>
        </DerivedField>
      </NeuralOutput>
    </NeuralOutputs>
  </NeuralNetwork>
</PMML>
e-mail info at dmg.org