Natural Computation Methods for Machine Learning Note 02

2020年2月4日 3804点热度 0人点赞 0条评论

Neural basics

A question, what is AI?

Artificial Intelligence: It is not the art of making computers behave as they do in the movies. It is motivated by the brain, but do not overdo it. We always regard it as a black box.  After opening this box, we can see a network of simple processing units(nodes/neurons) working in parallel. Here we should concern about the ' parallel'.  According to the 100 step rule. parallelism important, not individual speed.

Artificial neuron consists of ,

  • Equation
  • Activation function / Step function (it can be discrete/continuous)

For example, binary neuron is something like  this,


where (f) is a binary activation function, x_1,x_2, \cdots x_n are  inputs, w_1,w_2, \cdots w_n are  weights. Binary neuron may very well have more than  2 inputs.

The activation function can be,

0& \text{S<0}\
1& \text{S=0}


AND OR NAND neuron node structure figures.

  • Common properties of ANNS
  • Information is stored in the connections (as weights), not in the nodes.
  • ANN’s are trained (by modifying the weights), not programmed. [Motivate the advantages of this (e.g. first lab, Volvo's car engines)]
  • Ability to generalize, i.e. to work in situations slightly different than
  • before (without retraining).
  • Adaptivity, i.e. ability to adapt to new circumstances (by retraining).
  • Parallelism
  • Fault tolerance

Training strategies

Two example: Hebb's rule. If two nodes are active, then reinforce the connection between them. Rosenblatt's Perceptron Convergence Procedure.

Supervised learning

Learning to imitate

Examples: PCP(BackPropagation), intro lab, learning to walk by copying a teacher's gait.

Reinforcement learning


Learning to trail-and-error

Example,Q-learning. playing a game (you may learn the
rules by a teacher, but you learn to play well by playing over and over again)


Unsupervised learning (UL)

Self-organization, clustering
Examples: Hebb, recognizing similarities, topological maps


Connection strategies (architectures)

Feedforward networks (focus of this course)

Description, information flow

Applications: Classification, function approximation, perception

Training: Most often supervised using some variant of backprop (overview).

Common issues: Dimensioning, weight information

//TODO need figure

Recurrent networks

Layered networks with recurrent connections between layers

Share term memory, also used for sequential problem

LSTMs (Long Short-Term Memory), commonly used now, are also recurrent

but not layered in quite the same way

Applications: Recognizing/generating sequences of patterns. Linguistics.

Training: Supervised

Fully interconnected recurrent networks

Description, information flow

Applications: associate memories, combinatorial optimization problems

Training. Often some version of Hebb

Common issues: Convergence, capacity.


Question: Why neural networks?

Why not use statistics or some rule based expert systems?

  • ANN is a statistical method! (not "model free" though, as sometimes said)
  • Currently, neural networks outperform other methods for many applications, but they have been used for a long time for other reasons as well:
    • Speed (at least if implemented in hardware)
    • Economical reasons: Projects, interviewing experts, etc. (Example NETTALK: Three months vs several years for DecTalk). Prototyping

Dong Wang

I will work as a PhD student of TU Graz in Austria. My research interests include Embedded/Edge AI, federated learning, computer vision, and IoT.