Generative AI
Generative AI operates differently from perception-based approaches. While traditional neural networks classify existing data, such as categorizing images of cats and birds, generative models understand the underlying structure of a given dataset and are capable of recreating that structure.
Transformer Neural Networks
A typical transformer network consists of encoders followed by decoders like e.g., like an auto encoder use for filtering e.g, to denoise. From neural network perspective also transformer bring nothing new. The value add is introduced by the so called “attention” mechanism, what realize a kind of lookup table and not via embedding. The lookup table has a query that maps to a key resulting in a target value. The approach of self attention compare to given input (query) with itself (key) e.g., to lookup references in a given input text like “The cat has two eyes”. The self-attention reflects the link between eyes and the cat.
Good go know
Embedding is a technique used to create input information in a lower-dimensional space, typically represented as a dense vector, such as one-hot encoding.
Advantages of embedding include:
- Dimensionality Reduction: Embeddings represent high-dimensional data in a lower-dimensional space, reducing computational complexity and enabling efficient data processing.
- Semantic Relationships: Embeddings capture semantic relationships between words, enabling algorithms to understand and reason about word meanings based on their proximity in the embedding space.
- Generalization: Embeddings generalize from known examples to unseen examples by representing similar words or texts with similar embeddings. This allows algorithms to make predictions or perform tasks on new, unseen data.
- Compatibility with Machine Learning Models: Embeddings provide a numerical representation that can be easily utilized by machine learning models. This allows models to leverage the learned representations for downstream tasks such as sentiment analysis, document classification, machine translation, and more.
Convolution (CNN) is a crucial step in extracting important features from a given tensor. It is performed prior to the neural network’s processing. Convolutional Neural Networks (CNNs) excel in automatically learning hierarchical representations of data. The process involves using trainable filters, known as kernels or convolutional filters, to scan input data in a sliding window manner. These filters convolve across the input, performing element-wise multiplications and summations, resulting in a feature map. The convolution operation enables the network to capture local patterns and spatial relationships between adjacent pixels or data points. However, it does not capture the interrelationships between patterns across the entire input tensor. For that purpose, the Capsule Network provides support.