Rabbit hole

The DORA Quick Check is a streamlined assessment tool designed to measure your team’s software delivery performance. It focuses on four key metrics:

1. Lead Time for Changes: The time it takes to move from code being committed to successfully running in production.
2. Deployment Frequency: How often your organization deploys code to production or releases it to end users.
3. Change Failure Rate: The percentage of changes to production or user releases that result in service degradation, such as impairments or outages.
4. Time to Restore Service: The average time required to recover and restore service after a change leads to service degradation or outage.

Explore more about it here: DORA Quick Check.

A first try for usual automotive cases:

A common pattern for solution architecture are different layer (aka tiers) e.g, frontend, backend and database. The common approaches such as micro-service and self contained (aka CSC- self contained system) are just different kinds of commonly used layer.

1. Micro-service: architecture usually share a common frontend layer – usually the load balancer – which distributes in the independent lanes of services.
2. CSC: Nothing is shared, the client has to manage the different services lanes by its own.

Micro-service architecture is a contemporary software design approach widely adopted by companies such as Netflix and Amazon, allowing them to release features at an impressive pace. This approach decomposes a large application into a collection of small, loosely coupled services. Each service operates independently, with its own process, and communicates with others through well-defined APIs. This separation enables each service to maintain its own codebase, which can be managed by small, focused teams and deployed independently.

For Software-defined Vehicles (SdVs), this architecture might seem ideal, but some common architecture building blocks (ABBs) face scrutiny in modern electrical/electronic (EE) architectures.

• For instance, a load balancer is a fundamental component in micro-service architecture that ensures incoming network traffic is evenly distributed across servers to prevent overload and maintain performance. In automotive EE architectures, zones might distribute requests to the appropriate ECUs, but this raises questions—why replicate such functionality when a single ECU usually serves a specific function?
• Another critical element is the Content Delivery Network (CDN), a distributed server system that delivers content closer to the end-user, reducing latency and improving user experience. In automotive systems, sensors and sensor fusion dominate, with less static content compared to traditional web systems. Machine learning models for ADAS/AD could be considered analogous to static content, but they are more aptly classified as code rather than content.
• The API gateway functions as a bridge between clients and micro-services, routing requests to the appropriate service. In automotive systems, domain frameworks increasingly assume this role, often aligning with centralized vehicle architectures. Effective tools for monitoring and managing micro-services are also essential to ensure smooth operation and communication, a function similarly supported by domain frameworks in automotive contexts.

The advantages of micro-service architecture are notable. Scalability stands out, as individual services can be scaled independently without impacting the rest of the system. This independence also facilitates agile development and deployment processes, with smaller, isolated codebases enabling faster feature rollouts. The architecture’s resilience is another benefit—if one service fails, the rest of the system remains unaffected.

However, this approach is not without challenges. The increased number of services introduces significant complexity, complicating management and maintenance efforts. Ensuring data consistency across services becomes a challenge since each service manages its own data. Inter-service communication may lead to network latency, potentially affecting performance. Debugging and error handling also become more difficult, as identifying root causes in a distributed system requires considerable effort and coordination.

While micro-service architecture offers clear benefits in cloud-based environments, its adoption in in-vehicle contexts introduces complexities that may outweigh its advantages in terms of flexibility, scalability, and resilience.

My takeaways from the book “Software Engineering at Google: Lessons Learned from Programming Over Time” can be grouped into cultural, organizational, and technical insights. Open communication and psychological safety are emphasized as critical components of a successful engineering culture. These principles foster an environment where team members feel comfortable sharing ideas and concerns, leading to better collaboration and innovation. The book also highlights the importance of documentation and knowledge sharing, which not only enhance productivity but also reduce the risk of teams becoming overly reliant on specific individuals.

On the technical side, the automation of tests and code reviews is deemed essential for identifying errors early and maintaining high code quality. Code reviews, as facilitated by tools like Google’s Critique system, play a central role in ensuring quality, promoting knowledge sharing, and enhancing collaboration among engineers. A noteworthy point here is that Google avoids mandatory pull requests (PRs) and instead encourages engineers to voluntarily submit their changes for review, creating a culture of trust and ownership.

In terms of development practices, the use of a monorepo and a robust build system like Bazel offers significant advantages in terms of efficiency, consistency, and scalability. The monorepo structure simplifies dependency management and ensures uniformity across projects. Additionally, tools like “Code Search” streamline navigation and searching within large codebases, enhancing developer efficiency and collaboration across teams. These practices and tools collectively enable Google’s engineers to tackle the complexities of large-scale software development effectively.

In a nutshell:

• Trunk based development in a mono report with active code review sharing instead of PRs
• Documentation and making the own code accessibl and searchable

As a technology manager, navigating new solution spaces to improve technology adoption, mitigate risks, and discover opportunities can be a complex task. The TRIZ (Theory of Inventive Problem Solving) methodology provides a structured approach to evaluate existing technological solutions from a fresh perspective. Key TRIZ-inspired patterns for technology management:

• (Un-)Structure: Organize or simplify solutions through stacking, mediating, segmenting (splitting), reducing (pruning) or merging.
• (In-)Balance: Create or disrupt balance by employing asymmetry (localization), symmetry , opposites, reversals or mediation.
• Abstract: Adapt solutions through generalization, specialization, preceding actions or post-processing steps.
• Value Engineering: Optimize solutions by preventing issues, reducing complexity and distinguishing between critical (primary) and non-critical (secondary) components.

Traditional neural network design relies heavily on gradient flow, which refers to the movement of gradients during the training process. A gradient is a value (vector or scalar) that represents the direction and magnitude of the steepest increase of a function, helping to indicate how the function changes between consecutive points. The cumulative movement of these gradients across all layers in a neural network is referred to as gradient flow.

Each layer in a neural network is essentially a linear transformation derived from the gradients, adhering to the universal approximation theorem (UAT). According to the UAT, a neural network with a single hidden layer containing a finite number of neurons can approximate any continuous function to a reasonable degree of accuracy within a compact subset, provided suitable activation functions are used. The non-linearity required to model complex patterns is introduced via activation functions, as stacking linear transformations alone would be insufficient for learning intricate relationships in data.

The Kolmogorov-Arnold Network (KAN) fundamentally rethinks this traditional approach. Instead of relying on fixed activation functions, as in traditional neural networks, KAN leverages the Kolmogorov-Arnold theorem, which provides a way to exactly represent continuous functions as sums of univariate functions. This approach allows KAN to be more accurate and interpretable while requiring fewer parameters. Unlike fixed activation functions, KAN introduces learnable activation functions. By making the activation function dynamic, the model can adapt better to the data it is trained on.

KAN achieves this flexibility by using B-Splines to train its activation functions. B-Splines are efficient representations of curves, using a series of connected lower-degree polynomial segments rather than a single high-degree polynomial. This approach ensures smooth transitions and reduces the risk of overfitting. The positions of the B-Spline control points are made learnable, allowing the model to adjust the shape of the activation functions dynamically during training. This enables KAN to develop arbitrary, data-driven activation functions that best fit the problem at hand.

One of the key strengths of KAN is its capacity for continual learning. However, a notable drawback is its slower training speed. Research shows that KANs can be up to 10 times slower to train compared to multilayer perceptrons (MLPs) with the same number of parameters. Despite this, KAN often outperforms MLPs in terms of overall efficiency, as it typically requires fewer parameters to achieve similar or better performance.

In summary, KAN’s innovative use of learnable activation functions via B-Splines and its foundation in the Kolmogorov-Arnold theorem make it a powerful, flexible, and interpretable alternative to traditional neural network architectures, albeit with the trade-off of slower training.

Nice video:

Gradient flow describes the movement of gradients during the training of a machine learning model. A gradient is a value (vector or scalar) that shows the direction and magnitude of the steepest increase of a function, indicating how a function changes between consecutive points. The cumulative movement of these gradients across all steps is referred to as gradient flow.

This concept is important not only for learning but also for optimizing neural networks after training. Many neurons in a trained network may contribute very little to its performance while still consuming memory. This can be determined by looking at their activations. In neural networks, the activation of a neuron is its output value after applying an activation function to its weighted input. For a simple linear component, this is the result of the equation mx+b.

Neurons with consistently low activations across multiple test inferences have minimal impact on the network’s output. These neurons can be safely pruned to reduce the network’s size and improve efficiency without significantly affecting its performance.

Different approach to achieve parallelism in training deep neural networks:

1. Split layer-wise – introduce a bottleneck because the data flow through the layers must be finally ensured
2. Split neuron-wise – often splits based on the capability of the matrix operation to get decomposed
3. Split training data-wise – the later merge of gradient updates may vanish the training results.

Any better idea?

In gradient descent, each parameter update is based solely on the current gradient, which can lead to undesirable oscillations during the optimization process. Similar to control tasks, where a filter is often required, a momentum mechanism is introduced. Momentum acts as a filter by incorporating a moving average of past gradients during the learning process.

A widely used control algorithm in industrial applications for smoothing input (sensor) data is the Proportional-Integral-Derivative (PID) control. The PID algorithm, as its name implies, is based on three core components: proportional, integral, and derivative. These coefficients are adjusted to achieve an optimal response, effectively reducing fluctuations and ensuring smoother adjustments.

Gradient flow refers to the movement of gradients during the training of a machine learning model. A gradient is a vector or scalar that represents the direction and magnitude of the steepest increase of a function. Mathematically, it quantifies the change in a function between consecutive points. The cumulative movement of gradients across all steps is referred to as the flow of the gradient.

In machine learning, gradients are essential for optimizing a model’s parameters to minimize errors. They play a central role in algorithms like gradient descent, where the model’s weights and biases are iteratively updated based on the gradient flow. This process adjusts the function and its components to improve model performance.

The concept of a gradient relies on the differentiability of a function. Differentiability ensures that gradients are well-defined and that the function is smooth, without sharp corners or discontinuities in the region of interest. These properties are critical for computing gradients effectively, forming the mathematical basis for training neural networks. Differentiable functions enable the calculation of gradients, allowing neural networks to adjust their parameters and improve performance. Without differentiability, the learning process would be hindered due to the inability to compute reliable gradients.

Simply put, gradient flow is vital for a neural network to learn. If gradients flow, the network can adjust and improve; without it, learning cannot occur. Ensuring proper gradient flow is essential for training neural networks effectively. A network’s ability to (re)learn determines whether it can be trained successfully.

Neural network training commonly involves four primary methodologies, typically applied when the network is split horizontally:

1. Transfer learning: The left section of the network remains stable (retrained part), with no gradient flow, while the right section (newly added neurons) is updated and continuously learned.
2. Fine-tuning: Both the left and right sections are further updated and learned.
3. Multitask learning: Similar to fine-tuning, but the right section is split into task-specific branches for multiple objectives.
4. Federated learning: Updates are collected across applications and centrally applied to the left section, while the right section undergoes local fine-tuning, often referred to as “personalization.”