Large Context Model

In modern large language models (LLMs), tokenizers break down a given input (i.e., a prompt) into smaller units called tokens. For example, the sentence The OEMs vehicle features are defined by software” would be split into tokens – e.g., every word a token . The LLM processes these tokens and generates a response based on the same tokenized vocabulary. However, this approach differs from how humans analyze information. Humans apply multiple levels of abstraction, such as visual representations like schematic diagrams.

This concept can be likened to creating a presentation slide deck, where visuals are often used instead of dense blocks of text. These layers of abstraction can be referred to as concepts. According to the paper Large Concept Models: Language Modeling in a Sentence Representation Space, concepts represent ideas or actions that are not tied to a specific language or modality. The paper provides an illustrative example of this idea.

Source: 2412.08821v2

One key advantage of this approach is improved handling of long-context information. Since large amounts of text are compressed into a smaller set of concepts, it enhances efficiency. Additionally, the use of concepts enables hierarchical reasoning, which is particularly useful in image-processing tasks where relationships between elements must be understood at different levels.

Concepts can be viewed as a form of compression, where words (or word vectors) are mapped into a concept space through dimensionality reduction. This transformation can also be achieved using neural networks, leading to what is known as neural composition (though this topic is beyond the scope of this discussion).

Now, let’s consider inference. Similar to traditional LLMs, where a sequence of tokens predicts the next token, in this approach, the sequence predicts the next concept instead of a token. The paper illustrates this with diagrams, further expanding on techniques such as diffusion (de-noising) and multi-tower (seperaton-of-concerns) architectures.

Source: 2412.08821v2

Posted in Uncategorized | Leave a comment

The Role of Multi-Paradigm Programming Languages in Modern Software Development

Introduction

The landscape of software development has undergone significant transformations over the years. While earlier programming languages were primarily designed around a single paradigm, modern software increasingly demands flexibility and adaptability. Multi-paradigm programming languages have emerged as a crucial solution, offering developers the ability to blend different styles—object-oriented, functional, procedural, and more—to build efficient, scalable, and maintainable applications.

Benefits of Multi-Paradigm Programming

Versatility in Problem-Solving

Different programming paradigms cater to different problem domains.

  • Object-oriented programming (OOP) is well-suited for modular applications.
  • Functional programming is ideal for handling immutable data and concurrency.
  • Procedural programming allows structured execution of logic.

Languages such as:

  • Rust integrate low-level system programming with functional and object-oriented constructs (in weighted order: functional, OOP, procedural).
  • JavaScript enables a mix of procedural, OOP, and functional programming for web applications (in weighted order: !event-driven!, functional, OOP).
  • Python combines procedural simplicity with OOP structure and functional capabilities (in weighted order: procedural, OOP, functional).

This flexibility enables developers to apply the most appropriate paradigm for the given context.

Improved Efficiency and Performance

By leveraging multiple paradigms, developers can optimize efficiency.

  • Functional programming, for instance, enhances parallel processing and reduces side effects,
  • while object-oriented (OOP) principles structure large codebases for clarity and reuse.

This combination reduces redundancy and boosts maintainability.

Scalability and Maintainability

Multi-paradigm programming allows for modular and adaptable software architectures.

  • Functional programming reduces unwanted dependencies,
  • OOP encapsulates complex logic and
  • procedural paradigms enforce clear execution paths.

These factors lead to better testability, easier debugging, and improved scalability.

Enhanced Collaboration Across Teams

Teams with diverse expertise can benefit from multi-paradigm languages, as different developers can work within their preferred styles while maintaining a cohesive codebase. This fosters better collaboration and reduces the learning curve for newcomers.

Alignment with Modern Hardware Trends

Modern hardware architectures emphasize multi-core processing. Functional programming aids parallel computation by eliminating shared states, while procedural and object-oriented approaches organize system logic efficiently. Combining these paradigms leads to better utilization of computing resources.

Multi-Paradigm Programming and Design Patterns

Design patterns are essential for software architecture. Multi-paradigm programming enables developers to integrate different design patterns effectively, such as:

  • OOP Patterns: Singleton, Factory, Observer, Strategy
  • Functional Patterns: Functors, Monads, Pipelines
  • Middleware Pattern: Frequently used in frameworks like Express.js, FastAPI, and Actix Web to structure request-handling workflows.

The Impact of AI (LLMs)

Artificial intelligence, particularly large language models (LLMs), is reshaping how developers interact with code. AI-assisted development tools introduce new opportunities and challenges in multi-paradigm programming:

  • AI-generated code may not always align with project-specific requirements or architectural decisions.
  • Developers risk losing a deep understanding of paradigms if they depend too heavily on AI-generated suggestions.
  • AI-assisted code might introduce subtle errors that require manual intervention to fix.

But a single file problems can be well solved: https://github.com/BDUG/llmgames

Conclusion

The evolution of programming languages toward multi-paradigm support reflects the growing need for flexibility, efficiency, and maintainability in modern software development. By combining different paradigms, developers can tailor solutions to fit diverse scenarios, resulting in more robust and scalable applications.

Posted in Uncategorized | Leave a comment

CNCF landscape overview

The https://landscape.cncf.io can be morphed according to the following schema:

Demand ActivityPurpose/GoalTools
Requirement GatheringCollect and define project needs and objectives.Jira, Confluence, Miro, Trello, Google Docs
Architecture DesignCreate high-level designs and system architecture.Lucidchart, Draw.io, AWS Well-Architected Tool, Azure Architecture Center, Visio
Source Code ManagementTrack, review, and manage code changes.Git, GitHub, GitLab, Bitbucket
Infrastructure as Code (IaC)Automate infrastructure provisioning and management.Terraform, Pulumi, AWS CloudFormation, Ansible, Chef, SaltStack
ContainerizationPackage applications with dependencies for portability.Docker, Buildpacks, Podman
Continuous Integration (CI)Automate code builds and initial testing.Jenkins, GitHub Actions, GitLab CI, CircleCI, Bamboo
Continuous Deployment (CD)Automate application deployment to environments.ArgoCD, Spinnaker, FluxCD, Tekton, Harness
Testing (Automated)Ensure application quality through automated tests.Selenium, JUnit, Cypress, Postman, SonarQube, OWASP ZAP
Monitoring and ObservabilityTrack system performance and detect issues.Prometheus, Grafana, Datadog, New Relic, Splunk, ELK Stack
Logging and TracingCollect logs and trace application requests for debugging.Elasticsearch, Logstash, Kibana (ELK), Fluentd, OpenTelemetry
Vulnerability ScanningIdentify security vulnerabilities in code and dependencies.Snyk, Trivy, Clair, Aqua Security, Twistlock
Secrets ManagementSecurely manage sensitive information (e.g., passwords).HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, CyberArk
Deployment OrchestrationManage deployments across clusters.Kubernetes, Helm, kubectl, Docker Swarm
Scaling and Resource ManagementDynamically scale resources based on demand.Kubernetes Autoscaler, AWS Auto Scaling, GCP Cloud Functions, Azure Monitor Autoscale
Alerting and Incident ResponseNotify teams of issues and resolve incidents.PagerDuty, Opsgenie, VictorOps, Prometheus Alertmanager, Splunk On-Call
Cost Management and OptimizationAnalyze and reduce cloud costs.AWS Cost Explorer, GCP Cost Management, Azure Cost Management, CloudHealth by VMware
Compliance ManagementEnsure adherence to industry standards and regulations.Prisma Cloud, Open Policy Agent (OPA), AWS Audit Manager, Azure Security Center
Runtime SecurityProtect running applications from threats.Falco, Sysdig, Aqua Security, Twistlock, StackRox
Posted in Uncategorized | Leave a comment

RUST and the SOLID principals

Rust as multi-paradigm language supports many of the design goals outlined in the SOLID principles. It emphasizes safety, performance, and concurrency, making it an interesting candidate for evaluating how well it adheres to these principles.

Single Responsibility Principle (SRP)

Components should be small and focused, with each handling only one responsibility.

  • Rust encourages modularity through its strict ownership and borrowing rules, which inherently promote smaller, focused units of functionality.
    • Instead of large “God objects,” Rust favors composition through structs, enums, and modules, allowing each component to handle a single responsibility.
    • The type system and traits (Rust’s version of interfaces) also promote well-defined, single-purpose abstractions.
struct Order {
    id: u32,
    total: f64,
}

struct OrderValidator;

impl OrderValidator {
    fn validate(&self, order: &Order) -> bool {
        // Validate the order (e.g., non-negative total, valid ID, etc.)
        order.total > 0.0
    }
}

struct PaymentProcessor;

impl PaymentProcessor {
    fn process_payment(&self, order: &Order) {
        // Process payment for the order
        println!("Processing payment for order ID: {} with total: ${}", order.id, order.total);
    }
}

struct OrderNotifier;

impl OrderNotifier {
    fn send_confirmation(&self, order: &Order) {
        // Notify the customer about the order
        println!("Sending confirmation for order ID: {}", order.id);
    }
}

// Putting it together
fn main() {
    let order = Order { id: 1, total: 100.0 };

    let validator = OrderValidator;
    let payment_processor = PaymentProcessor;
    let notifier = OrderNotifier;

    if validator.validate(&order) {
        payment_processor.process_payment(&order);
        notifier.send_confirmation(&order);
    } else {
        println!("Invalid order. Cannot process.");
    }
}

Open/Closed Principle (OCP)

Systems should allow for extending functionality without modifying existing code.

  • Rust’s traits allow for extensibility without modifying existing code. You can define new traits or implement existing ones for new types to extend functionality.
    • Enums with pattern matching also enable adding variants while minimizing changes to the code handling them.
    • However, since Rust lacks inheritance, extending behavior often requires composition rather than subclassing, which can sometimes make adhering to OCP slightly more verbose.
trait Shape {
    fn area(&self) -> f64;
}

struct Circle {
    radius: f64,
}

impl Shape for Circle {
    fn area(&self) -> f64 {
        3.14 * self.radius * self.radius
    }
}

Liskov Substitution Principle (LSP)

Subtypes must be substitutable for their base types without altering the correctness of the program.

  • Rust avoids classical inheritance, so LSP is achieved through traits and polymorphism.
    • By implementing traits, different types can be substituted for each other as long as they conform to the expected interface.
    • Rust’s strict type system and compile-time checks ensure that substitution errors are caught early.
trait Logger {
    fn log(&self, message: &str);
}

struct ConsoleLogger;

impl Logger for ConsoleLogger {
    fn log(&self, message: &str) {
        println!("Console log: {}", message);
    }
}

struct FileLogger;

impl Logger for FileLogger {
    fn log(&self, message: &str) {
        println!("Writing to file: {}", message); // Simulate file logging
    }
}

fn use_logger(logger: &dyn Logger, message: &str) {
    logger.log(message);
}

Interface Segregation Principle (ISP)

Interfaces should be minimal and specific to avoid forcing components to implement unused functionality.

  • Rust’s trait system inherently aligns with ISP. Traits can be designed to provide minimal, focused functionality rather than large, monolithic interfaces.
  • A struct or type can implement multiple small traits, avoiding the pitfalls of being forced to implement unnecessary methods.
trait Flyable {
    fn fly(&self);
}

trait Swimable {
    fn swim(&self);
}

struct Duck;

impl Flyable for Duck {
    fn fly(&self) {
        println!("Duck is flying");
    }
}

impl Swimable for Duck {
    fn swim(&self) {
        println!("Duck is swimming");
    }
}

struct Penguin;

impl Swimable for Penguin {
    fn swim(&self) {
        println!("Penguin is swimming");
    }
}

Dependency Inversion Principle (DIP)

High-level modules should depend on abstractions rather than low-level implementations to ensure flexibility and maintainability.

  • Rust promotes dependency inversion through its ownership model and the use of trait objects or generics.
    • High-level modules depend on abstractions (traits), and low-level modules implement these abstractions.
    • The use of Box<dyn Trait> or generics (impl Trait) allows developers to decouple components effectively.
trait PaymentProcessor {
    fn process_payment(&self, amount: f64);
}

struct PayPal;

impl PaymentProcessor for PayPal {
    fn process_payment(&self, amount: f64) {
        println!("Processing ${} payment via PayPal", amount);
    }
}

struct Stripe;

impl PaymentProcessor for Stripe {
    fn process_payment(&self, amount: f64) {
        println!("Processing ${} payment via Stripe", amount);
    }
}

struct PaymentService<'a> {
    processor: &'a dyn PaymentProcessor,
}

impl<'a> PaymentService<'a> {
    fn new(processor: &'a dyn PaymentProcessor) -> Self {
        Self { processor }
    }

    fn pay(&self, amount: f64) {
        self.processor.process_payment(amount);
    }
}

In summary, Rust aligns well with the SOLID principles, albeit with a different approach than traditional OOP languages. 

Posted in Uncategorized | Leave a comment

System design Attributes

While reflecting on how automotive software system design differs from other industries, I ended up at this overview of common attributes.

Scalability mechanism

  • Data with the aspect of availability is ensured through replication, which involves creating copies of active and mirrored data or besides replication, the sharding of multiple active data portions e.g., for caching purposes.
  • Compute is managed with horizontal and vertical partitioning to enable separation and isolation, along with load balancing to control access to the partitions. The access follows either a synchronous, asynchronous or publish-subscribe communication style, with the purpose of data (aka resource) orientation, action orientation (aka procedure call) or question oriented e.g. GraphQL.

Posted in Uncategorized | Leave a comment

ARM features classified

The new ARM features can be categorized into two main groups, each addressing distinct aspects of ISA (Instruction Set Architecture) functionality and design.

The program flow related functionality

The first group encompasses program flow related features, which operates primarily at the CPU core level, affecting the memory management unit (MMU) and cache hierarchy. These features directly influence the feature programmers, compiler settings and finally the resulting program flow of the feature itself. 

For instance, ARM’s Memory Tagging Extension (MTE) is a significant innovation within this group. MTE operates through the MMU and cache hierarchy, enhancing debugging and runtime memory safety. By detecting potential memory safety violations, it can trigger predefined actions such as logging incidents or terminating processes. 

// Allocate memory and set a tag
void *buffer = malloc(64);

// Assign tag 0x3 to the memory region
mte_set_tag(buffer, 0x3);      
char *ptr = buffer;

// Assign the same tag to the pointer
mte_set_pointer_tag(ptr, 0x3);  

// Tags match: Access allowed
ptr[0] = 'A';   
   
// Unsafe access (out-of-bounds): Fault raised
ptr[80] = 'B';  

// Free memory
free(buffer);

While MTE serves to prevent memory safety issues, its implementation introduces notable changes to control flow, creating additional software variants, increasing demands on quality management, and ultimately contributing to technical debt.

Other examples of ISA-related features include:

  • Branch Target Identification (BTI): During execution, the processor verifies whether an indirect branch or jump targets a valid BTI-marked location. If it does not, the processor halts execution or traps the error.
  • Pointer Authentication (PAC): PAC employs cryptographic keys stored securely within the processor (e.g., in hardware registers). It appends a Pointer Authentication Code to pointers and validates their integrity during execution, protecting against unauthorized modifications.

The supervision related functionality 

The second group consists of features that rely primarily on CSRs (Control and Status Registers). These registers enable configuration, control and monitoring of hardware mechanisms, particularly for managing multi-tenant performance. The program flow of a specific feature is less to not impacted. Here a supervisor e.g., hypervisor for VM or operating system for processes inject the necessary control.

Note: To be clear, the CPU cores also have to do processings based on the register values, but the technical realization primarily involves many other components. 

A key example is MPAM (Memory Partitioning And Monitoring), which provides resource partitioning and monitoring capabilities for shared resources like caches, interconnects, and memory bandwidth in ARM-based systems-on-chip (SoCs). 

MPAM, introduced in the ARMv8.4-A architecture, is implemented in components such as L3 caches, DRAM controllers and interconnects to enforce quality-of-service (QoS) policies and monitor resource usage.

MPAM is designed to enable fine-grained control over system resources in multi-core systems. Its functionalities include:

  • Partitioning: Allocating memory and cache resources to specific processes, cores and/ or virtual machines (VMs). Most configurations can be made via EL1 register. 
  • Monitoring: Tracking resource usage, such as memory bandwidth consumption, to facilitate system profiling and optimization.
  • Enforcement: Preventing resource starvation by ensuring fair allocation and predictable performance across workloads.

Key components involved for MPAM:

  • L3 Cache (DSU): Segments shared cache (e.g., L3) into partitions for different cores or tasks. Usually per core cluster. 
  • Interconnect (Coherent Mesh Network, short CMN, as part of ARM’s CoreLink product family): Enforces QoS policies, controlling memory bandwidth and traffic prioritization.
  • Memory Controllers: Allocates memory regions and enforces bandwidth quotas based on MPAM policies.
    • Accelerators (GPUs): Tags traffic to manage their impact on shared resources.
      • Peripherals (I/O): Regulates peripheral access to shared system memory and bandwidth.

Note: Memory controller is not MMU.

  • The Memory controller is responsible for managing physical access to the computer’s memory (RAM). It acts as the interface between the CPU and the physical memory.
  • The MMU is responsible for virtual memory management and translates virtual addresses (used by software) into physical addresses (used by hardware).

Posted in Uncategorized | Leave a comment

Agentin AI versus AI Agent

Agentic AIAI Agent
Autonomy LevelCan act independentlyNeeds human input
DriverGoal orientedTask oriented
Learning capabilityContinuous learn and improvePredefined rules
ComplexityDynamic environment(pre)structured tasks
Decision makingBased on reasoningPre-programmed
Interactionproactivereactive

Google Gemini 2.0 can be both …

Posted in Uncategorized | Leave a comment

Technology management in diverse societies

This post contains thoughts of Giles Crouch (The digital anthropologist)

Technological development increasingly reflects regional and cultural identities (compare external link), yet its rapid pace often outstrips society’s ability to adapt. Unlike earlier innovations like the railroad or telephone, which allowed time for cultural assimilation, modern technologies create a cultural lag, compelling societal behaviors to evolve quickly. For instance, swiping gestures on smartphones have become second nature for those born into a world where such interactions are the norm, fulfilling basic needs as outlined in Maslow’s hierarchy.

A notable example is the rise of WeChat Pay and Alipay in China. Shaped by regional behaviors, these systems exemplify rapid societal adaptation, with cashless transactions now the norm. QR code payments, ubiquitous for daily activities like grocery shopping or public transport, feel natural to those born into this ecosystem. These technologies address fundamental needs such as safety (secure payments) and belonging (participation in a cashless society) while meeting demands for trust and convenience, demonstrating how cultural contexts influence technological success.

Technology not only shapes belonging and culture but is itself shaped by cultural philosophies. In the West, development is informed by Cartesian dualism, emphasizing competition and existential concerns over superintelligent systems. In Japan, “kokoro” (heart and spirit, compare external link) guides the view of tools and robotics as harmonious companions rather than human replacements. India integrates ancient Vedantic wisdom (compare external link), focusing on purpose and utility (“dharma” and “Maya”, compare external link) rather than rigid distinctions of real versus fake. African nations, guided by the communal philosophy of “ubuntu” (“I am because we are”, compare external link), prioritize technology that serves society over the individual. For instance, Kenya uses technology in agriculture to preserve traditional knowledge while boosting productivity.

Across cultures, data has become the modern equivalent of colonial-era spices, powering “techno-feudalism” (compare external link). Technology companies wield immense influence akin to colonial “company states,” with oversight boards acting as global arbiters and marketplace rules functioning like commercial laws. Unlike physical commodities, data grows in value with use, creating dependencies that resemble colonial-era networks. Tech giants’ “civilizing missions,” such as Meta’s goal to connect the world or Google’s aim to organize the worlds information, echo colonial narratives, raising concerns about digital colonialism.

The digital divide adds another layer of complexity, extending beyond economic disparities to socio-technical gaps (compare external link and idea of “Value Engineering”). Those who embrace advanced tools gain significant advantages, while those who lack access or reject such technologies risk falling further behind, exacerbating societal inequality. Ironically, former colonized regions and their “company states” (compare external link) are now leading new forms of digital colonization, as partnerships between corporations redefine power dynamics and challenge foundational cultural concepts.

Posted in Uncategorized | Leave a comment

Why Small Language Model matter

I just read the following article: https://medium.com/towards-data-science/your-company-needs-small-language-models-d0a223e0b6d9. I personally liked the following illustration:

Original sources: source1source2source3source4

From a non technical view I was shocked by the following illustration:

Original source: https://www.linkedin.com/posts/bgamazay_openai-has-announced-o3-which-appears-to-activity-7276250095019335680-sVbW

At least this is why SML will be the next step.

Posted in Uncategorized | Leave a comment

The Seq2Seq issue history repeats for transformer

Classical sequence-to-sequence (Seq2Seq) models based on Recurrent Neural Networks (RNNs) condense the entire input sequence into a single fixed-length context vector. While effective for short sequences, they struggle with longer or more complex inputs due to their inability to capture fine-grained, long-range dependencies. RNNs also process tokens sequentially, which:

  • Slows down training and inference (no parallelization).
  • Suffers from vanishing gradients, hindering long-term context retention.
  • Leads to performance degradation on longer sequences.

Transformers address many of these issues with self-attention, allowing any token to relate to any other token in parallel. A “cross-table” (example) of attention weights quantifies relationships between tokens, enabling global context-modeling. However, transformers lack an inherent notion of order, so positional encodings are used to incorporate sequence ordering information.

AspectSeq2SeqTransformers
Context HandlingSingle, fixed-length vectorFull pairwise attention (“cross-table”)
ParallelizationSequential (no parallel computation)Highly parallelizable (self-attention)
Long-Range DependenciesSusceptible to vanishing gradients & limited contextCaptures distant dependencies effectively
Positional EncodingImplicit through recurrenceRequires explicit positional encoding
Typical BottlenecksSlow training, limited context vectorLarge memory usage for attention over very long sequences

Modern applications involve multiple tasks and conversations, creating fragmented and distributed contexts. While transformers excel within a single conversation, they face new obstacles when context must persist and link across-multiple interactions. Each conversation (or task) may be isolated, yet users expect systems to recall or relate information from previous tasks.

Key challenges:

  • Context Fragmentation: Each conversation is treated independently, losing continuity and coherence across tasks.
  • Memory Bottlenecks: Attention mechanisms typically have fixed context windows, limiting the retention of long-term history.
  • Cross-Task Relationships: Transformers are optimized for intra-task focus; linking knowledge between tasks is more complex.
  • Global Context Across Tasks: Demands new architectures (memory-augmented, retrieval-based) to store and retrieve shared context.

To handle context across tasks and conversations, researchers explore

  • Memory-Augmented Networks: Maintain an external memory component that persists beyond a single conversation.
  • Retrieval-Augmented Generation (RAG): Dynamically fetch relevant pieces of information from a knowledge base.
  • Hierarchical Attention Mechanisms: Organize multiple attention layers or modules to capture context at different scopes (per conversation, per session, or globally).

As conversations and tasks become more fragmented and intertwined, ensuring continuity and consistency of context becomes a central requirement, especially with newly upcoming AI agent approaches.

Posted in Uncategorized | Leave a comment