LLM Architecture

AI and LLMs - why GPUs, what happened to CPUs?

This has been written with the help of Copilot.

Central Processing Units (CPUs)

• CPUs are versatile and handle a wide range of tasks (execution of instructions, general purpose computing, coordinating various components within the computer system (memory, I/O, peripheral devices, etc.).
• They are the “brains” of a computer, executing instructions from software programs.
• CPUs excel at sequential processing, where tasks are executed one after the other. They also support parallel processing to some extent through techniques like multi-core architectures and pipelining.
• They manage tasks like operating system functions, application execution, and handling I/O operations.
• CPUs are essential for general-purpose computing, including running applications, managing memory, and coordinating system resources.

Graphics Processing Units (GPUs), 

GPUs were initially associated primarily with gaming, and other applications like scientific simulations and data processing, however they now have transcended their original purpose as GPUs have evolved beyond just gaming. Let’s explore how this transformation occurred and why GPUs are now indispensable for various computing tasks beyond gaming.

• GPUs are specialized hardware components designed for parallel processing.
• Their architecture consists of thousands of cores, each capable of handling computations simultaneously.
• Originally developed for graphics rendering (such as gaming), GPUs evolved to handle complex mathematical operations efficiently.
• GPUs excel at tasks like matrix operations, image processing, and parallel algorithms.
• In recent years, GPUs have become crucial for AI, machine learning, scientific simulations, and data-intensive workloads.

1. Evolution of GPUs:

   • Early Days: GPUs were initially designed for rendering lifelike graphics in video games. Their primary role was to alleviate the burden on the Central Processing Unit (CPU) by handling graphical computations.
   • Parallel Processing: Over time, GPUs evolved significantly. Modern GPUs are equipped with thousands of cores optimized for parallel processing. Unlike CPUs, which focus on sequential tasks, GPUs excel at performing numerous calculations simultaneously.
   • Massive Computational Power: Today’s GPUs can achieve teraflops (trillions of floating-point operations per second), a capability that was inconceivable just 15 years ago1.

2. Beyond Gaming: Diverse Applications:

   • Artificial Intelligence (AI) and Machine Learning:
     • GPUs play a pivotal role in training neural networks for AI and machine learning.
     • Their parallel architecture accelerates tasks like natural language processing and computer vision.
   • Data Science and Analytics:
     • GPUs handle massive datasets efficiently, reducing computation times for tasks like data preprocessing and statistical analysis.
   • High-Performance Computing (HPC):
     • Scientific research, weather forecasting, and simulations rely heavily on GPUs.
     • They excel in solving complex mathematical models with remarkable accuracy.
   • Medical Imaging and Research:
     • GPUs process large amounts of visual data in medical imaging techniques like MRI and CT scans, aiding diagnosis and research1.

3. The Trajectory of GPUs:

   • As technology advances, the demand for high-performance computing, especially in AI and machine learning, is expected to soar.
   • GPUs will continue to evolve, integrating more efficiently with different technologies and expanding their role beyond traditional applications.
   • Optimizing GPU usage involves understanding their architecture, using optimized libraries (like CUDA or OpenCL), and parallelizing tasks to maximize potential.

LLMs - words vs tokens

https://kelvin.legal/understanding-large-language-models-words-versus-tokens/#:~:text=The%20size%20of%20text%20an,the%20cost%20and%20vice%20versa.

Tokens can be thought of as pieces of words. Before the model processes the prompts, the input is broken down into tokens. These tokens are not cut up exactly where the words start or end - tokens can include trailing spaces and even sub-words. -- Llama.

The size of text an LLM can process and generate is measured in tokens. Additionally, the operational expense of LLMs is directly proportional to the number of tokens it processes - the fewer the tokens, the lower the cost and vice versa. 

Tokenizing language translates it into numbers – the format that computers can actually process. Using tokens instead of words enables LLMs to handle larger amounts of data and more complex language. By breaking words into smaller parts (tokens), LLMs can better handle new or unusual words by understanding their building blocks.

Agile in the age of AI - Henrik Kniberg

 https://hups-com.cdn.ampproject.org/c/s/hups.com/blog/agile-in-the-age-of-ai?hs_amp=true

Agile methodologies like Scrum are being impacted by the rise of AI. The traditional assumptions about team dynamics, roles, and development cycles are being challenged.

• Cross-Functional Teams: AI's vast knowledge and productivity acceleration are reshaping the need for cross-functional teams. Smaller teams and more teams with AI assistance may become the norm.
• Superteam: There is a possibility of super team, where these smaller teams will have a kind of standups to sync up, coordinate, and address dependencies and issues. Purpose and structure of these meetings will change from what they do now.
• Changing Developer Roles: With AI's capability to generate code, developers may shift to decision-making and oversight roles, with AI handling much of the coding work.
• Redefining Sprints: Agile sprints may become shorter or disappear as AI speeds up development cycles, making traditional timeboxing less relevant.
• Specialists in Agile Teams: Specialists may become roaming or shared resources, complementing AI capabilities within smaller teams.
• Evolution of Scrum Master Role: Scrum Masters may transition to coaches, guiding teams in effectively utilizing AI technologies.
• User Feedback Loop: AI-driven mock users could supplement real user feedback, allowing for more frequent and immediate input in Agile development.
• Additional Considerations: Various factors like 
• Product backlog prioritization: product backlog will need to be updated frequently. PO will focus more on strategic prioritization and stakeholder management. 
• Estimation methods: teams will need new ways to planning and forecasting. 
• Framework adaptations: Popular Agile frameworks like Scrum, Kanban, or SAFe might need to be adapted to accommodate the changes brought by AI. 
• Team dynamics: teams will require new ways to ensure human connection, creativity, and innovation in an AI-driven environment.
• Continuous learning will become even more cruicial as AI keeps taking up larger share of what it can contribute. Team members may need to focus on developing new skills, such as prompt engineering, AI model selection, and result evaluation.
• Ethical considerations need to be addressed in the AI-driven Agile landscape - biases, fairness and transparency. 

The Age of AI calls for a recalibration of Agile practices, with a focus on adapting to the new realities brought about by AI technologies.

Secure by Design

 Secure by Design (SBD) in the IT industry refers to an approach where security is integrated into the design phase of software, systems, or products rather than being added as an afterthought. The goal is to proactively identify and mitigate security risks throughout the development lifecycle rather than trying to patch vulnerabilities later.

The Secure by Design (SBD) Engineer works closely with the Project Manager (PM) from the project's outset. Together, they examine the architecture with a focus on security. If any vulnerabilities or risks are identified, the SBD Engineer provides recommendations to address them.

As the project progresses, and typically during the mid-stage of System Integration Testing (SIT) when major defects are resolved, the SBD Engineer requests a Fortify scan of the codebase before deployment. If the scan reveals no issues, the process continues smoothly. However, if vulnerabilities are found, the team undertakes code refactoring to address them. After refactoring, the SBD Engineer ensures that any changes do not affect the system's functionality through SIT regression testing.

This meticulous approach ensures that security is integrated into every phase of the project, ultimately resulting in a more resilient and secure IT system.