Introduction to Data Analytics - IBM

Python Programming

•  List vs Tuple

List is mutable whereas Tuple is unmutable. You can change values in a list once it is defined, however for a Tuple, once the values are defined, they cannot be changed / manipulated. 

• List

Mutable: Can be modified after creation (elements can be added, removed, or changed).
Ordered: Maintains the order of elements.
Indexable: Allows access to elements by their index.
Dynamic Size: Can grow or shrink as needed.
Syntax: Defined with square brackets, e.g., [1, 2, 3].
Methods: Extensive set of methods for modification and querying (e.g., append(), extend(), remove(), pop(), sort()).

my_list = [1, 2, 3]

my_list.append(4)  # [1, 2, 3, 4]

my_list[1] = 5     # [1, 5, 3, 4]

• Tuple

Immutable: Cannot be modified after creation. 

Ordered: Maintains the order of elements.

Indexable: Allows access to elements by their index.

Fixed Size: The size is fixed upon creation.

Syntax: Defined with parentheses, e.g., (1, 2, 3).

Methods: Limited set of methods mainly for querying (e.g., count(), index()). 

•  

Business Analysis Fundamentals

As a business analyst, 

1. You'll facilitate communication between different stakeholders, business users, IT teams and management, 
2. Organizing meetings, 
3. Gathering requirements, and ensuring everyone is aligned, 
4. Driving projects forward with clarity and efficiency. 
5. Gather and document business requirements for projects or initiatives, 
6. Conducting interviews, surveys, or workshops with stakeholders to understand their needs and priorities, 
7. Translating these requirements into actionable plans for development teams. 
8. Testing and validation, you'll assist with user acceptance testing or UAT, ensuring that new software meets end users needs before deployment. 
9. Support project managers in coordinating and tracking project activities, milestones and timelines, 
10. Updating project plans, monitoring progress, and communicating status updates to stakeholders. 
11. Participate in process improvement initiatives, identifying areas for optimization, and proposing solutions to streamline operations. From suggesting workflow improvements to enhancing inventory management processes, your insights will drive continuous improvement within your organization. 

Tasks of a BA

• Problem identification / business needs / root cause
• Stakeholder management
• Risk management
• Process analysis and process improvement
• Data analysis
• Gathering requirements and documenting them
• Solution evaluation and recommendations
• Testing and validation
• Implementation support
• Continuous improvement

End to End Process

Electricity supply company. 

Issue is customer churn. Nearly 50% of their customers do not renew their contracts with the electricity supplier.

BA is involved here at Project Initiation / Requirements gathering / define project scope / conduct feasibility studies.

Before conducting the stakeholder workshops on the requirements, BA goes thru:

> Churn statistics

> Customer complaints

> Surveys

> Trace customer journeys

> Review Google Analytics of customer journeys

Introduction to Generative AI - Andrew NG

 

What is Generative AI

AI systems that can generate high quality content, specifically text, images and video and audio. Best example is OpenAI's ChatGPT.

What is Generative AI

Supervised Learning (Labeling Things)

They are trained to repeatedly predict the next word.

LLMs as a thought partner

AI as a general purpose technology

• AI is a general purpose technology - it's useful for a lot of things. (Like electricity)
• Internet is another general purpose technology.

• App --- LLM that will be better off built into an automated software system.
• Web --- Web interface

Generative AI Applications

• Use as brainstorming partner
• For writing - example, write a press release.
• Translation - example translate the following into Hindi / formal Hindi / formal spoken Hindi
• Translate text into Pirate English for teseting purposes. 
• Proofreading
• Summarizing a long article.

• Summarizing call centre conversations. (Record --> Run Phone calls --> Speech Recognition --> Long Test Transcripts --> LLM -- Summarise converstaion --> Generate short summary --> spot issues or trends (Manager)

• Customer email analysis: 
• Complaint?
• Route this email to a particular department

• Reputation monitoring --- sentiment analysis (positive or negative)

• Can build an Alert System which can alert you if sentiment is dropping negative. 

Chatting

• Specialized chatbots
• Customer service chatbot
• Trip planning chatbot
• Advise bots - career coaching, cooking a meal, etc.
• Some can ACT if sufficiently empowered.

Chatbots in Customer Services

Deploying chatbots

> First deploy internal-facing chatbots that chat with internal users only.

> When sufficiently confident of their performance, move over to human-in-the-loop where a human will validate the popped up message for certain action.

> Once both the above are confidently proven (Only if deemed safe, allow the bot to communicate directly with customers), you can deploy them into production environment. (Above two ensures that you will avoid public mistakes that could prove to be costly. 

Using Generative AI in Software Applications

Before the rise of LLMs, some of the software existed (as below), which were trained using supervised learning. The whole process took months. With the advent of LLMs, the entire timeline has been squeezed and same functionality can be achieved within weeks or days even.

Example - reading restaurant reviews for reputation monitoring, the code used to read as below.

A few hundred or thousands of data points as these

Same using a prompt engineering is below - you simply pass the string to an LLM and get back the result. 

Trying Generative AI Code

Building a Reputation Monitoring System

What are are trying to achieve:

1. We will set up the environment before progressing.

import openai

import os 

openai.api_key = os.getenv("OPENAI_API_KEY")

def llm_response(prompt):

    response = openai.ChatCompletion.create(

        model='gpt-3.5-turbo',

        messages=[{'role':'user','content':prompt}],

        temperature=0

    )

    return response.choices[0].message['content']

Once done, run the above piece of code. There won't be any output, but this will set up the environment variables for our programs to run. 

2. Now, create a list of reviews. 

all_reviews = [

    'The mochi is excellent!',

    'Best soup dumplings I have ever eaten.',

    'Not worth the 3 month wait for a reservation.',

    'The colorful tablecloths made me smile!',

    'The pasta was cold.'

]

all_reviews

When you run the above code, output will be:

['The mochi is excellent!',
'Best soup dumplings I have ever eaten.',
'Not worth the 3 month wait for a reservation.',
'The colorful tablecloths made me smile!',
'The pasta was cold.']

3. Write a code to classify the above reviews as poistive or negative.

all_sentiments = []
for review in all_reviews:
prompt = f'''
Classify the following review
as having either a positive or
negative sentiment. State your answer
as a single word, either "positive" or
"negative":
{review}
'''
response = llm_response(prompt)
all_sentiments.append(response)
all_sentiments
Output will be ['positive', 'positive', 'negative', 'positive', 'negative']

4. Write code to count the number of positive and number of negative reviews.

num_positive = 0
num_negative = 0
for sentiment in all_sentiments:
if sentiment == 'positive':
num_positive += 1
elif sentiment == 'negative':
num_negative += 1
print(f"There are {num_positive} positive and {num_negative} negative reviews.")
Output is: There are 3 positive and 2 negative reviews.

Lifecycle of a Generative AI Project

The above output is wrong because tonkotsu ramen is pork soup, and the above response is not considered positive.

Now feedback to the system to improve

Tools to improve performance

Prompting is a higly empirical process
Idea -- Prompt -- LLM Response -- Tweak Idea and the cycle repeats
RAG - Retrieval Augmented Generation (RAG)
Give LLM access to external data sources.
Fine tune models
Adapt LLMs to your task.
Pretrain models
Train LLMs from scratch
Example

Above, where the calorific value is asked, you can go over to RAG.

How much do LLMs Cost?


A Token is a word or a sub part of a word. 





300 words == 400 tokens

number of tokens is loosely about number of words.



Retrieval Augment Generation (RAG)


RAG will refer to additional inputs from your company to answer the question above.





Example of RAG
This is called Retrieval Augmented Generation or RAG, because we're going to generate an answer to this, but we're going to augment how we generate text by retrieving the relevant context or the relevant information and augmenting the prompt with that additional text.

> AI Bots that allow you to chat with PDFs

> AIs that allow you to get answers for questions on website's articles

> New form of search on web

LLM as a Reasoning Engine

Pre-training and Fine Tuning AI

> To carry out a task that is difficult to define in a prompt. 

> LLMs are fine tuned to gain speicfic knowledge. 

Example medical notes




Legal notes