2024-01-01 19:19:10 [scrapy.core.engine] INFO: Spider opened ... 2024-01-01 19:19:10 [quotes] DEBUG: Saved file quotes-1.html 2024-01-01 19:19:10 [quotes] DEBUG: Saved file quotes-2.html 2024-01-01 19:19:10 [scrapy.core.engine] INFO: Closing spider (finished)
-
Social Media Analytics
Social Media Analytics
Why Social Media Matters for Brands
Insights Gleaned from Social Media Platforms
Strengths of Social Media Data
Limitations of Social Media Data
Understanding Social Data
Social Media Platforms: Key Features
Structured and Unstructured-Data
Social Data Mining
Social Data Mining Process
Social Data Mining Techniques
Social Data Mining Challenges
Application Programming Interfaces
How APIs Work
Working with APIs
Endpoints
Twitter (X) API
Twitter (X) API — Securing Access
Twitter (X) REST API in Python
Facebook API
Facebook Graph API
Facebook API — Securing Access
Facebook API in Python
Advantages and Limitations of APIs
Data Cleaning Techniques
Natural Language Processing
Natural Language Toolkit (NLTK)
Social Media Data Types
Textual Data Encoding
Text Processing Techniques
Tokenization
Word Tokenization
Character Tokenization
Sub-Word Tokenization
Stemming and Lemmatization
Stemming
Lemmatization
Stemming and Lemmatization in Python
N-grams, Bigrams, and Trigrams
Applications of N-grams
Applications of N-grams in Sentiment Analysis
Topic Modelling with N-grams
Vectorization
Bag-of-Words
TF-IDF Vectorizer
Facebook Brand Page Analysis
Extracting Insights from Facebook Brand Pages
Facebook — Social Data Analysis Process
Facebook — Data Extraction
Text Analytics
Text Analytics Process
Part of Speech (POS) Tagging
Noun Phrases
Text Data Processing in Python
Word Cloud (FB data) in Python
Time Series Analysis and Visualization of FB Comments
Emotion Analysis
IBM Watson Natural Language Understanding
Accessing IBM Cloud Services
Emotion Analysis Using Watson NLU
Sentiment Analysis
Forms of Sentiment Analysis
Types of Sentiment Analysis
Visual Sentiment Analysis and Facial Coding
Applications of Facial Coding
Sentiment Analysis in Text
Analysis of Behaviours and Sentiments
Sentiment Analysis Process
Sentiment Analysis — Classification
VADER Classifier
Standard Sentiment Analysis
Customised Sentiment Analysis
Model Validation – Confusion Matrix
K-fold Cross-validation
Named Entity Recognition (NER)
NER Process Overview
Stanford NER
Challenges in NER
Stanford NER Implementation in Python
Web Scraping
Web Scraping Techniques
Applications of Web Scraping
Legal and Ethical Considerations
Beautiful Soup
Scraping Quotes to Scrape
Scraping of Fake Jobs Webpage
Scrapy
Scrapy Concepts
Scrapy Framework
Scrapy Limitations
Beautiful Soup vs. Scrapy — A Comparison
Selenium
Topic Modelling
Topic Modelling — Illustration
Topic Modelling Techniques
Topic Modelling Process
Latent Dirichlet Allocation (LDA) Model
Topic Modelling Tweets with LDA in Python
Social Influence on Social Media
Key Forms of Social Influence
Social Influence on Social Media Platforms
Examples of Organized Social Influence
Social Network Analysis
Topic Networks and User Networks
Online Social Networks — The Basics
Analysing Topic Networks
Centrality Measures
Degree Centrality
Betweenness Centrality
Closeness Centrality
Eigenvector Centrality
Use Case — Marketing Analytics Topic Network
Social Network Analysis Process
SNA — Uncovering User Communities
Appendix — Python Basics: Tutorial
Installation — Anaconda, Jupyter and Python
Python Syntax
Variables
Data Types
If Else Statement
While and For Loops
Functions (def)
Lambda Functions
Modules
JSON
Python Requests — get(), json()
User Input
Exercises
Appendix — Python Pandas
Basic Usage
Reading, Writing and Viewing Data
Data Cleaning
Other Features
Appendix — Python Visualization
Matplotlib
Matplotlib — Basic Plotting
NumPy
Matplotlib — Beyond Lines
Analysis and Visualization of the Iris Dataset
Word Clouds
Word Cloud in Python
Seaborn — Statistical Data Visualization
Seaborn Visualization in Python
Appendix — Scrapy Tutorial
Creating a Project
Writing a Spider
Running the Spider
Extracting Data
Extracting Data — CSS Method
Extracting Data — XPath Method
Extracting Quotes and Authors
Extracting Data in Spider
Storing the Scraped Data
Pipeline
Following Links
Appendix — HTML Basics
HTML Tree Structure, Tags and Attributes
Tags
Attributes
My First Webpage
- New Media
- Digital Marketing
- YouTube
- Social Media Analytics
- SEO
- Search Advertising
- Web Analytics
- Execution
- Case — Prop-GPT
- Marketing Education
- Is Marketing Education Fluffy and Weak?
- How to Choose the Right Marketing Simulator
- Self-Learners: Experiential Learning to Adapt to the New Age of Marketing
- Negotiation Skills Training for Retailers, Marketers, Trade Marketers and Category Managers
- Simulators becoming essential Training Platforms
- What they SHOULD TEACH at Business Schools
- Experiential Learning through Marketing Simulators
-
MarketingMind
Social Media Analytics
Social Media Analytics
Why Social Media Matters for Brands
Insights Gleaned from Social Media Platforms
Strengths of Social Media Data
Limitations of Social Media Data
Understanding Social Data
Social Media Platforms: Key Features
Structured and Unstructured-Data
Social Data Mining
Social Data Mining Process
Social Data Mining Techniques
Social Data Mining Challenges
Application Programming Interfaces
How APIs Work
Working with APIs
Endpoints
Twitter (X) API
Twitter (X) API — Securing Access
Twitter (X) REST API in Python
Facebook API
Facebook Graph API
Facebook API — Securing Access
Facebook API in Python
Advantages and Limitations of APIs
Data Cleaning Techniques
Natural Language Processing
Natural Language Toolkit (NLTK)
Social Media Data Types
Textual Data Encoding
Text Processing Techniques
Tokenization
Word Tokenization
Character Tokenization
Sub-Word Tokenization
Stemming and Lemmatization
Stemming
Lemmatization
Stemming and Lemmatization in Python
N-grams, Bigrams, and Trigrams
Applications of N-grams
Applications of N-grams in Sentiment Analysis
Topic Modelling with N-grams
Vectorization
Bag-of-Words
TF-IDF Vectorizer
Facebook Brand Page Analysis
Extracting Insights from Facebook Brand Pages
Facebook — Social Data Analysis Process
Facebook — Data Extraction
Text Analytics
Text Analytics Process
Part of Speech (POS) Tagging
Noun Phrases
Text Data Processing in Python
Word Cloud (FB data) in Python
Time Series Analysis and Visualization of FB Comments
Emotion Analysis
IBM Watson Natural Language Understanding
Accessing IBM Cloud Services
Emotion Analysis Using Watson NLU
Sentiment Analysis
Forms of Sentiment Analysis
Types of Sentiment Analysis
Visual Sentiment Analysis and Facial Coding
Applications of Facial Coding
Sentiment Analysis in Text
Analysis of Behaviours and Sentiments
Sentiment Analysis Process
Sentiment Analysis — Classification
VADER Classifier
Standard Sentiment Analysis
Customised Sentiment Analysis
Model Validation – Confusion Matrix
K-fold Cross-validation
Named Entity Recognition (NER)
NER Process Overview
Stanford NER
Challenges in NER
Stanford NER Implementation in Python
Web Scraping
Web Scraping Techniques
Applications of Web Scraping
Legal and Ethical Considerations
Beautiful Soup
Scraping Quotes to Scrape
Scraping of Fake Jobs Webpage
Scrapy
Scrapy Concepts
Scrapy Framework
Scrapy Limitations
Beautiful Soup vs. Scrapy — A Comparison
Selenium
Topic Modelling
Topic Modelling — Illustration
Topic Modelling Techniques
Topic Modelling Process
Latent Dirichlet Allocation (LDA) Model
Topic Modelling Tweets with LDA in Python
Social Influence on Social Media
Key Forms of Social Influence
Social Influence on Social Media Platforms
Examples of Organized Social Influence
Social Network Analysis
Topic Networks and User Networks
Online Social Networks — The Basics
Analysing Topic Networks
Centrality Measures
Degree Centrality
Betweenness Centrality
Closeness Centrality
Eigenvector Centrality
Use Case — Marketing Analytics Topic Network
Social Network Analysis Process
SNA — Uncovering User Communities
Appendix — Python Basics: Tutorial
Installation — Anaconda, Jupyter and Python
Python Syntax
Variables
Data Types
If Else Statement
While and For Loops
Functions (def)
Lambda Functions
Modules
JSON
Python Requests — get(), json()
User Input
Exercises
Appendix — Python Pandas
Basic Usage
Reading, Writing and Viewing Data
Data Cleaning
Other Features
Appendix — Python Visualization
Matplotlib
Matplotlib — Basic Plotting
NumPy
Matplotlib — Beyond Lines
Analysis and Visualization of the Iris Dataset
Word Clouds
Word Cloud in Python
Seaborn — Statistical Data Visualization
Seaborn Visualization in Python
Appendix — Scrapy Tutorial
Creating a Project
Writing a Spider
Running the Spider
Extracting Data
Extracting Data — CSS Method
Extracting Data — XPath Method
Extracting Quotes and Authors
Extracting Data in Spider
Storing the Scraped Data
Pipeline
Following Links
Appendix — HTML Basics
HTML Tree Structure, Tags and Attributes
Tags
Attributes
My First Webpage
- New Media
- Digital Marketing
- YouTube
- Social Media Analytics
- SEO
- Search Advertising
- Web Analytics
- Execution
- Case — Prop-GPT
- Marketing Education
- Is Marketing Education Fluffy and Weak?
- How to Choose the Right Marketing Simulator
- Self-Learners: Experiential Learning to Adapt to the New Age of Marketing
- Negotiation Skills Training for Retailers, Marketers, Trade Marketers and Category Managers
- Simulators becoming essential Training Platforms
- What they SHOULD TEACH at Business Schools
- Experiential Learning through Marketing Simulators
Python Scrapy: Running the Spider
To run the spider, navigate to the project’s top-level directory and run:
scrapy crawl quotes
This command will initiate the spider, sending requests to the quotes.toscrape.com domain. The output will look something like this:
Two new files would have been created: quotes-1.html and quotes-2.html, with the content for the URLs listed in the spider.
Alternatively, instead of implementing start_requests(), you can define a start_urls class attribute as follows:
Spider using start_urls
import scrapy
from pathlib import Path
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
"https://quotes.toscrape.com/page/1/",
"https://quotes.toscrape.com/page/2/",
]
# def start_requests() not required.
def parse(self, response):
page = response.url.split("/")[-2]
filename = f"quotes-{page}.html"
Path(filename).write_bytes(response.body)
import scrapy
from pathlib import Path
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
"https://quotes.toscrape.com/page/1/",
"https://quotes.toscrape.com/page/2/",
]
# def start_requests() not required.
def parse(self, response):
page = response.url.split("/")[-2]
filename = f"quotes-{page}.html"
Path(filename).write_bytes(response.body)
Previous Next
Use the Search Bar to find content on MarketingMind.
Contact | Privacy Statement | Disclaimer: Opinions and views expressed on www.ashokcharan.com are the author’s personal views, and do not represent the official views of the National University of Singapore (NUS) or the NUS Business School | © Copyright 2013-2025 www.ashokcharan.com. All Rights Reserved.