Topic Modelling Process

Exhibit 25.39 Topic Modelling Process.

The topic modelling process generally involves several key steps:

1. Data Scraping: Collecting textual data from various sources.
2. Textual Data Cleaning and Processing: Preparing the text for analysis by removing noise and normalising the data.
3. Topic Model and Analysis: Applying a topic model like LDA or LSA to discover topics within the cleaned data.
4. Recommendations: Interpreting the topics and providing actionable insights based on the analysis.

Data Scraping

Web scraping is a valuable technique for gathering large datasets from the web, which can then be used for topic modelling. By using web scraping tools like Scrapy or Beautiful Soup, you can extract vast amounts of textual data from websites, forums, news articles, or social media platforms. This scraped data provides the corpus of documents needed for topic modelling algorithms such as Latent Dirichlet Allocation (LDA) or Latent Semantic Analysis (LSA).

Data Cleaning and Processing

To prepare textual data for topic modelling, it is essential to preprocess it. This involves data cleaning steps such as removing extra spaces, HTML tags, URLs, and punctuation marks. Additionally, standardizing words, splitting attached words, and converting text to lowercase are crucial. Finally, eliminating stop words (common words like “the”, “and”, “a”) and tokenizing the text into individual words or phrases is necessary to create a clean and structured dataset for analysis.