Stock Sentiment Analysis on Reddit: A NER Approach

Reddit Investing NER Dataset

Dataset Description

The Reddit Investing NER (Named Entity Recognition) Dataset comprises 880 items collected from the Reddit API, specifically from the r/investing subreddit's new feed. The dataset facilitates Named Entity Recognition using Natural Language Processing (NLP) techniques.

Dataset Details

The dataset includes the following columns:

• Name: Unique identifier for each post item.
• Created_utc: Timestamp indicating when the post was created, in Coordinated Universal Time (UTC).
• Subreddit: The subreddit from which the post originated (r/investing).
• Title: Title of the post, providing a brief summary or topic description.
• Selftext: Content of the post, typically containing additional details or information beyond the title.
• Upvote_ratio: Ratio of upvotes to total votes received by the post.
• Ups: Number of upvotes received by the post.
• Score: Overall score assigned to the post, calculated based on upvotes and other factors.

Purpose

The primary purpose of this dataset is to support the development and evaluation of Named Entity Recognition models within the domain of investing-related discussions on Reddit. Researchers and data scientists can utilize this dataset to train, test, and fine-tune NER models, ultimately enhancing their ability to extract relevant information and entities from unstructured text data.

Potential Use Cases

1. Entity Identification: Extract named entities related to investment topics mentioned in post titles and selftext.
2. Sentiment Analysis: Explore sentiment trends within the investing community based on score, upvote ratio, and upvotes.
3. Temporal Analysis: Investigate patterns and trends over time using the created_utc timestamp.

Notebook Steps

Import spaCy:

Utilize the en_core_web_sm model.

Import Data:

Fetch the Reddit Investing NER Dataset.

Get Specific Entities Handler:

Implement a handler to extract specific entities, updating it to ignore blacklist words.

Add Organizations Feature:

Enhance the dataset by adding a new feature for organizations.

Entity Frequency:

Analyze entity frequency, incorporating a blacklist to filter out inaccuracies.

NER with Sentiment:

Perform Named Entity Recognition with sentiment analysis, identifying organizations and their sentiment scores.

Top 10 Positives & Negatives:

Present the top 10 organizations with the highest and lowest average sentiment scores, respectively.

Top 10 Positive Entities

Top 10 Negative Entities