How to download twitter data?

http://www.r-bloggers.com/search/twitter

Dominik Batorski

University of Warsaw

You can use

Twitter APIs:

REST API https://dev.twitter.com/rest/public and Streaming API https://dev.twitter.com/streaming/overview

R packages:

e.g. twitteR, RTwitterAPI

See also blog posts on Twitter data on R-bloggers: http://www.r-bloggers.com/search/twitter

19 Recommendations

Dariusz Prokopowicz

Cardinal Stefan Wyszynski University in Warsaw 🏛️

Ljubomir Jacić

Technical College Požarevac

Chuck A Arize

Texas A&M University-Commerce

D A Gayan Nayanajith

University of Kelaniya

Joseph Stroud

Bond University

Chee Siang Ang

University of Kent

have you tried NodeXL? http://nodexl.codeplex.com/

2 Recommendations

Francesc Grau

Universitat Oberta de Catalunya

You can download your own file going to your Settings -> "Request your archive" button. And you'll get to your first tweet :-)

1 Recommendation

Indian Institute of Technology Madras

@Chee Siang Ang: I didn't try. I am using R packages.

Indian Institute of Technology Madras

@ Francesc grau: Thank you for your reply. I want to download some other tweets not my own tweets.

Francesc Grau

Universitat Oberta de Catalunya

You can't access to file of others by this way :-(

Indian Institute of Technology Madras

@ Francesc Grau: Can you tell me, is there a way to do it?

Francesc Grau

Universitat Oberta de Catalunya

You can try https://www.exporttweet.com or http://www.tweetdownload.net

I used them long time ago and both were well. Nowadays, I don't know. Sorry

1 Recommendation

Indian Institute of Technology Madras

@ Francesc Grau: Thanks a lot.

http://www.r-bloggers.com/search/twitter

Dominik Batorski

University of Warsaw

You can use

Twitter APIs:

REST API https://dev.twitter.com/rest/public and Streaming API https://dev.twitter.com/streaming/overview

R packages:

e.g. twitteR, RTwitterAPI

See also blog posts on Twitter data on R-bloggers: http://www.r-bloggers.com/search/twitter

19 Recommendations

Iñaki San Vicente

Elhuyar Fundazioa

You probably want to connect to the public streams. There are libraries for many languages: https://dev.twitter.com/overview/api/twitter-libraries

So far I've used the Perl Net::Twitter successfully.

Depending on your setup (i.e. account/keywords you pass to the API to look for) you may not get many tweets at the beginning, and so you will need to be connected to the stream for quite some time,

1 Recommendation

Marius Rohde Johannessen

University of South-Eastern Norway

Depending on what you are after, NodeXL can be a good package. Does not require programming skills, it's simply a plugin for Excel. And the guys behind it are helpful if you get stuck.

2 Recommendations

Ecole Internationale des Sciences du Traitement de lInformation

you can use putty and hive to download tweets if you are using twitter Api...That was the apps I used to extract the tweets

http://reference.wolfram.com/language/ref/SocialMediaData.html

2 Recommendations

Mariano Pierantozzi

Università degli Studi G. d'Annunzio Chieti e Pescara

You can have a look at this link.

Mathematica allows you to download and manage a lot of Social Media data, including twitter.

Hope it is useful.

Mariano.

1 Recommendation

Indian Institute of Technology Madras

Thank you all for your kind reply.

@ Marius: Is NODE XL useful for doing sentiment analysis?. I tried NODE XL, it is useful for doing network analysis. I want to use tweets for doing sentimental analysis.

@ Udeh: Is puuty and hive are mobile apps?

@ Mariano: I will try this.

1 Recommendation

Ecole Internationale des Sciences du Traitement de lInformation

you can install an oracle VM as the server and then use putty to connect to it. You need some queries to extract the tweets in hive and then add some json file for transforming the data before using it for sentiment analysis.

1 Recommendation

Indian Institute of Technology Madras

@ Udeh: Thanks a lot. If you have any material related to putty, can you please share with me.

Günther Fliedl

Alpen-Adria-Universität Klagenfurt

Try TWICTION, which is connected to a Twitter Account and realized (what I know) via a Twitter/streaming API.

For further information have a look at: www.econob.com

1 Recommendation

Indian Institute of Technology Madras

@ Gunther Fliedl: Thank you for your reply. Could you please explain how it helps to my study?

Marius Rohde Johannessen

University of South-Eastern Norway

@venkata I'm not sure if NodeXL is the best tool for sentiment analysis. I've only used it for SNA.

Indian Institute of Technology Madras

@ Marius: Thank you.

Timothy Cribbin

Brunel University London

Some good suggestions above, but you may also want to take a look at Chorus Tweetcatcher (TCD) for a simple GUI data collection tool. You can run queries and download user-specific timelines. It can also be used in conjunction with a prototype visual analytics tool called Tweetvis which provides summary statistics and various visualizations to help you explore your datasets.

Chorus is free to download and use: http://chorusanalytics.co.uk

16 Recommendations

Jolly Soparia

Zydus School for Excellence

where can i download the movie reviews where users have given opinions using rating. I tried using rotten tomatoes. But its not working. I want it for Sentiment Analysis. Please any body can help me.

Also suggest me if there are other product sites that allow users to download opinions of rating on movies or any other product.

1 Recommendation

http://cran.r-project.org/web/packages/twitteR/index.html

Telkom University

i suggest you use R package TwitteR (for backward search) or StreamR (for forward search). I found this package is easy to use and we can crawl up to 1500 on one go. i attach the appearance of this TwitteR package

Screen Shot 2015-03-01 at 8.06.03 PM (
2).jpg
472.78 KB

2 Recommendations

Indian Institute of Technology Madras

@ Andry: Thank you for your reply. I have used twitteR package in R software. But i am able to download only 3 tweets. I want minimum 1000 tweets.

Telkom University

@Vankata if you dont specify anything by default twitter search will get 25 last tweet from your search. This is the default :

searchTwitter(searchString, n=25, lang=NULL, since=NULL, until=NULL, locale=NULL, geocode=NULL, sinceID=NULL, retryOnRateLimit=120, ...)

the more variable you put above makes the search is more specific and better accuracy. i am able to get more accurate result by putting for example n=1000, since(date) or sinceID (twitter ID) or geocode ..

1 Recommendation

Indian Institute of Technology Madras

_{@Andy: Thank you for your reply. I am specifying n=1000 still it is giving only 3 tweets.}

Telkom University

@Venkata i forgot one thing the API can not retrieve tweet older than 7 days. other than that i never had any problem to crawl twitter data. by the way you can give me one of your keyword and i will check from my end if i experience the same problem.

Indian Institute of Technology Madras

@Andry: Ya i agree. API will not provide more than 7days tweets. My keyword is stayzilla. Please tell me how many you are able to download.

Ecole Internationale des Sciences du Traitement de lInformation

Do you know programming in R? I will soon publish a project on data visualization in R. So you can use the analysis to download tweets

Indian Institute of Technology Madras

@ Udeh: ya i know R programming.

Ecole Internationale des Sciences du Traitement de lInformation

ok so you can also download tweets using R, then save it and use it for your project

Indian Institute of Technology Madras

@udeh: I am using twitteR package to downlaod tweets. I am able to download only 5 tweets , i want at least 1000 tweets. Do you know how to download?

Ecole Internationale des Sciences du Traitement de lInformation

> library(twitteR)

> library(wordcloud)

> library(RColorBrewer)

> library(plyr)

> library(ggplot2)

> library(sentiment)

# Find OAuth settings for twitter:

> library(httr)

> oauth_endpoints(“twitter”)

## <oauth_endpoint>

## request: https://api.twitter.com/oauth/request_token

## authorize: https://api.twitter.com/oauth/authenticate

## access: https://api.twitter.com/oauth/access_token

# Register an application (API) at https://apps.twitter.com/

# Once done registering, look at the values of api key, secret and token

# Insert these values below:

> api_key <- “ ”

> api_secret <- “ ”

> access_token <- “ ”

> access_token_secret <- “ ”

> setup_twitter_oauth(api_key,api_secret,access_token,access_token_secret)

## [1] “Using direct authentication”

# Now let us collect some tweets (2000 in our example) containing the term '#UDEH_BIGDATA' from twitter (language = English,

# if you wish you can set other languages to fetch tweets in those languages in your analytics)

> tweets = searchTwitter('#UDEH_BIGDATA', n=2000, lang=”en”)

2 Recommendations

Marco Bianchi

Fondazione Ugo Bordoni

You should specify the programming language you are using. Anyway, a list of libraries to interact with the Twitter API is available at https://dev.twitter.com/overview/api/twitter-libraries.

Depending on your needs, you could use:

the REST API (https://dev.twitter.com/rest/public), if you want to execute searches to retrieve tweets matching some specified criteria, or
the STREAMING API (https://dev.twitter.com/streaming/overview), if you need to monitor some keywords or account (https://dev.twitter.com/streaming/reference/post/statuses/filter) or if you want to sample the whole twitter stream (https://dev.twitter.com/streaming/reference/get/statuses/sample).

If you are using Java as programming language, consider that Twitter4J (http://twitter4j.org) is, probably, the best library to interact with Twitter. It is also well-documented.

2 Recommendations

Telkom University

@Venkata sorry for late reply over the weekend ..

your keyword appearance just to few in the twitter, i am only able to gather 184 data when specify n = 1000, i will try on another computer today ..

Indian Institute of Technology Madras

@ Udeh: Sorry for the late reply. I am using the similar code in R. Still i can download only 6 tweets.

@ Marco: Thank you very much for your reply. I am using R Software. I will try.

@ Andry: Thank you very much. It has 2176 tweets. Let me know how many you are able to download.

Noha A. Nagi

Cairo University

Do you have to code for that?

I mean: there are ready made software for this. You have to check Deen Freelon list : https://docs.google.com/document/d/1UaERzROI986HqcwrBDLaqGG8X_lYwctj6ek6ryqDOiQ/edit

This list is curated by Deen but collected in contribution of a lot of internet researchers' chat through the AOIR list (air-l@listserv.aoir.org). "Proudly" I have contributed too in this :)

I also suggest you begin following this list for there are many helpful guys there !

Good luck!

Indian Institute of Technology Madras

@Noha: Thank you for your reply. Definitely i will look into this.

Noha A. Nagi

Cairo University

You have to!

Good luck :)

Jolly Soparia

Zydus School for Excellence

thank you all for your support. Not able to thank all specifically, sorry for that.

really it helped me.

Lone Sorensen

University of Leeds

Hi, I'm in a similar situation. I need historical twitter data for social network analysis. What would be the advantages/disadvantages of scraping tweets from Twitter's advanced search and downloading data from the Internet Archive respectively? Both in terms of coverage/completeness of data set and in terms of the tools and/or programming skills I would need. Thanks.

Dominik Batorski

University of Warsaw

@Lone Sorensen

It seems to me that to obtain full historical data from Twitter is not possible. Please see this paper http://www.sciencedirect.com/science/article/pii/S0378873314000057 for a discussion of biases of different methods of data collection.

1 Recommendation

Ecole Internationale des Sciences du Traitement de lInformation

You can learn the procedures by following the steps in this project https://www.researchgate.net/publication/275947996_BIG_DATA_STATISTICS_WITH_R?ev=prf_pub

Conference Paper BIG DATA STATISTICS WITH R

1 Recommendation

Lone Sorensen

University of Leeds

@Dominik - thanks for the article! I gather that I can't use the API as it won't provide me with historical data (I need to go further back than one week). Hence I'm wondering what the bias is of using Twitter's advanced search, compared to the firehose and/or API. I've seen studies comparing the API and the firehose like the one you posted but nothing about the web search itself. Have you heard of any such studies? And am I right in guessing that the Internet Archive is based on the streaming API?

@Udeh - thanks, this will be really useful as a next step!

Yeimy Arevalo

National University of Colombia

@ Udeh: thank you very much for you R code. That function perfectly!. mayble if someone doesn´t know how to get the api and tocken codes could get it in https://apps.twitter.com .

Fabian Huettig

Universitätsklinikum Tübingen

The Software MAXQDA 12 provides an Interface for selective downloads.

Emma Lee

Peking University

Hi,

You can get twitter data from Podargos Data. All you have to do is to tell them your needs.

Podargos can provide twitter historical and realtime data. In additional, it's able to help you retrieve other mobile App data of social networks and e-commerce, and supports for nearly 100 languages. I hope this information can help you.

Podargos' website is http://www.podargos.com

Here is the data sample:

http://goo.gl/dRxyow

2 Recommendations

Noha A. Nagi

Cairo University

I would also suggest that you check: twitteR

Kamaalini Subramani

Velammal Institute of Technology

Hi venkat....I am also facing the same problem. But i have a solution to download data from facebook using netvizz app. But Now a days i can't download data from FB.

Any one can help me how to download fb comments using netvizz or some other easiest way and also i need .gdf file to import in Gephi.

Please guide me.

Shaheen Khatoon

University of East London

Hi, You can scrap data using "Search Twitter" operator in RapidMiner. With this operator you can specify query and get Twitter statuses containing this query.

Ghulam Asrofi Buntoro

Universitas Muhammadiyah Ponorogo

with R Programming

Alistair.walsh@unimelb.edu.au

Alistair Walsh

University of Melbourne

Hi,

I'm a researcher from University of Melbourne in Australia. The group I work in is scraping Twitter data to analyse aspects of emotion control. After many months we are now collecting tweets and user timelines in a mongoDB database using docker, pymongo and tweepy. Currently have a dataset of 20 million tweets from 8000 individual users and it's growing daily.

Happy to share information as I think helping other researchers will help us.

http://tomkdickinson.co.uk/2016/12/extracting-a-larger-twitter-dataset/

Tiago Santos

ISCTE-Instituto Universitário de Lisboa

If you are looking for information older than a week, a current limitation of the Twitter API, you can crawl the advanced search with the script in the link below.

Emma Lee

Peking University

Hi, you can get twitter data from Podargos Data. It can provide twitter historical and realtime data. In additional, it's able to help you retrieve other mobile App data of social networks and e-commerce, and supports for nearly 100 languages. I hope this information can help you.

Pordargos' website is http://www.podargos.com.

Here is the data sample:

http://goo.gl/dRxyow

https://vimeo.com/170157685

Stuart William Shulman

Texifter

Realtime data collection using the Twitter Search API and the Gnip PowerTrack 2.0 is available using a point and click (no programming) interface at https://discovertext.com. Here are the top 10 reasons to use DiscoverText:

For three free historical Twitter data estimates per day, create a free account on Sifter (https://sifter.texifter.com).

For more information about "Gnip" PowerTrack data, please refer to:

https://discovertext.com/gnip-enabled-access-for-discovertext-users/

academicstrust
dt.png
583.99 KB

Zuheb Siddique Mohammed

Editage

I used this tool a month ago to download twitter data, am not sure if the free trial would let you export 30,000 rows at a time,. Feel free to check it out though https://www.exporttweet.com/

1 Recommendation

Mohamed Ali Saip

University of Bradford

I used KNIME (https://www.knime.com/) to monitor and download Twitter data daily.

Abdullah Önden

Yalova Üniversitesi

If you try Twitter API, you'll face 7 day limitation. Now Twitter offers a Premium API for bulk requests. Still I don't understand why they don't offer any option for educational researches. But still you can still try another options like trimming data from Twitter search services. There's a good option here, even if you don't know Python, this tool will help you to get bulk tweets and also you can get different time spans: https://github.com/Jefferson-Henrique/GetOldTweets-python

And also follow our project SentiTweet: https://www.sentitweet.com

1 Recommendation

Hiba J. Aleqabie

University of Kerbala

streaming twitter API using python language the package of twepy

Jakub Rogalski

Brand24

Have you considered using a social media monitoring tool?

There are many you could test, and I'd advise using a paid tool. Free ones usually lack features that you might actually find the most valuable, whether it's analytics or data visualisation.

As far as my experience goes, the tool I've spent quite a big chunk of my time with is Brand24 (https://brand24.com). It's a media monitoring tool that allows for monitoring both social and traditional web and then exporting the data to .xls files, infographics or .pdf reports.

Coming back to crawling Twitter data, it's very effective due to the access to Twitter's API. However, media monitoring tools are able to collect only those mentions that are publicly available and don't have access restrictions assigned to them. On top of that, historical data is limited, so the tool is the most accurate from the moment of creating the monitoring project onwards.

As far as other tools are concerned, you might also want to have a look at Brandwatch(https://brandwatch.com) or Sysomos (https://sysomos.com)

Hope you'll find this answer helpful!

Alberto Burchi

Università degli Studi di Perugia

Thank you for your contribution. I read with great interest. I am now able to download the tweets as described in some links. Now, I would like to build a database of tweets to be continuously enriched so as to overcome the constraint of the 18,000 tweets and dey 7 days free of the free API. Do you have any advice for me?

https://github.com/Jefferson-Henrique/GetOldTweets-python

Hatice tıraşoğlu

Gazi University

You can use this:

1 Recommendation

Piyush Khatri

Indian Institute of Management Indore

Hi, if you want to download historical tweets related to any search term then you can fetch it from Twitter API for past 7 days.

As per the description provided by you, it looks like that you had already tried that method and received just 3 tweets in past 7 days.

Hence, if you want to go further than 7 days then you will either need to purchase the data directly from Twitter to use its Premium API or purchase it from third party data providers like TrackMyHashtag.

Here is the link to submit your query at TrackMyHashtag: https://www.trackmyhashtag.com/historical-twitter-data

Salar Zrary

University of Mosul

Mr. Venkata Prasad Palakiti

At the end, which website or software did you use to download Twitter data?

Alberto Burchi

Università degli Studi di Perugia

Python and R. Between the two I am more familiar with R but I am moving towards Python: for my uses of artificial intelligence it is more powerful.

Abdelkader Mohamed Elsayed

Good Answer Abdullah Önden

Nisha Arora

You can use Twint. It is an advanced Twitter scraping tool written in Python that allows for scraping Tweets from Twitter profiles without using Twitter's API.

Some of the benefits of using Twint vs Twitter API:

Can fetch almost all Tweets (Twitter API limits to last 3200 Tweets only);
Fast initial setup;
Can be used anonymously and without Twitter sign up;
No rate limitations.

Reference: https://github.com/twintproject/twint

1 Recommendation

Shivani Thakur

Dayalbagh Educational Institute

You ca use scrapy.

Gargy Gupta

Yes absolutely!

There are tools that have the capability to provide all the tweets of any public Twitter account and FollowersAnalysis is one of them.

FollowersAnalysis is a Twitter data analytics platform that can help you download the Tweets from virtually any account. You can get the data in CSV/ Excel format.

Here is the website link: https://www.followersanalysis.com/

You can go through another amazing tool: https://www.trackmyhashtag.com/

Salah Ahmed

SN-University

Dear all!

There are some free and paid tools. In this article, you can get an idea about these tools

https://www.bestproxyreviews.com/twitter-scraper/

If you are familiar with python, I suggest you use one of these packages to craping Data:

Also, read this Q&A, you may find some other tools.

2 Recommendations

Abdelhameed Ibrahim

Dear Salah Ahmed

Thanks for the valuable links.

7 Recommendations

Salah Ahmed

SN-University

Abdelhameed Ibrahim

It was my pleasure :-)

1 Recommendation

Gargy Gupta

Hello sir, I can suggest you an alternative method to collect data from Twitter.

FollowersAnalysis https://www.followersanalysis.com/ is an amazing Twitter analytics tool that can help you extract Twitter data related to any @username or targeted keywords and hashtags. Scrape the past tweets of your brand’s Twitter profile, competitors, influencer marketing candidates, etc.

FollowersAnalysis doesn’t just give you access to Twitter data you also get access to analytical insights and predictive analytics. The comprehensive PDF reports provide actionable insights that can help you make data-driven decisions.

Often brands, marketers, and researchers need custom Twitter datasets for research purposes, competitor analysis, or analyzing the impact of implemented strategies. FollowersAnalysis also offers an option to request custom Twitter datasets which the user can export into Excel CSV or JSON formats.

https://discovertext.com/mentions/

Stuart William Shulman

Texifter

You might browse this list:

Can we use rho_c rather than rho_a to test composite reliability if rho_c meets the threshold while rho_a of some constructs is below 0.7?

In my case, rho_c is 0.7 and above whereas rho_a of some constructs is 0.5 or 0.6. Can I use rho-c in this case if I dont want to remove the items? Please share a valid reference to cite for the same. Regards,

3 answers

Asked 23rd Sep, 2023

Rukhshinda Begum

Seeking Research Team Membership - Machine Learning & Data Science

Discussion

2 replies

Asked 6th Aug, 2023

Fatimah Altuhaifa

Hi there! I'm a Python expert with strong skills in Machine Learning, Data Science, and Data Analysis. I'm eager to join a research team to collaborate on exciting projects. My experience includes supervised/unsupervised learning, and data manipulation using NumPy/Pandas. I'm proficient in statistical analysis. I'm committed to open communication and teamwork for successful outcomes. Let's explore how I can contribute to your research team's endeavors. If you need any person to be in your team to conduct a research and publish it, I will be happy to be in your team.

Hello , why the new citations to my paper don't show in Google Scholar ?

What is the difference between Composite reliability (rho_a) and Composite reliability (rho_c)?

12 answers

Asked 26th Apr, 2023

Hamid Bahadori

Please answer me

In SmartPLS 4's Construct reliability and validity output have Composite reliability (rho_a) and Composite reliability (rho_c).

6 answers

Asked 11th Nov, 2022

Alan Tang

What is the difference between Composite reliability (rho_a) and (rho_c)?

Which one should use for my study?

A research topic for Master's thesis ?

I am looking for a research topic for my Master's thesis in cybersecurity, kindly assist with possible issues and objectives.

8 answers

Asked 2nd Nov, 2022

Alsharif Hasan Mohamad Abu Rbeian

Implementing machine learning in cybersecurity will be preferred.

Thank you all

I don't have institutional access to WoS & Scopus, and can't download data for bibliometrics analysis. What can I do?

Though I have institutional email, I don't have institutional access to WoS & Scopus, and can't download data for bibliometrics analysis. What can I do? or how can I get access into Web of Science/Scopus easily and extract data. Is there any other database that can also be used for the same purpose? Hope you all tell me how to solve this problem.

6 answers

Asked 9th Oct, 2022

Mohammad Kamrul Hasan

N.B: I can use Dimensions.

Thank you.

What threshold should be set for standardized coefficients in a first difference model?

A common threshold for standardized coefficients in structural equation models is 0.1. But is this also valid for first difference models?

8 answers

Asked 5th Apr, 2022

Philipp Brüggemann

Is there any way that I can have access to scopus database for bibliometric analysis?

My university does not have Scopus subscription. Is there any way that I can get the access to database?

5 answers

Asked 23rd Mar, 2022

Rakesh Ahlawat

How can I create Multi-page table over both columns in two-column mode?