[FIXED] Change nltk.download() path directory from default ~/ntlk_data

Issue

I was trying to download/update python nltk packages on a computing server and it returned this [Errno 122] Disk quota exceeded: error.

Specifically:

[nltk_data] Downloading package stop words to /home/sh2264/nltk_data...
[nltk_data] Error downloading u'stopwords' from
[nltk_data] <https://raw.githubusercontent.com/nltk/nltk_data/gh-
[nltk_data] pages/packages/corpora/stopwords.zip>: [Errno 122]
[nltk_data] Disk quota exceeded:
[nltk_data] u'/home/sh2264/nltk_data/corpora/stopwords.zip
False

How could I change the entire path for nltk packages, and what other changes should I make to ensure errorless loading of nltk?

Solution

This can be configured both by command-line (nltk.download(..., download_dir=) or by GUI. Bizarrely nltk seems to totally ignore its own environment variable NLTK_DATA and default its download directories to a standard set of five paths, regardless whether NLTK_DATA is defined and where it points, and regardless whether nltk's five default dirs even exist on the machine or architecture(!). Some of that is documented in Installing NLTK Data, although it's incomplete and kinda buried; reproduced below with much clearer formatting:

Command line installation

The downloader will search for an existing nltk_data directory to install NLTK data. If one does not exist it will attempt to create one in a central location (when using an administrator account) or otherwise in the user’s filespace. If necessary, run the download command from an administrator account, or using sudo. The recommended system location is:

• C:\nltk_data (Windows) ;
• /usr/local/share/nltk_data (Mac) and
• /usr/share/nltk_data (Unix).

You can use the -d flag to specify a different location (but if you do this, be sure to set the NLTK_DATA environment variable accordingly).

• Run the command python -m nltk.downloader all

• To ensure central installation, run the command: sudo python -m nltk.downloader -d /usr/local/share/nltk_data all

• But really they should say: sudo python -m nltk.downloader -d $NLTK_DATA all

Now as to what recommended path NLTK_DATA should use, nltk doesn't really give any proper guidance, but it should be a generic standalone path not under any install tree (so not under <python-install-directory>/lib/site-packages) or any user dir. Hence, /usr/local/share, /opt/share or similar. On MacOS 10.7+, /usr and thus /usr/local/ these days are hidden by default, so /opt/share may well be a better choice. Or do chflags nohidden /usr/local/share.

Answered By - smci