JSTOR Lab Text Analyzer – A Summary and a Short Test

JSTOR Labs has developped a new application: http://www.jstor.org/analyze/. It analyzes a text that you upload on the JSTOR-Lab-Site by detecting and priorizing terms, and suggests related documents.


The Text Analyzer is a tool built by http://labs.jstor.org/. It analyzes a text within the document to find key topics and terms, and then uses prioritized terms to find similar content in JSTOR. Results can be adjusted by adding, removing or changing the importance of the prioritized terms. Find here first a summary following the information of the JSTOR-Site and second the results of a short test:

The Analyzer performs its processing in three steps:

The first step involves text extraction from a document. JSTOR says that once text extraction has been performed the document is completely removed from its system.

The second step consists of different parallel text analyzes:

  • Topics explicitly mentioned in the text are identified with a huge vocabulary.
  • Latent topics are inferred using an LDA topic model trained with JSTOR content.
  • Persons, locations and organizations are identified using multiple entity recognition tools like Alchemy, OpenCalais, the Stanford Named Entity Recognizer, and Apache OpenNLP. Named entities recognized by the individual services are aggregated and ranked to estimate the relative importance of the entity to the source document.
  • The top topics found are displayed and the five most significant terms are then used in the initial seed query. The tooltip will identify whether the term was obtained from the source text or whether it was derived from co-occurring terms in similar documents. In all cases the value of the calculated weight is provided.

In a third step, the five most significant topics and recognized named entities are used in a query to identify similar content from the JSTOR archive.  Selected documents are then ranked using a scoring calculation that considers both the weight of the term from the input text and the importance of the term in the selected document. The preselected terms can be removed and new terms can be added.

The five terms identified in the source document should be viewed as a starting point and adjusted for a specific area of interest. The tool is designed to be used in an iterative process. A new query is automatically performed after any change in the query terms. In that way the terms in a query are continually updated to reflect current preferences.

I tested my book of 2016: Geschichte Digital – Historische Welten neu vermessen. And indeed: JSTOR Lab found some documents (articles, books) which should be taken into account for its issue. So Jan Hodel is quite right in his review stating, that there are some gaps in the book – see: http://geschichtefuerheute.de/, No 1, 2017. Some of them are due to the rapid development of the Digital Humanities. So, when writing books or articles, Text Analyzer of JSTOR can help to cope with this rapid change.

Actually, there is one important restriction. JSTOR is working for English texts. My books is in German. However, it did work as for the book contains many English terms like Digital History, Digital Humanities etc. Naturally, I had to change a lot of terms.

Guido Koller

Senior Historian, Swiss Federal Archives, CH-3003 Berne, Switzerland

More Posts - Website

Follow Me:

Leave a Reply

Your email address will not be published. Required fields are marked *