[TfTi] Will the AI Overlords Be Reviewing Your Book?
Imagine getting an email with the subject line, “An AI wrote a review of your book.”
Would you be excited? Baffled? Afraid?
That’s not quite what happened to me, but I think my recent experience is similar enough to offer some lessons for indie book authors — and anyone else with a product or service they are marketing online.
Lessons Twenty Years in the Making
A lifetime ago, shortly after I met Yvonne and before we’d even started in the book publishing industry together, I had an article published as the cover story in the June 2003 New York State Bar Association Journal, entitled “Beyond Words: New Tools Can Enhance Legal Writing” (reprint copy available here).
At some point in that dim past, I uploaded a copy of the article to Academia.edu, a site for academics to share their work that boasts, “Papers uploaded to Academia get 69% more citations.” And I will say that my article did get at least one complimentary citation in a footnote to an article entitled, “Painting with Print: Incorporating Concepts of Typographic and Layout Design into the Text of Legal Writing Documents,” by Law Professor Ruth Anne Robbins, in the Fall 2004 issue of the Journal of the Association of Legal Writing Directors (open the PDF and see fn. 3, pg. 110).
I admit I was quite proud of that citation, though by then Yvonne and I were well on the way to starting our first venture into what’s now called hybrid book publishing.
Fast forward more that twenty years and I get that email with the subject line, “An AI wrote a review of your paper BEYOND WORDS.” I was baffled and curious. What would an AI review look like?*
So the first lesson is to get your work out there. You can never tell who — or what — may see it and engage. Nor how long it may take!
Can AI write reviews?
Further lessons require taking a look at what the AI did. But first I want to mention another arrival in my inbox, linking to a podcast with our friend and author marketing expert, Penny Sansieveri, entitled “The Pay-to-Play Dilemma: Paying for book reviews & coverage.”
One of the points she makes is that it’s a growing — and to me troubling — trend for book reviewers to use AI bots to do the reading and writing, warning:
“Some authors have received AI-generated blurbs that spoil key plot points or sound robotic—making them unusable for promotion.”
It seems to me unprofessional for a paid reviewer to put their name on a review they didn’t write, not to mention having the machine do the reading, as well.
There’s the second lesson: Ask up front if the reviewer or review site you’re considering uses AI to generate any or all of the text in their reviews. Because an AI generated book review could do more harm than good.
What the AI got out of my article
The general overview that the AI wrote reads like an academic abstract. It used the summary headline that the editors put on the cover of the journal, in place of the actual article title shown above. But it does capture the main points my co-author and I were making:
“‘Beyond Words: The Role of Graphics’ by Thomas G. Collins and Karin Marlett, explores the transformative impact of visual aids in legal writing. The article advocates for integrating visuals like charts and diagrams within legal documents to enhance clarity and retention, building on Herald Price Fahringer’s advice from the 1980s. The authors argue that technological advancements now allow for easy incorporation of high-quality graphics, which facilitates clearer communication. They aim to shift the legal profession’s traditionally text-centric focus towards embracing visuals as fundamental tools for effective advocacy.”
No real harm, but no real complimentary snippets to quote in marketing content, either. The AI summary lacks any words or phrases connecting with human emotion.
The next section, Relevant References, hints at the real purpose of this review, critiquing the article to make it more attractive for submission to scholarly journals. A bit odd, since it was already published in the journal directly related to the audience I was seeking at the time.
The AI found ten articles relating to the use of graphics or design principles in legal writing. All were published after mine, so I could not have cited them. But then an all too common problem of AI output reared its ugly head. Only five of the cited articles had links to them and two of them did not work.
By strange coincidence, one of the broken links was to Robbins’ “Painting with Print” article, delivering me to a 404 page. Since I knew that one existed, I was able to do my own search and find the link I used above. The second broken link went to the Southwestern Law School website and its general law review page, with no apparent way to find the specific article.
I probably could have found it, and perhaps the other five with no links. But there have been enough cases of lawyers using AI for research and being disciplined by courts for citing non-existent cases and authorities that had been confidently made up by ChatGPT for me to repeat this lesson from an earlier post: verify every source cited by your AI tool.
In the next section called Strengths and a later one called Reviewer Commentary, the AI did provide several quotable nuggets. But even with those, I found myself wondering if I had wanted to use them in marketing, who would I credit them to? I find it hard to believe anyone would be impressed by:
“The authors make a compelling case for the use of graphics in legal writing, effectively demonstrating both the theoretical and practical benefits.”
— AI Reviewer at Academia.edu
Where the AI lost its credibility
But it was in its more specific criticisms that the AI really went off the rails. In one spot, it asserted,
“Some terms, such as ‘graphical excellence,’ are used without precise definition or source attribution. Clarifying these terms early in the article or including footnotes with definitions would help readers unfamiliar with the subject matter.”
Really? No definition or sourcing? Here’s a clipping from the article, where I used the term:
And here are the footnotes:
The AI elsewhere asserted:
“Establishing guidelines or best practices for ensuring the integrity of visual aids in legal documents could significantly strengthen the work’s practical applicability.”
I won’t get into the details, but under that heading “Practicing Graphical Excellence” I had in fact provided eight specific “guidelines or best practices,” with these subheadings:
- Know why you’re using a graphic
- Force visual comparison
- Show causality
- Capture complexity
- Keep graphics adjacent to the text being illustrated
- Minimize non-data ink
- Use small multiples
- Use color with a purpose
Each of those had as many as seven paragraphs of explanatory text, altogether filling more than three pages of the article. That portion included footnotes 28 through 44.
And that list of “dos” was preceded by a crucial “don’t”: “don’t let your table, graph, drawing, or picture distort or misrepresent the information” — citing Edward Tufte’s extensive treatment of “graphical integrity.”
Don’t let an AI muddle your marketing integrity
In the end, I was left wondering whether the AI had actually read “Beyond Words.” Or had it simply extrapolated from the title what it must have been about and then generated its criticisms from a generic list of strengths and weaknesses of academic papers in its LLM.
That would mean, not only had the review site abdicated responsibility for having a human read the piece to be reviewed, but its AI was also too “lazy” to bother!
Paid reviews are fine. Sites like Kirkus Reviews, BookLife (by Publishers Weekly), Online Book Club, and others offer reviews by their stable of human reviewers who receive part of the fee.
I think your work is too important to leave crucial marketing materials to the mechanical output of an AI reviewer. Take the time to understand what you are paying for.
* The AI review is behind a login screen, although there is a free membership; but on the chance that you have or want to create an account at Academia.edu, here’s the link to its unsolicited AI review: https://www.academia.edu/ai_review/36061043
Leave a Reply