Emoji Support in ExpressionEngine

Sure, emoji are ? right now. Anytime a phone vendor adds or updates their emoji, it makes headlines. We all have friends (or are that person) that the new emoji are the compelling reason to upgrade. What’s the draw?

Humans have been been using ideograms for centuries, from heiroglyphs to modern city signs. Text based emoticons (smileys, like :wink: and :stuck_out_tongue:) surfaced in the early 80s as a useful way to add emotional context to online communication. Image-based emoticon popularity surged with internet forums. Emoji seem to bridge the gap between all three. They can add a simple straightfoward emotion, represent literal objects, or they can stand in for an abstract concept. They’re used by young and old, across all cultures, both formally and informally.

For example, for those who don’t want to know any of the technical details, we can flag that as skippable for them in this article with two emoji: ??.

At first it may have felt like emoji may be a trend, a passing fad of youthful communication. But the solidification of over 2500 emoji into the Unicode standard means they are here to stay. They will certainly evolve along with language, but we are past the tipping point where it’s optional for a software developer to care. Especially one that focuses on empowering its customers to publish their universe, like us.

Since it meant increasing our system requirements, we couldn’t support emoji any sooner than our next major version. So, ExpressionEngine 4 brings official support for emoji.

??: Emoji are represented by multi-byte characters, and MySQL was not able to store them properly until version 5.5.3. Specifically, a superset of utf8 collation that uses up to four bytes to store each character—utf8mb4—is required. ExpressionEngine 4 increased the minimum MySQL requirement to 5.5.3 so it can use utf8mb4 by default.

Upgrading from earlier versions

Converting a database to allow it to store emoji is fairly straightforward. But it sometimes needs some manual tweaking, and if you have a large database, it might take a very long time to complete the operation. So a v4 upgrade does not automatically convert your database. Instead we made a standalone utility that will perform the conversion for you. You can download it here, and run it whenever you’re ready.

On an average install, it will complete in seconds. On a ridiculously large test database, the backup and conversion took about 20-30 minutes total for us. If you’ve ever had to take your site offline to perform a MySQL ALTER command, then you know you fall into that latter camp. If you aren’t sighing and nodding your head knowingly at ?, you will probably be able to run the conversion in seconds. The conversion tool will also warn you if there are third-party tables that require modifications before you can convert.

??: Character limits in MySQL are expressed in terms of 3-byte characters. Because emoji take 4 bytes to store, table indexes that might contain them can only hold 191 characters instead of 255. ⌊255 * 3 / 4⌋ = 191. Maths, oy. The conversion utility will spell it out for you, and also help add-on developers be aware of what steps they need to take to support utf8mb4.

The conversion utility was made for ExpressionEngine 3 since it will mostly be used by folks who are upgrading. It will also work on v4 for those who have already upgraded. (Fun fact: it required no changes for v4 compatibility.) This means it can even enable your ExpressionEngine 3 installation to store emoji, addressing our all-time favorite bug report, titled: “EE can’t handle the Poo emoticon (and many others)”. But until you upgrade to v4 you won’t have access to the coolest features:

Emoji Shorthand Codes

Storing emoji in the database properly is fine. But we also wanted to give content authors a simpler, more natural way to use emoji than having to open a character palette. So we decided to support emoji shorthand codes. If you’re a Slack or GitHub user, you are already familiar with them. You surround an emoji’s descriptive name with colons, and it renders the emoji. So I can type :joy: and display the most popular emoji of all, ?. Or I can talk about how writing this blog post led me down a :rabbit::hole: (??) of research on emoji usage, statistics, cultural differences, and even a podcast talking about the process of submitting and getting an emoji accepted into the Unicode canon.

Any field that has formatting performed on it (Auto HTML, Markdown, etc.) will parse emoji shorthand automatically. Entries, comments, forum posts, wiki articles, add-ons, wherever. The source for short names comes from the Emoji Catalog, and at times it is slightly behind Unicode. For instance, right now in iOS 11 there are some sweet new emoji like ?‍♀️, ?, ?, and ?. If you’ve been looking for them on Slack and missing them, this is why. We use the same datasource for shorthand, and it takes a little bit to update after new emoji are added to the lexicon.

Emoji Template Tags

Lastly we added an Emoji module that gives you some template tags to work with emoji. You might build your own front-end emoji picker with the {exp:emoji:emoji_list} tag:

{exp:emoji:emoji_list}
    <div class="item"><a href="#" onclick="addEmoji(':{short_name}:');">{html_entity}</a></div>
{/exp:emoji:emoji_list}

Or you might parse emoji shorthand codes from content sourced outside of ExpressionEngine with {exp:emoji:parse_shorthand}:

{exp:emoji:parse_shorthand}
  Flying a :rocket: to Mars with Elon Musk
{/exp:emoji:parse_shorthand}

?

What are you waiting for? Run the converter, upgrade to ExpressionEngine 4 and unleash all the feels!