mRpostman Basics

Introduction

The goal of mRpostman is to make it easy to connect to your IMAP (Internet Message Access Protocol) server and execute commands, such as list mailboxes, search for and fetch messages in a tidy way.

In this vignette we will present all available functions of this package, but not all the possiilities one can explore.

First things first - Allowing Less Secure Apps Access

Before using mRpostman, it is essential to configure your mail account. Many mail providers today require authorizing “less secure apps” to access your account from a third part app.

See how to do it for Gmail, Yahoo Mail and AOL Mail.

Gmail

Go to Gmail website and log in with your credentials.
Then, go to https://myaccount.google.com/u/1/lesssecureapps?pageId=none

Set “Allow less secure apps” to ON.

Yahoo Mail

Go to Yahoo Mail website and log in with your credentials.
Click on “Account Info”.

Click on “Account Security” on the left menu.

After, set “Allow apps that use less secure sign in” ON

AOL Mail

Go to AOL Mail website and log in with your credentials.
Click on “Options” and then on “Account Info”.

Click on “Account Security” on the left menu.

After, set “Allow apps that use less secure sign in” ON

Package Structure

The package is divided in 7 groups of functions:

configuration: configure_imap();
mailboxes commands: list_mailboxes(), select_mailbox(), examine_mailbox(), rename_mailbox();
options listing: list_server_capabilities(),flag_options(), section_or_field_options(), metadata_options();
search: search_before(), search_since(), search_period(), search_on(), search_sent_before(),search_sent_since(), search_sent_period(), search_sent_on(), search_string(), search_flag(), search_smaller_than(), search_larger_than(), search_younger_than(), search_older_than(), custom_search();
custom search helper functions:
- relational operators functions: AND(), OR();
- criteria definition functions: before(), since(), on(), sent_before(), sent_since(), sent_on(), string(), flag(), smaller_than(), larger_than(), younger_than(), older_than();
fetch: fetch_full_msg(), fetch_msg_header(), fetch_msg_text(), fetch_msg_metadata();
miscellania: copy_msg(), get_min_id(), get_max_id(), delete_msg(), expunge(), add_flags(), remove_flags(), replace_flags(), move_msg().

How do I start?

1) Configuration

After enabling (if needed) “Non secure apps access” in your Mail provider, you have to configure your IMAP settings:

library(mRpostman)

# IMAP settings
# Gmail
imapconf <- configure_imap(url="imaps://imap.gmail.com",
                          username="your_user",
                          password=rstudioapi::askForPassword(),
                          verbose = TRUE
                          )

# Yahoo Mail
# imapconf <- configure_imap(url="imaps://export.imap.aol.com/",
#                           username="your_user",
#                           password=rstudioapi::askForPassword()
#                           )

# AOL Mail
# imapconf <- configure_imap(url="imaps://export.imap.aol.com/",
#                           username="your_user",
#                           password=rstudioapi::askForPassword()
#                           )

# you can try another IMAP server

2) Mailbox Commands

2.1) Listing Mailboxes

# Listing
imapconf %>%
  list_mailboxes()

From now on, you will have to select a mailbox to issue further commands.

2.2) Selecting a Mailbox

imapconf %>%
  select_mailbox(mbox = "INBOX") # be careful - case sensitive

2.3) Examining a Mailbox

Select a mailbox and count the number of existent and recent messages.

imapconf %>%
  select_mailbox(mbox = "UC Riverside") %>% # be careful - case sensitive
  examine_mailbox()

2.4) Renaming a Mailbox

imapconf %>%
  select_mailbox(mbox = "CRAN messages") %>% # be careful - case sensitive
  rename_mailbox(new_name = "CRAN messages2") %>%
  list_mailboxes() # and list again to check

3) Options Listing

Before executing search and fetch commands, it is recommended to check which are your options regarding to flags, message’s section and fields, and you server capabilities as well.

3.1) Server capabilities

Knowing you server capabilities is important when searching because you can execute an optimized search using “ESEARCH” extension. Also, if your server has the “WITHIN” extension, you can use search_younger_than() and search_older_than() functions.

If your server provides “MOVE” capabilities, you can use move_msg() for moving messages between two mailboxes.

Note that Gmail does not provide the “WITHIN” extension.

imapconf %>%
  list_server_capabilities()

3.2) Flag Options

Flags work like tags attached to messages. You can check the options with:

flag_options()

3.2) Section and Fields Options

To see which section and/or fields of a message you can specify in some searches or when fetching a message, use:

section_or_field_options()

3.3) Metadata Options

You can search for a specific content in a message’s metadata as well. Check your options:

metadata_options()

4) Search

All search functions will return a list with an imapconf object and a “msg_id” vector containing the results from search. The first object will be important when you want to put all the commands in a chain and perform a fetch after searching for messages. In this case, you will need the IMAP configuration object (imapconf) and the message’s ids (msg_id).

If you do not want to return the imapconf object, but only the search results, you can set return_imapconf = FALSE in any of the search functions.

You can also NEGATE all search expressions by setting negate = TRUE.

If your server supports ESEARCH, use it. It will prevent your results from being stripped. With “ESEARCH”, the results will be condensed to groups of sequence similar to what R does. For instance, if your search returns 1000 results, it is better having condensed groups such as 1:10, 12, 23:27, ... instead of a sequence of 1, 2, 3, 4, 5, 6, ..., 10, 12, 23, 24, 25, .... If you can’t use ESEARCH, or if your results are being stripped even with ESEARCH, you can try to increase your buffersize in configure_imap() in order to avoid stripping. This is related to curl’s buffersize.

4.1) Search by Date

search_before(), search_since(), search_on(), and search_period() use internal date, which reflects the moment when the message was received. search_sent_before(), search_sent_since(), search_sent_on(), and search_sent_period() use RFC-2822 date header (origination date), which “specifies the date and time at which the creator of the message indicated that the message was complete and ready to enter the mail delivery system” (Resnick, 2008). Dates in both methods must be the same most of time. Nonetheless, using internal date for search is faster (Babcock, 2016).

4.1.1) By internal date

4.1.1.1) Before a date

result <- imapconf %>%
  select_mailbox(mbox = "CRAN messages2") %>%
  search_before(date_char = "17-Apr-2019", esearch = TRUE)

result$msg_id

You can use UID (unique identifier) instead of MSN (message sequence number) and a flag also.

result <- imapconf %>%
  select_mailbox(mbox = "UC Riverside") %>%
  search_before(date_char = "02-May-2019", 
               by = "UID", 
               flag = "UNANSWERED",
               esearch = TRUE)

result$msg_id

Remember, if you do not want to return the imapconf object, but only the search results, you can set return_imapconf = FALSE in any of the search functions.

result <- imapconf %>%
  select_mailbox(mbox = "UC Riverside") %>%
  search_before(date_char = "02-May-2019", 
               by = "UID", 
               flag = "UNANSWERED",
               esearch = TRUE, return_imapconf = FALSE)

result

You can also NEGATE the statement to search for messages NOT BEFORE a date:

results <- imapconf %>%
  select_mailbox(mbox = "UC Riverside") %>%
  search_before(date_char = "02-May-2019", 
               negate = TRUE, 
               by = "UID", 
               flag = "UNANSWERED",
               esearch = TRUE)

results$msg_id

4.1.1.2) Since a date

results <- imapconf %>%
  select_mailbox(mbox = "CRAN messages2") %>%
  search_since(date_char = "17-Apr-2019",
              esearch = TRUE)

4.1.1.3) By period

results <- imapconf %>%
  select_mailbox(mbox = "CRAN messages2") %>%
  search_period(since_date_char = "02-Jan-2019",
               before_date_char = "30-Jun-2019",
               flag = "ANSWERED",
               esearch = TRUE)

You can NEGATE period searches too:

results <- imapconf %>%
  select_mailbox(mbox = "CRAN messages2") %>%
  search_period(since_date_char = "17-Jun-2019",
               before_date_char = "30-Jun-2019", 
               negate = TRUE,
               flag = "ANSWERED",
               esearch = TRUE)

4.1.1.4) On a specific date

results <- imapconf %>%
  select_mailbox(mbox = "CRAN messages2") %>%
  search_on(date_char = "30-Apr-2019", 
           flag = "SEEN",
           esearch = TRUE)

4.1.2) By origination date

4.1.2.1) Sent Before a date

results <- imapconf %>%
  select_mailbox(mbox = "CRAN messages2") %>%
  search_sent_before(date_char = "17-Apr-2019")

results$msg_id

You can use UID (unique identifier) instead of MSN (message sequence number) and a flag also.

results <- imapconf %>%
  select_mailbox(mbox = "UC Riverside") %>%
  search_sent_before(date_char = "02-Jan-2019", 
                   by = "UID", 
                   flag = "UNANSWERED",
                   esearch = TRUE)

You can also NEGATE the statement to search for messages NOT SENTBEFORE a date:

results <- imapconf %>%
  select_mailbox(mbox = "UC Riverside") %>%
  search_sent_before(date_char = "02-May-2019", 
                   negate = TRUE, 
                   by = "UID", 
                   flag = "UNANSWERED",
                   esearch = TRUE)

4.1.2.2) Sent Since a date

results <- imapconf %>%
  select_mailbox(mbox = "CRAN messages2") %>%
  search_sent_since(date_char = "17-Apr-2019")

4.1.2.3) Sent by period

results <- imapconf %>%
  select_mailbox(mbox = "CRAN messages2") %>%
  search_sent_period(since_date_char = "17-Jun-2019",
               before_date_char = "30-Jun-2019",
               flag = "ANSWERED")

You can NEGATE period searches too:

results <- imapconf %>%
  select_mailbox(mbox = "CRAN messages2") %>%
  search_sent_period(since_date_char = "17-Jun-2019",
               before_date_char = "30-Jun-2019", negate = TRUE,
               flag = "ANSWERED")

4.1.2.4) Sent On a specific date

results <- imapconf %>%
  select_mailbox(mbox = "CRAN messages2") %>%
  search_sent_on(date_char = "30-Apr-2019", 
               flag = "SEEN",
               esearch = TRUE)

4.2) Search by String

You can search for a simple or compound string either in the whole message, in a section, or in a specific header field. You can also NEGATE the statement and search for messages (or specific part of a message) not containing that string.

When searching by string, it is not possible to use flags.

Searching “TO” field:

results <- imapconf %>%
  select_mailbox(mbox = "CRAN messages2") %>%
  search_string(section_or_field = "TO", string = "hadley@rstudio.com",
               negate = TRUE) # not TO "hadley@rstudio.com"

results$msg_id

Searching “TEXT” section.

results <- imapconf %>%
  select_mailbox(mbox = "UC Riverside") %>%
  search_string(section_or_field = "TEXT", string = "Dear Allan")

To search for the string in the whole message, use section_or_field = "BODY".

results <- imapconf %>%
select_mailbox(mbox = "UC Riverside") %>%
  search_string(section_or_field = "BODY", string = "Dear Allan")

Check section_or_fields_options() to see which fields and sections you can specify.

4.3) Search by Flag

results <- imapconf %>%
  select_mailbox(mbox = "INBOX") %>%
  search_flag(flag = "RECENT", by = "UID")

Check flag_options() to see which flags you can specify.

4.4) Search by Size

The size is specified in bytes.

4.4.1) Smaller Than

results <- imapconf %>%
  select_mailbox(mbox = "INBOX") %>%
  search_smaller_than(size = 512000) # smaller than 512KB

4.4.1) Larger Than

results <- imapconf %>%
  select_mailbox(mbox = "INBOX") %>%
  search_larger_than(size = 512000, esearch = TRUE) # Larger than 512KB

4.5) Search by Within Extension

Severs who support “WITHIN” EXTENSION enable searching for messages within a span of time, i.e. younger than “x” seconds, or older than “x” seconds.

4.5.1) Younger Than

results <- imapconf %>%
  select_mailbox(mbox = "INBOX") %>%
  search_younger_than(seconds = 3600) # msgs received in less than one hour (3600 sec)

4.5.2) Older Than

results <- imapconf %>%
  select_mailbox(mbox = "INBOX") %>%
  search_older_than(seconds = 3600) # msgs received in not less than one hour (3600 sec)

4.6) Custom Search

custom_search() and its helper functions allow you to create a vast number of customized search by combining different criteria using all the types of searches presented before.

These are the helper functions you can use inside custom_search():

Relational operators: AND(), and OR();
Criteria definition: before(), since(), on(), sent_before(), sent_since(), sent_on(), string(), flag(), smaller_than(), larger_than(), younger_than(), and older_than().

NOTE: IMAP queries follows Polish notation, i.e. operators such as OR come before arguments, e.g. “OR argument1 argument2”. Therefore, the relational operators functions in this package should be used like the following examples: OR(before("17-Apr-2015"), string("FROM", "Jim")). Even though there is no “AND” operator in IMAP, this package adds a helper funcion AND() to indicate multiples arguments that must be searched together, e.g. AND(since("01-Jul-2018"), larger_than(16000)).

Example 1: Searching for messages containing the string “toronto” in the “FROM” field AND that were received before “12-Apr-2019”.

results <- imapconf %>%
  select_mailbox(mbox = "INBOX") %>%
  custom_search(custom_request = 
                 AND(
                   string(section_or_field = "FROM", string = "Toronto"),
                   before(date_char = "12-Apr-2019")
                   )
               )

results$msg_id

Example 2: Searching for messages containing the string “Contract 2872827” in the “TEXT” section OR the string “Buying operation” in the “Subject” field of the “HEADER”.

results <- imapconf %>%
  select_mailbox(mbox = "INBOX") %>%
  custom_search(custom_request = 
                 OR(
                   string(section_or_field = "TEXT", string = "Contract 2872827"),
                   string(section_or_field = "Subject", string = "Buying operation")
                   )
               )

results$msg_id

5) Fetch

You can fetch full messages, message headers, message texts, or only metadata field(s) of messages.

We usually fetch messages after a search. Given the output of search functions in mRpostman, you will have to use the exposition pipe %$% in order to retrieve and correctly map the imapconf object and the msg_id (search results).

You can also choose to write fetch results to disk (working directory) using write_to_file = TRUE. If you choose to do so, mRpostman creates a folder with the mailbox name in your wd and save the parts of the messages you are fetching to text files using each message sequence number (MSN) or unique identifier (UID) as names.

5.1) Fetch Full Message

results <- imapconf %>%
  select_mailbox(mbox = "UC Riverside") %>%
  search_before(date_char = "10-Mar-2019", by = "UID") %$% #exposition pipe operator
  fetch_full_msg(imapconf = imapconf, msg_id = msg_id,
                 by="UID", write_to_file = TRUE, keep_in_mem = TRUE,
                 partial = "0.789")

Do not forget to re-specify by = "UID" inside fetch_full_msg().

5.2) Fetch Message Header

results <- imapconf %>%
  select_mailbox(mbox = "UC Riverside") %>%
  search_since(date_char = "15-Aug-2019", by = "UID") %$% #exposition pipe operator
  fetch_msg_header(imapconf = imapconf, 
                   msg_id = msg_id, 
                   fields = c("DATE", "SUBJECT"),
                   by = "UID")

results

5.3) Fetch Message Text

results <- imapconf %>%
  select_mailbox(mbox = "INBOX") %>%
  search_since(date_char = "17-Aug-2019", by = "UID") %$% #exposition pipe operator
  fetch_msg_text(imapconf = imapconf, msg_id = msg_id, by = "UID", try_b64decode = TRUE)

5.4) Fetch Message Metadata

results <- imapconf %>%
  select_mailbox(mbox = "INBOX") %>%
  search_on(date_char = "10-may-2019", by = "UID") %$% #exposition pipe operator
  fetch_msg_metadata(imapconf, msg_id = msg_id, by = "UID",
                   metadata = c("INTERNALDATE", "UID", "ENVELOPE"))

You can check metadata options with metadata_options().

6) Miscellanea

Here we present other functions to perform useful IMAP operations.

6.1) Copy Message(s)

Copying seach results form “CRAN messages2” to “INBOX” mailbox:

results <- imapconf %>%
  select_mailbox(mbox = "CRAN messages2") %>%
  search_since(date_char = "10-may-2019") %$% #exposition pipe operator
  copy_msg(imapconf = imapconf, msg_id = msg_id, to_mbox = "INBOX")

6.2) Get minimum (smaller) message ID with specific flag

results <- imapconf %>%
  select_mailbox(mbox = "[Gmail]/Trash") %>%
  get_min_id(flag = "UNSEEN")

6.3) Get maximum (larger) message ID with specific flag

results <- imapconf %>%
  select_mailbox(mbox = "[Gmail]/Trash") %>%
  get_max_id(flag = "UNSEEN")

6.4) Delete message(s)

results <- imapconf %>%
  select_mailbox(mbox = "[Gmail]/Trash") %>%
  search_before(date_char = "10-may-2012", by = "UID") %$% #exposition pipe operator
  delete_msg(imapconf = imapconf, msg_id = msg_id)

Deleting a specific “msg_id” without a previous search:

results <- imapconf %>%
  select_mailbox(mbox = "INBOX") %>%
  delete_msg(msg_id = 66128)

6.5) Expunge

Expunges message(s) marked with the “DELETED” flag in a mailbox or a specific message using the specific_UID attribute.

results <- imapconf %>%
  select_mailbox(mbox = "[Gmail]/Trash") %>%
  delete_msg(msg_id = 71171, by = "UID") %$%
  expunge(imapconf = imapconf, specific_UID = msg_id)

6.6) Add/Remove/Replace Flags

Adding, removing and replacing one or more flags to messages.

6.6.1) Add Flags

results <- imapconf %>%
  select_mailbox(mbox = "INBOX") %>%
  search_since(date_char = "18-Aug-2019", by = "UID") %$% #exposition pipe operator
  add_flags(imapconf = imapconf, msg_id = msg_id, flags_to_set = "RECENT")

You cannot add a flag that is the antonym of an already existent flag in the message, e.g. cannot add “UNSEEN” flag when a message already has flag “SEEN”. Use replace_flags() instead.

6.6.2) Remove Flags

results <- imapconf %>%
  select_mailbox(mbox = "INBOX") %>%
  search_since(date_char = "18-Aug-2019", by = "UID") %$% #exposition pipe operator
  remove_flags(imapconf = imapconf, msg_id = msg_id, flags_to_unset = "RECENT")

6.6.3) Replace Flags

results <- imapconf %>%
  select_mailbox(mbox = "INBOX") %>%
  search_before(date_char = "10-may-2012", by = "UID") %$% #exposition pipe operator
  replace_flags(imapconf = imapconf, msg_id = msg_id, 
               flags_to_set = flags_to_set = c("SEEN", "DRAFT"))

6.7) Move Message(s)

move_msg() uses IMAP “MOVE” EXTENSION. Check if your server supports “MOVE” capability with list_server_capabilities().

# copy search results from "Sent" to "INBOX"
results <- imapconf %>%
  select_mailbox(mbox = "[Gmail]/Sent Mail") %>%
  search_before(date_char = "10-may-2012") %$% #exposition pipe operator
  move_msg(imapconf = imapconf, msg_id = msg_id, to_mbox = "CRAN messages2")

If your server does not provide “MOVE” capability, the same result can be achieved with a combination of add_flags() and expunge():

results <- imapconf %>%
  select_mailbox(mbox = "[Gmail]/Sent Mail") %>%
  search_before(date_char = "10-may-2012") %$% #exposition pipe operator
  copy_msg(imapconf = imapconf, msg_id = msg_id, to_mbox = "CRAN messages2") %$%
  add_flags(imapconf = imapconf, msg_id = msg_id, flags_to_set = "Deleted") %$%
  expunge()

References

Babcock, N., Introduction to IMAP, Blog, May 2016, https://nbsoftsolutions.com/blog/introduction-to-imap.

Crispin, M., INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1, RFC 3501, DOI: 10.17487/RFC3501, March 2003, https://www.rfc-editor.org/info/rfc3501.

Ooms, J. curl: A Modern and Flexible Web Client for R. R package version 3.3, 2019, https://CRAN.R-project.org/package=curl

Stenberg, D. Libcurl - The Multiprotocol File Transfer Library, https://curl.haxx.se/libcurl/

Freed, N. and N. Borenstein, Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types, RFC 2046, DOI: 10.17487/RFC2046, November 1996, https://www.rfc-editor.org/info/rfc2046.

Resnick, P., Ed., Internet Message Format, RFC 2822, DOI: 10.17487/RFC2822, April 2001, https://www.rfc-editor.org/info/rfc2822.

Resnick, P., Ed., Internet Message Format, RFC 5322, DOI: 10.17487/RFC5322, October 2008, https://www.rfc-editor.org/info/rfc5322.