PowerShell/EWS: Remove duplicate items from a mailbox

PowerShell script that uses EWS to connect to an Exchange mailbox and search for and delete duplicate items.

 
 
 
 
 
5 Star
(4)
Add to favorites
2/7/2017
E-mail Twitter del.icio.us Digg Facebook
Sign in to ask a question


  • EWS Throttling Error
    1 Posts | Last post May 11, 2019
    • I'm still running into the EWS throttling issue (error below). I've read below that this could be an issue with interrupted internet connectivity. So I ran the command from an Azure server to alleviate any local issues and ran into the same error.
      
      Cannot access mailbox information store, error:
      Exception calling "LoadPropertiesForItems" with "2" argument(s): "The request failed. The read operation failed, see
      inner exception."
      
      I've tried multiple combinations of parameters and it seems to be only affecting running in Full mode (full script below).
      
      
      .\Remove-DuplicateItems.ps1 -Mailbox user.name@domain.com -Credentials $cre
      ds -PriorityFolders 'Inbox' -MailboxWide  -mode Full -MailboxOnly -Type mail
  • False Positives
    1 Posts | Last post February 06, 2019
    • I needed an Exchange 2010 dedupe script for a migration project to run against 4000 users so decided I would thoroughly test this before letting it loose. I discovered that whilst internetmessageid, subject and dates are checked the sender is not checked. This resulted in a lot of false positives for me, many of them were out of office responses back to the mailbox however I found a few examples of system generated messages for example which met the criteria. So I adjusted the property set to include the message id then added the following code in the IsDuplicateEmail function.
      
                      if (($dupSubject.DateTimeReceived -eq $item.DateTimeReceived) -and (!$dupSubject.IsFromMe)) 
                      { 
                          # Same subject and received date, so this is a duplicate 
      					 $itemtemp = [Microsoft.Exchange.WebServices.Data.EmailMessage]::Bind($global:service, $item.id)
      					 write-host $itemtemp
      					 write-host $itemtemp.From.Address
      					Write-host "Doing it again for the dupsubject item"
      					 $dupitemtemp = [Microsoft.Exchange.WebServices.Data.EmailMessage]::Bind($global:service, $dupsubject.id)
      					 write-host $dupitemtemp.From.Address
      					 if ($itemtemp.From.Address -eq $dupitemtemp.From.Address)
      					 {
      						write-host "From address matches, this really is a duplicate"
      						write-host $dupitemtemp.Subject
      						return $true 
      					 }
      					 else
      					 {
      					 	 write-host "From address did not match, not a duplicate"
      						 write-host $dupitemtemp.Subject
      						 return $false
      					 }
      					
                      } 
  • stop displaying EWS trace during execution
    2 Posts | Last post April 03, 2018
    • I have got the script working for EX2010 but during execution it displays the trace from the EWS calls. The trace is specifically enabled in code to detect EWS throttling.
      
      Is there any solution, or just use the log file to review results ?  I have tried setting a tracefilename but it doesn't seem to be used.  Stepping through shows that the compiler for the tracelistener fails as it cant find EWS.Dll , yet it does exist on the paths nominated in the refencelist. (EWS2.2)  (it actually exists twice...)
      
      other changes i had to make... 
          # Progress Reporting 
          $itemCount = $folder.TotalCount   always seems to be null, causing the percent progress to error out.
      
      added     if(!$itemCount) { $itemCount = 0}
      
              $view.Offset = $results.NextPageOffset
              $itemCount += $results.Items.Count
      
      also modified log to display date of message deleted
      Log ([string]::Format("Would delete: {0} @ {1}", $dupe.Subject,$dupe.DateTimeReceived))
      
      
    • sorry. found my problem.
      on my system        $EWSManagedApiPath 
      returned two paths to the dll (both valid), that is, it returns an array of size 2.
      
      if i use         #$Params.ReferencedAssemblies.Add($EWSManagedApiPath[0]) all is well.  now just to test if array or not.
  • Still duplicate items remaining
    1 Posts | Last post October 23, 2017
    • I've successfully ran this script to find that duplicate items that have attachments are not being removed. Do I need to add some other flag to also remove items with duplicates?
      
      Let me know if you needed an example.
  • I am getting this following error
    9 Posts | Last post February 07, 2017
    • 
      Failed to bind to root folder: Exception calling "Bind" with "2" argument(s): "The specified object was not found in the  store., The process failed to get the correct properties."
      
      I have verified that the user account i am using has full access to the mailbox.
    • Can you post the command you used to call the script, please? Change any values as required to protect privacy, but keep the syntax as close to what you actually used as possible.
    • Hi David, Thanks for the quick reply, following are the command and the error:-
      command
      -------------------------------
      PS C:\tools> .\RemoveDuplicate.ps1 -RecurseFolders -MatchEntireMailbox -mailbox username@domain.com -Credentials
       $Credentials -AllowInsecureRedirection -Whatif -LogFile C:\tools\userlogfile.txt
      Using managed API 15.00.0516.014 found at: C:\Program Files\Microsoft\Exchange\Web Services\2.0\Microsoft.Exchange.WebSe
      rvices.dll
      ----------------------
      
      error
      -------------------------
      Processing mailbox username@domain.com
      Failed to bind to root folder: Exception calling "Bind" with "2" argument(s): "The specified object was not found in the
       store., The process failed to get the correct properties."
      This could be due to lack of permissions to the mailbox, or invalid credentials.
      -----------------------
      
      I ran the same command to the user for which the credentials are saved, which is running fine.
    • I don't think I'd tested using FullAccess to another mailbox.  I've just tested it now, and I'm getting an error also.  If you use ApplicationImpersonation it will work (grant ApplicationImpersonation role to your authenticating account, then use -impersonate parameter).  I'll see if I can work out why it isn't working with FullAccess.
    • Its working fine with the impersonate switch. there were few errors in the middle with red colored text, but somehow those are not logged in the log. As the mailbox was big, the log file itself reached 5MB, so could not check line by line. But this time I will try to take a screenshot and upload here. Thanks for the script. David.
    • I'll post an updated script tomorrow.  I discovered why the FullAccess wasn't working for me - I'd forgotten to pass credentials to the script!  However, while working this out I made various improvements including adding a trace file option so will upload when I can.  I have confirmed that the version of the script I have is working as expected using FullAccess rights to a shared mailbox.
      
      Red errors that aren't logged are most likely PowerShell errors - these won't be logged, but also they shouldn't generally occur.  It would be interesting to see what errors you were getting to see if I can handle them more gracefully.
    • Hi David, Looks like the script is not yet updated. Could you please upload the new one, if you are ready. thanks a lot.
    • I took screenshots of the error, how can I send them to you.
    • Sorry, I forgot to upload the updated script.  I have just done so, please could you test with that (I have confirmed that it works as expected with FullAccess permissions)?
  • Duplicate Check - Email
    3 Posts | Last post January 12, 2017
    • I have run this script in WhatIf mode and noticed duplicate messages that are false positives. The date and time is not precise enough for high volume messages and only compares the subject. I would recommend adding to or replacing the message check duplicate check (line 411) with inclusion of the MessageId, a more unique value.
      if (($dupSubject.DateTimeReceived -eq $item.DateTimeReceived) -and (!$dupSubject.IsFromMe) -and ($dupSubject.InternetMessageId -eq $item.InternetMessageId))
    • good input
      thanks
    • The script already does a check of internetmessageid at line 378, how is that section not doing the same thing as what is proposed here? We were also getting some false positives & adding this code dropped the number of duplicates by 2 thirds and now checking if it's leaving out legitimate duplicates or not. Interested to know what this is even needed though?
  • Autodiscover failed: Excepción al llamar a "AutodiscoverUrl" con los argumentos "1":
    2 Posts | Last post December 03, 2016
    • I want remove email duplicated in exchange online and set the next command.
      
      $UserCredential = Get-Credential
      .\removecorreosduplicados.ps1 -Mailbox ****@***.com -RecurseFolders -MatchEntireMailbox -Credentials $UserCredential
      
      But show next error. In credentials i am set global administrator.
      
      Autodiscover failed: Excepción al llamar a "AutodiscoverUrl" con los argumentos "1": "Autodiscover blocked a potentially
       insecure redirection to https://autodiscover-s.outlook.com/autodiscover/autodiscover.xml. To allow Autodiscover to foll
      ow the redirection, use the AutodiscoverUrl(string, AutodiscoverRedirectionUrlValidationCallback) overload."
      Invalid credentials can cause failure here, even if the error looks generic (e.g. service could not be located)
      
      Help Please,
      
      Best Regard,
      
      Felix Huachaca
    • Either use -AllowInsecureRedirection parameter to allow the redirection, or to save having to autodisccover at all, you could specify the EWS URL by using -EWSUrl "https://outlook.office365.com/EWS/Exchange.asmx"
  • I have OWA , but where is EWS
    3 Posts | Last post January 11, 2016
    • First it wouldn't accept the -Password or the -Domain parms then I got the Failed to locate EWS error.  I thought the API was installed with OWA and that it would be found.  I must admit total ignorance of the EWS API.
      
      [PS] Microsoft.PowerShell.Core\FileSystem::\\esssbs2011\RedirectedFolders\MFaklis\Scripts>./Remove-DuplicateItems.ps1 -RecurseFolders -MatchEntireMailbox -WhatIf Username="MFaklis"
      Failed to locate EWS Managed API, cannot continue
    • You need to download and install the EWS Managed API.
      https://www.microsoft.com/en-us/download/details.aspx?id=42951
    • Thank you.  My problem was using incorrect syntax for passing arguments.  For example:
      -Username:"mfaklis" 
      instead of -Username "mfaklis".
      
      Thank you for the documenting this script.
  • Options for a domain
    2 Posts | Last post October 15, 2015
    • Could this be modified to run against an entire domain name? Just thinking about migrating multiple users and having to run this script on one mailbox at a time.
    • Yes, it can be run against multiple mailboxes.  It can be incorporated into other PowerShell scripts quite easily too.  See this blog for some examples: http://blogs.msdn.com/b/emeamsgdev/archive/2014/10/22/powershell-merge-mailbox-folders.aspx
  • Thank you
    1 Posts | Last post July 22, 2015
    • Just want to say thank you that this powershell works.