Windows 7 UI Automation Client API C# sample (hyperlink processor) Version 1.0

A few weeks ago I demo’d the sample C++ app which uses the Windows 7 UI Automation Client API to interact with hyperlinks in a browser, (http://code.msdn.microsoft.com/Windows-7-UI-Automation-9131f729), at the CSUN 2011 conference in San Diego. I got feedback that a similar C# sa

C# (303.8 KB)
 
 
 
 
 
4.7 Star
(3)
4,579 times
Add to favorites
5/2/2011
E-mail Twitter del.icio.us Digg Facebook
Sign in to ask a question


  • Is it still compatible with Windows 10 ?
    3 Posts | Last post September 06, 2016
    • Hi,
      Is it still compatible with Windows 10 ? If not, are you planning to release another version for windows 10 soon ? Thanks
    • Hi, this is a really great question, and until you asked, not one I'd looked into. But I've just done some experiments and shared the results out at https://blogs.msdn.microsoft.com/winuiautomation/2016/07/12/building-and-running-a-uia-sample-app-on-windows-10/. I'm sorry to have to say that I doubt I'll be able to upload a new sample any time soon, but hopefully the steps I describe at that post will help you get the sample working for you on Windows 10.
      
      Guy
    • Stuff still works I added example Excel 2013 with VBA in comments below
      https://blogs.msdn.microsoft.com/winuiautomation/2016/07/12/building-and-running-a-uia-sample-app-on-windows-10/
  • Running Automantion.RootElement before can cause different result
    1 Posts | Last post October 06, 2015
    • Hi Experts,
      
      When i run the sample project, it works fine in finding all the hyperlinks on IE. 
      
      But if i add one managed uia code step "AutomationElement e= AutomationElement.RootElement;"(UIAutomationClient, UIAutomationType are added into project reference) before the "elementArray = elementBrowser.FindAll(interop.UIAutomationCore.TreeScope.TreeScope_Descendants, condition);" step in LinkProcessor.cs, the hyperlinks found in elementArray counts 0.
      
      Our project use managed uia, but i am trying to add unmanaged uia support in our project too.
      Does managed uia affect the unmanaged uia api?
      
      Thanks
  • I''m having trouble getting the C# sample to run
    2 Posts | Last post February 05, 2015
    • Hi Guy,
      
      I'm getting errors from the pre build step. Is there a specific version of the UIAutomationCore.dll which is supposed to run?  
      
      The C++ app runs perfectly on my Win 7 sp1 64 bit platform. This is the same platform I'm trying to run the C# app.  
      
      Thanks
    • Hi, the problem with mine using Visual Studio 2012 Update 4 is with the references. You can find and Add in the COM References the UIAutomationCoreRes using the Reference Manager of your project. After that, you can fix the 'using' call. 
      
      using UIAutomationBlockingCoreLib;
      
      This allowed me to build the project on my end.
  • steps
    2 Posts | Last post January 29, 2014
    • How can i perform this steps? "%PROGRAMFILES%\Microsoft SDKs\Windows\v7.0A\bin\tlbimp.exe" %windir%\system32\UIAutomationCore.dll /out:..\interop.UIAutomationCore.dll"
      
      
      Thank you :-) 
    • Hi, it's been a while since I've run this step, but the goal is to generate an interop dll which can be used to allow your C# code to call into the unmanaged Windows UIA API.
      
      You need to run the step at a command prompt. The step runs the tlbimp.exe tool, and I've just found on my machine, the file "%PROGRAMFILES%\Microsoft SDKs\Windows\v7.0A\bin\tlbimp.exe" doesn't exist. But I expect that's because I'm at an x64 machine now, and I wasn't when I originally posted the blog. I can see the file ok if I look for "%PROGRAMFILES(X86)%\Microsoft SDKs\Windows\v7.0A\bin\tlbimp.exe". If you look around on your machine, you'll probably find tlbimp.exe somewhere. 
      
      Having found the tool, I then ran at the command prompt...
      
      "%PROGRAMFILES(X86)%\Microsoft SDKs\Windows\v7.0A\bin\tlbimp.exe" %windir%\system32\UIAutomationCore.dll /out:..\interop.UIAutomationCore.dll
      
      When I did this, I got lots of warnings reported, but the interop file called interop.UIAutomationCore.dll did get generated, (and it was put in the folder above where I currently was.) This interop dll can then be referenced by your project in Visual Studio, and your C# code can call into the unmanaged UIA API.
      
      Thanks,
      
      Guy
  • what
    1 Posts | Last post January 20, 2014
    • What is this step? "%PROGRAMFILES%\Microsoft SDKs\Windows\v7.0A\bin\tlbimp.exe" %windir%\system32\UIAutomationCore.dll /out:..\interop.UIAutomationCore.dll"
  • Thanks for posting this!
    5 Posts | Last post October 22, 2012
    • Hi Guy,
      
      I got my first Win 8 UIA client working today.  It's based on your example here. I'm really grateful for it and for your excellent comments in the source.  I can now find the controls on a Windows Store app and invoke the buttons. 
      
      Thanks a lot.
      
    • Hi, thanks very much for letting me know. I'm really pleased that the sample's been useful, and that you've used it to build a Windows 8 client app from it. I had a lot of fun building the original sample, but I've not had an opportunity to port it to Windows 8 yet.
      
      It seems that I can't submit a long reply here, so I'm posting my reply in a few parts to work around that...
    • I had thought about what I'd do with the sample if I built a Windows 8 version from it. One of the reasons that I built the Windows 7 samples was to try to help developers avoid issues with their client apps due to them not being aware of some of the rules around using UIA in Windows 7. Particularly, you need to use background MTA threads if you handle UIA events or if you call UIA to work with your own app's UI. It's easy to miss those rules, and if you do, your Windows 7 app can experience unexpected delays or even hang. I actually hit that problem myself when I built by first Windows 7 UIA client app, so I went back to it and added the background MTA thread, along with a queue to allow the main UI thread to communicate with the background thread. By putting the resulting app up as a sample, anyone can pick this up and reuse the threading code I added, (including myself.)
      
      But one of the great improvements around UIA in Windows 8 is that the above two rules around threading no longer exist. (There is still an active third rule around not adding or removing event handlers from within an event handler.) So I was considering porting one of my Windows 7 samples to Windows 8, and in the process strip out a whole load of unnecessary code about threading. If the threading code can be pulled out, it makes the sample much more interesting, as the reader can focus on the specifics of the UIA API calls. All it takes to port a client UIA app to Windows 8, is have it CoCreateInstance a CUIAutomation8 instead of a CUIAutomation. Once you've done that, you no longer need the background MTA threads.
      
    • Having said that, if you already have the background thread code in place, it will still work in Windows 8. What's more, you may want to consider whether you want a single client UIA exe to be able to run on both Windows 8 and Windows 7. If it's running on Windows 8, you’ll be able to CoCreate the CUIAutomation8, and all will be well. But if you're running on Windows 7, the CoCreate will fail. In that case you could CoCreate a CUIAutomation instead. If you do that, then you will need all the MTA threading code in place to account for the Windows 7 UIA rules around threading. I felt that if I did port the sample to Windows 8, it would be a Windows 8-only app, so that I could pull out all the old threading code.
      
      As we can see up at http://msdn.microsoft.com/en-us/library/windows/desktop/hh437316(v=VS.85).aspx, there are a number of enhancements introduced to the UIA API in Windows 8. My favourite enhancement (in addition to the threading simplification mentioned above), relates to preventing UIA client calls from blocking. In Windows 7, if a client app called UIA to gather some information about some provider app's UI, and that provider app had already hung, then the UIA call could block. This meant the client app would hang too. We did a lot of work with UIA in Windows 8 to allow client API calls to timeout if the provider app was unresponsive. So if the call comes back to you saying it had timed out, (because the provider app was unresponsive), you can decide what the most appropriate course of action is. (For example, if you’re an assistive technology app such as a screen reader, you could notify your user that the provider app's not responding.) If you find the default timeout is too short for your purposes, you can change it through IUIAutomation2::ConnectionTimeout().
    • There are some scenarios where the UIA client might still experience a longer delay than expected before timing out, but overall UIA is much more useful in Windows 8 when interacting with unresponsive apps than it was in Windows 7.
      
      If you've built an assistive technology app for Windows 8, you might find some of the features you've built work as expected when interacting with classic desktop apps, but not with Store apps. This might be due to the security model around Store apps. If so, assistive technology apps that have signed binaries might consider using the UIAccess privileges available for accessibility tools. That might enable the functionality you require with Store apps.
      
      Thanks,
      
      Guy
      
  • Win8 Example
    3 Posts | Last post October 05, 2012
    • Hi Guy,
      Thanks for posting this great example!  I'm trying to make this work on Windows8 Consumer Preview with and haven't had much success.  Could you post a win8 example?
      
    • Hi, 
      
      Thanks for the feedback, it's good for me to know that the sample's proving useful.
      
      I'll set to work on a Windows 8 sample next. In general UIA in Windows 8 is an enhancement of UIA in Windows 7. Many of the principles are exactly the same, and so if you know UIA in Windows 7, you'll know it in Windows 8. But there are some specific extensions to the API set, and there are two particular enhancements that I find really exciting. I'll describe these in more detail when I build my next sample, but the quick version is...
      
      (i) Some of the threading rules with client apps followed in Windows 7 no longer apply. In Windows 7, if the client app used event handlers or interact with its own UI, then it needed to create background MTA threads. Such thread usage is standard for many apps, but it meant my samples had to do more work than I originally expected. With these rules gone in Windows 8, my samples can focus on specific UIA calls rather than including a fair bit of thread-related code too.
      
      (ii) In Windows 7, if a client app used UIA to interact with some provider app's UI, but the provider app was unresponsive, then the UIA call could block. In Windows 8, the calls will timeout if the provider's unresponsive. This is really important for assistive technology (AT) apps that use UIA.
      
      So, I'll start planning out a Windows 8 sample now! :)
      
      Guy
      
    • Hi Guy,
      
      Automating Win 8 apps is my goal.  A a Windows 8 example would be awesome.  
      
      This blog post says your stuff is a great place to start learning how to do the UIA client.  
      http://blogs.msdn.com/b/windowsappdev/archive/2012/09/04/automating-the-testing-of-windows-8-apps.aspx
      
      Thanks
  • I''m having trouble getting the C# sample to run
    4 Posts | Last post October 05, 2012
    • Hi Guy,
      
      I'm getting errors from the pre build step. Is there a specific version of the UIAutomationCore.dll which is supposed to run?  
      
      The C++ app runs perfectly on my Win 7 sp1 64 bit platform. This is the same platform I'm trying to run the C# app.  
      
      Thanks
    • Hi, could you let me know exactly what error you're getting in the pre-build step? When I first uploaded this sample, the build steps worked fine. But I have since built it on other computers and found the steps in the sample project to generate the COM wrapper didn’t always work. (A COM wrapper is required to allow the C# code in the sample to call the native-code UIA API.) 
      
      So I couple of things I’d check would be…
      
      During the build, VS tries to run the tlbimp.exe tool to generate the COM wrapper. So it’s worth checking that the path to the tlbimp.exe file is valid. If the path is not valid on your computer, you’d want to update the path in the build step to a place which is valid. Eg on my computer I can see tlbimp.exe beneath “C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\x64”. 
      
      By the way, it’s also worth checking that any paths containing spaces in the build steps are surrounded by double quotes.
      
      Hopefully once you’ve done the above, the pre-build step of generating the COM wrapper will succeed. The next step is to verify that the COM wrapper is being referenced as required. If for some reason the reference to the assembly isn’t working. You could manually add the COM wrapper to the list of referenced assemblies. (In fact you only ever need to build the COM wrapper once, so once you’ve added the COM wrapper to the list of referenced assemblies, you could remove the pre-build step if you want.)
      
      All being well, once you have the COM wrapper successfully referenced, the full project should build ok for you.
      
      Thanks,
      
      Guy
      
    • Open the project: Click View, Solution Explorer, Right click References then Add Reference. click "Browse" and Find the C# program.. inside the folder "C#\Win7UIAClientManaged_LinkProcessor\bin\x64\Debug" you will find the file "interop.UIAutomationCore.dll" add it.
      
      I had the same problem
    • It builds now.  Thanks Qwicks.  It also runs.  :)
      
      I'm getting about 20 warnings from the pre-build step.  I'm  inclined to just ignore them at this point.  
      
      If you're interested, I've pasted them at the end.
      
      ------ Build started: Project: Win7UIAClientManaged_LinkProcessor, Configuration: Debug x64 ------
      warning CS1607: Assembly generation -- Referenced assembly 'System.Data.dll' targets a different processor
      warning CS1607: Assembly generation -- Referenced assembly 'mscorlib.dll' targets a different processor
      
      Compile complete -- 0 errors, 2 warnings
      TlbImp : warning TI0000: At least one of the arguments for 'IUIAutomationAndCondition.GetChildrenAsNativeArray' cannot be marshaled by the runtime marshaler.  Such arguments will therefore be passed as a pointer and may require unsafe code to manipulate.
        Microsoft (R) .NET Framework Type Library to Assembly Converter 3.5.30729.1
        Copyright (C) Microsoft Corporation.  All rights reserved.
        
        Type library imported to C:\Users\mess\Desktop\Windows 7 UI Automation Client API C%23 sample (hyperlink processor) Version 1.0\C#\Win7UIAClientManaged_LinkProcessor\bin\x64\interop.UIAutomationCore.dll
      TlbImp : warning TI0000: At least one of the arguments for 'IUIAutomationOrCondition.GetChildrenAsNativeArray' cannot be marshaled by the runtime marshaler.  Such arguments will therefore be passed as a pointer and may require unsafe code to manipulate.
      TlbImp : warning TI0000: At least one of the arguments for 'IUIAutomation.IntSafeArrayToNativeArray' cannot be marshaled by the runtime marshaler.  Such arguments will therefore be passed as a pointer and may require unsafe code to manipulate.
      TlbImp : warning TI0000: At least one of the arguments for 'IUIAutomation.SafeArrayToRectNativeArray' cannot be marshaled by the runtime marshaler.  Such arguments will therefore be passed as a pointer and may require unsafe code to manipulate.
      TlbImp : warning TI0000: At least one of the arguments for 'CUIAutomation.IntSafeA