Skip to main content
SpringerLink
Log in

Concepts and design space for a better understanding of multi-device user interfaces

  • Long Paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

This paper discusses the motivations behind and the characterising concepts of multi-device user interfaces by looking at the main design issues that have been addressed and the various solutions proposed. The discussion of relevant systems and frameworks highlights their main features, which are then used as the basis for comparative discussion. It compares different approaches and perspectives adopted in this area (e.g. responsive design, cross-device, distributed, migratory user interfaces). The features constitute a design space that can be used to facilitate analysis and comparison of tools and frameworks for multi-device user interfaces. Such aspects can be exploited by user interface designers and developers to analyse and compare various options when addressing existing and new applications. The analysis provided may inspire the design and development of new tools and frameworks as well.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. http://getbootstrap.com/.

  2. https://ifttt.com/.

  3. http://mozilla.com/mobile/sync.

  4. https://www.pushbullet.com/.

References

  1. Arthur, R., Olsen, D.R.: Privacy-aware shared UI toolkit for nomadic environments. Softw. Pract. Exp. 42, 601–628 (2011)

    Article  Google Scholar 

  2. Barralon, N., Coutaz, J.: Coupling interaction resources in ambient spaces: there is more than meets the eye! In: Gulliksen, J., Harning, M.B., Palanque, P., van der Veer, G.C., Wesson, J. (eds.) Engineering Interactive Systems. Engineering Interactive Systems, vol. 4940. Springer, Berlin (2008)

    Chapter  Google Scholar 

  3. Barsotti, M., Paternò, F., Pulina, F.: A web framework for cross-device gestures between personal devices and public displays. In: The 16th International Conference on Mobile and Ubiquitous Multimedia (MUM2017), Suttgart, November 2017, pp. 69–78, ACM Press

  4. Bellotti, V., Edwards, W.K.: Intelligibility and accountability: human considerations in context-aware systems. Hum. Comput. Interact. 16(2–4), 193–212 (2001)

    Article  Google Scholar 

  5. Bellucci, F., Ghiani, G., Paternò, F., Santoro, C.: Engineering JavaScript state persistence of web applications migrating across multiple devices. In: ACM EICS, pp. 105–110 (2011)

  6. Brusilovsky, P.: Adaptive hypermedia. User Model. User-Adapt. Interact. 11(1–2), 87–110 (2001)

    Article  Google Scholar 

  7. Buyukkokten, O., Kaljuvee, O., Garcia-Molina, H., Paepcke, A., Winograd, T.: Efficient web browsing on handheld devices using page and form summarization. ACM Trans. Inf. Syst. Secur. 20(1), 82–115 (2002)

    Article  Google Scholar 

  8. Cantera, J.M.: Model-Based UI XG Final Report https://www.w3.org/2005/Incubator/model-based-ui/XGR-mbui-20100504/ (2010)

  9. Chang, T. H., Li, Y.: Deep shot: a framework for migrating tasks across devices using mobile phone cameras. In: Proceedings ACM CHI’11, pp. 2163–2172

  10. Chen, X. A., Grossman, T., Wigdor, D. J., Fitzmaurice, G.W.: Duet: exploring joint interactions on a smart phone and a smart watch. In: CHI 2014, pp. 159–168

  11. Chen, X., Li, Y.: Improv: an input framework for improvising cross-device interaction by demonstration. ACM Trans. Comput. Hum. Interact. 24(2), 15 (2017)

    Article  Google Scholar 

  12. Cheverst, K., Davies, N., Mitchell, K., Efstratiou, C.: Using context as a crystal ball: rewards and pitfalls. Pers. Ubiquitous Comput. 5(1), 8–11 (2001)

    Article  Google Scholar 

  13. Chi, P. Y. P., Li, Y.: Weave: scripting cross-device wearable interaction. In: ACM CHI 2015, Seoul, Korea, April 2015, pp. 3923–3932

  14. Corporate Act-Net Consortium.:The active database management system manifesto: a rulebase of ADBMS features. In: ACM SIGMOD Record, vol. 25 (1996)

  15. Coutaz, J., Lachenal, C., Dupuy-Chessa, S.: Ontology for multi-surface interaction. In: Proceedings of IFIP Conference on Human-Computer Interaction Interact’2003, IOS Press, Amsterdam, pp. 447–454 (2003)

  16. Coutaz, J., Nigay, L., Salber, D., Blandford, A., May, J., Young, R.: Four easy pieces for assessing the usability of multimodal interaction: the CARE properties. In: Proceedings INTERACT, pp. 115–120 (1995)

  17. Dearman, D., Pierce, J.: It’s on my other Computer!: computing with multiple devices. In: Proceedings of CHI’08, ACM Press, pp. 767–776 (2008)

  18. Demeure, A., Sottet, J. S., Calvary, G., Coutaz, J., Ganneau, V., Vanderdonckt, J.: The 4C reference model for distributed user interfaces. In: Fourth International Conference on Autonomic and Autonomous Systems, ICAS 2008, pp. 61–69, (2008)

  19. Dey, A.: Understanding and using context. Pers. Ubiquit. Comput. 5(1), 4–7 (2001)

    Article  MathSciNet  Google Scholar 

  20. Di Geronimo, L., Bertarini, M., Badertscher, J., Husmann, M., Norrie, M. C.: Exploiting mid-air gestures to share data among devices. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI ‘17), vol. 35 (2017)

  21. Dong, T., Churchill, E., Nichols, J.: Understanding the challenges of designing and developing multi-device experiences. In: Proceedings DIS 2016, pp. 62–72. ACM Press

  22. Fischer, C., Gellersen, H., Gostner, R., Guinard, D., Kortuem, G., Kray, C., Rukzio, E., Strengin, S.: Supporting device discovery and spontaneous interaction with spatial references. Mobile Spat. Interact. Pers. Ubiquitous Comput. (2008). https://doi.org/10.1007/s00779-008-0206-3

    Article  Google Scholar 

  23. Frosini, L., Paternò, F.: User interface distribution in multi-device and multi-user environments with dynamically migrating engines. ACM EICS 2014, 55–64 (2014)

    Google Scholar 

  24. Gajos, K.Z., Weld, D.S., Wobbrock, J.O.: Automatically generating personalized user interfaces with Supple. Artif. Intell. 174(12), 910–950 (2010)

    Article  Google Scholar 

  25. Ghiani, G., Isoni, L., Paternò, F.: Security in migratory interactive web applications. In: Proceedings of the 11th International Conference on Mobile and Ubiquitous Multimedia MUM 2012, vol. 15. ACM Press (2012)

  26. Ghiani, G., Manca, M., Paternò, F., Porta, C.: Beyond responsive design: context-dependent multimodal augmentation of web applic. In: MobiWIS 2014, LNCS, vol. 8640, pp. 71–85. Springer (2014)

  27. Ghiani, G., Paternò, F.: Supporting mobile users in selecting target devices. J. Univ. Comput. Sci. 16(15), 2019–2037 (2010)

    Google Scholar 

  28. Ghiani, G., Paternò, F., Santoro, C.: Push and pull of web user interfaces in multi-device environments. In: AVI 2012, pp. 10–17

  29. Ghiani, G., Paternò, F., Santoro, C.: Interactive customization of ubiquitous Web applications. J. Vis. Lang. Comput. 24(1), 37–52 (2013)

    Article  Google Scholar 

  30. Ghiani, G., Polet, J., Antila, V.: Towards intelligent migration of user interfaces. In: MobiWIS 2013, pp. 203–217 (2013)

    Chapter  Google Scholar 

  31. Ghiani, G., Manca, M., Paternò, F., Santoro, C.: Personalization of context-dependent applications through trigger-action rules. ACM Trans. Comput. Hum. Interact. 24(2), 14 (2017)

    Article  Google Scholar 

  32. Gjerlufsen, T., Klokmose, C. N., Eagan, J., Pillias, C., Beaudouin-Lafon, M.: Shared substance: developing flexible multi-surface applications. CHI 2011, pp. 3383–3392 (2011)

  33. Google Research Report.: The New Multi-screen World: Understanding Cross-Platform Consumer Behavior. http://services.google.com/fh/files/misc/multiscreenworld_final.pdf (2012). Accessed 3 April 2019

  34. Greenberg, S., Boring, S., Vermeulen, J., Dostal, J.: Dark patterns in proxemic interactions: a critical perspective. In Proceedings of DIS’14, pp. 523–532, ACM

  35. Hamilton, P., Wigdor, D.J.: Conductor: enabling and understanding cross-device interaction. In Proceedings of CHI’14, pp. 2773–2782. ACM Press (2014)

  36. Houben, S., Marquardt, N.: WatchConnect: a toolkit for prototyping smartwatch-centric cross-device applications. In: CHI 2015, pp. 1247–1256 (2015)

  37. Houben, S., Tell, P., Bardram, J.E. Bardram: ActivitySpace: managing device ecologies in an activity-centric configuration space. In: ITS 2014, pp. 119–128 (2014)

  38. Husmann, M., Nebeling, M., Pongelli, S., Norrie, M.C.: MultiMasher: providing architectural support and visual tools for multi-device mashups. WISE 2(2014), 199–214 (2013)

    Google Scholar 

  39. Husmann, M., Spiegel, M., Murolo, A., Norrie, M. C.: UI testing cross-device applications. In: Proceedings of the 2016 ACM on Interactive Surfaces and Spaces, ISS ‘16, pp. 179–188. New York. ACM (2016)

  40. Johanson, B., Ponnekanti, S., Sengupta, C., Fox, A.: Multibrowsing: moving web content across multiple displays. In: Proceedings of Ubicomp 2001, Springe, LNCS, vol. 2201, pp. 346–353 (2001)

    Chapter  Google Scholar 

  41. Johanson, B., Fox, A., Winograd, T.: The Interactive Workspaces Project: Experiences with Ubiquitous Computing Rooms. IEEE Pervasive Comput. 1(2), 67–74 (2002)

    Article  Google Scholar 

  42. Jokela, T., Ojala, J., Grassel, G., Piippo, P., Olsson, T.: A comparison of methods to move visual objects between personal mobile devices in different contexts of use. In: Proceedings of MobileHCI’15, pp. 172–181

  43. Jokela, T., Ojala, J., Olsson, T.: A diary study on combining multiple information devices in everyday activities and tasks. In: CHI 2015, pp. 3903–3912 (2015)

  44. Kulkarni, C., Klemmer, S.: Automatically adapting web pages to heterogeneous devices. In: CHI’11 Extended Abstracts on Human Factors in Computing Systems (CHI EA’11). ACM, New York, pp. 1573–1578 (2011)

  45. Lieberman, H., Paternò, F., Klann, M., Wulf, V.: End-user development: an emerging paradigm. In: Lieberman, H., Paternò, F., Wulf, V. (eds.) End-user Development. Human-Computer Interaction Series, pp. 1–8. Springer, Berlin (2006)

    Chapter  Google Scholar 

  46. Lo, J., Wohlstadter, E., Mesbah, A.: Imagen: runtime migration of browser sessions for javascript web applications. In: Proceedings of the International World Wide Web Conference (WWW), pp. 815–825 (2013)

  47. Manca, M. Paternò, F.: Customizable dynamic user interface distribution. In: Proceedings ACM EICS’16, Bruxells, pp. 27–37. ACM Press

  48. Marcotte, E.: Responsive Web Design, A Book Apart. http://www.abookapart.com/products/responsive-web-design (2011). Accessed 3 April 2019

  49. Marquardt, N., Diaz-Marino, R., Boring, S., Greenberg, S.: The proximity toolkit: prototyping proxemic interactions in ubiquitous computing ecologies. In: Proceedings of UIST 2011, pp. 315–326. ACM (2011)

  50. Marquardt, N., Hinckley, K., Greenberg, S.: Cross-device interaction via micro-mobility and f-formations. In: Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, pp. 13–22. ACM Press

  51. Marquardt, N., Ballendat, T., Boring, S., Greenberg, S., Hinckley, K.: Gradual engagement: facilitating information exchange between digital devices as a function of proximity. In: Proceedings of the 2012 ACM International Conference on Interactive tabletops and surfaces, pp. 31–40. ACM Press

  52. Melchior, J., Grolaux, D., Vanderdonckt, J., Van Roy, P.: A toolkit for peer-to-peer distributed user interfaces: concepts, implementation, and applications. In: Proceedings of EICS 2009, pp. 69–78. ACM, (2009)

  53. Melchior, J., Vanderdonckt, J., Van Roy, P.: A model-based approach for distributed user interfaces. In: Proceedings of the 3rd ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS ‘11). ACM, pp. 11–20 (2009)

  54. Meskens, J., Vermeulen, J., Luyten, K., Coninx, K.: Gummy for multi-platform user interface designs: shape me, multiply me, fix me, use me. In: Proceedings of the Working Conference on Advanced Visual Interfaces (AVI’08), pp. 233–240. ACM

  55. Myers, B., Hudson, S.E., Pausch, R.: Past, present, and future of user interface software tools. ACM Trans. Comput. Hum. Interact. 7(1), 3–28 (2000)

    Article  Google Scholar 

  56. Nacenta, M.A., Aliakseyeu, D., Subramanian, S., Gutwin, C.: A comparison of techniques for multi-display reaching. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘05). ACM, New York, pp. 371–380 (2005). http://dx.doi.org/10.1145/1054972.1055024

  57. Nebeling, M.: XDBrowser 2.0: semi-automatic generation of cross-device interfaces. In: Proceedings CHI 2017, pp. 4574–4584. ACM Press

  58. Nebeling, M., Husmann, M., Zimmerli, C., Valente, G., Norrie, M. C.: XDSession: integrated development and testing of cross-device applications. In: EICS 2015, pp. 22–27 (2015)

  59. Nebeling, M., Mintsi, T., Husmann, M., Norrie, M.C.: Interactive development of cross-device user interfaces, pp. 2793–2802. ACM CHI, Norrie (2014)

    Google Scholar 

  60. Nebeling, M., Speicher, M., Norrie, M. C.: CrowdAdapt: enabling crowdsourced web page adaptation for individual viewing conditions and preferences. In: EICS 2013, pp. 23–32 (2013)

  61. Park, S., Gebhardt, C., Rädle, R., Feit, A.M., Vrzakova, H., Dayama, N.R., Yeo, H.S., Klokmose, C.N., Quigley, A., Oulasvirta, A., Hilliges, O.: AdaM: adapting multi-user interfaces for collaborative environments in real-time. In: Proceedings ACM CHI 2018, Monterey. ACM Press (2018)

  62. Paternò,F., Santoro, C.: A logical framework for multi-device user interfaces. In: EICS 2012, pp. 45–50, ACM Press,Copenaghen (2012)

  63. Paternò Santoro, C., Spano, L.D.: MARIA: a universal language for service-oriented applications in ubiquitous environment. ACM Trans. Comput. Hum. Interact. 16(4), 191–1930 (2009)

    Google Scholar 

  64. Paternò, F., Santoro, C., Spano, L.D.: Engineering the authoring of usable service front ends. J. Syst. Softw. 84(10), 1806–1822 (2011)

    Article  Google Scholar 

  65. S. Ponnekanti, B. Lee, A.Fox, and P.Hanrahan, Icrafter: A service framework for ubiquitous computing environments. Proc. ACM Ubiquitous Computing (UbiComp’01) (Jan 2001), pp.56–75

  66. Ponnekanti, S. R., Johanson, B., Kiciman, E., Fox, A.: Portability, extensibility and robustness in iROS. In Proceedings of the 1st IEEE International Conference on Pervasive Computing and Communications (PERCOM ‘03)

  67. Rädle, R., Jetter, H. C., Marquardt, N., Reiterer, H., Rogers, Y.: HuddleLamp: spatially-aware mobile displays for ad-hoc around-the-table collaboration. In Proceedings ITS’14, pp. 45–54. ACM (2014)

  68. Rädle, R., Jetter, H. C., Schreiner, M., Lu, Z., Reiterer, H., Rogers, Y.: Spatially-aware or spatially-agnostic?: Elicitation and evaluation of user-defined cross-device interactions. In: CHI, pp. 3913–3922 (2015)

  69. Rekimoto, J.: Pick-and-drop: a direct manipulation technique for multiple computer environments. In: ACM Symposium on User Interface Software and Technology, pp. 31–39 (1997)

  70. Rekimoto J., Saitoh, M.: Augmented surfaces: a spatially continuous work space for hybrid computing environments. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘99). ACM, New York, pp. 378–385 (1999). Doi: http://dx.doi.org/10.1145/302979.303113

  71. Sanctorum, A. Signer, B.: Towards end-user development of distributed user interfaces. In: UAIS, pp. 1–15. Springer (2017)

  72. Santosa, S., Wigdor, D.: A field study of multi-device workflows in distributed workspaces. 2013. A field study of multi-device workflows in distributed workspaces. In: Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’13). ACM, New York, pp. 63–72

  73. Schreiner, M., Rädle, R., Jetter, H. C., Reiterer, H.: Connichiwa: a framework for cross-device web applications. In: CHI Extended Abstracts 2015, pp. 2163–2168

  74. Serrano, M., Ens, B., Yang, X. D., Irani, P: Developing a head-worn display interface to unify the interaction experience in distributed display environments. In: Proceedings of MobileHCI’15, pp. 16–171. ACM Press

  75. Sohn, T., Li, F.C.Y., Battestini, A., Setlur, V., Mori, K. Horii, H.: Myngle: unifying and filtering web content for unplanned access between multiple personal devices. In: Proceedings UbiComp 2011, pp. 257–266. ACM

  76. Tandler, P., Prante, T., Müller-Tomfelde, C., Streitz, N. and Steinmetz, R.: ConnecTables: dynamic coupling of displays for the flexible creation of shared workspaces. In: Proceedings of 14th ACM Symposium on UI Software and Technical UIST’01,, pp. 11–20. ACM Press, New York (2001)

  77. Terrenghi, L., Quigley, A., Dix, A.: A taxonomy for and analysis of multi-person-display ecosystems. Pers Ubiquitous Comput 13(8), 583–598 (2009)

    Article  Google Scholar 

  78. Vandervelpen, C., Conix, K.: Towards model-based design support for distributed user interfaces. In: Proceedings of NordiCHI 2004, pp. 61–70. ACM (2004)

  79. Wäljas, M., Segerståhl, K., Väänänen-Vainio-Mattila, K., Oinas-Kukkonen, H.: Cross-platform service user experience: a field study and an initial framework, In: Proceedings of MobileHCI 2010 Lisboa, Portugal, September 2010, pp. 219–228. ACM Press (2010)

  80. Yang, J. Wigdor, D.: Panelrama: enabling easy specification of cross-device web applications. In: ACM CHI 2014, pp. 2783–2792, Toronto (2014)

Download references

Author information

Authors and Affiliations

  1. HIIS Laboratory, CNR-ISTI, Pisa, Italy

    Fabio Paternò

Authors
  1. Fabio Paternò
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fabio Paternò.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Paternò, F. Concepts and design space for a better understanding of multi-device user interfaces. Univ Access Inf Soc 19, 409–432 (2020). https://doi.org/10.1007/s10209-019-00650-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-019-00650-5

Keywords

Search

Navigation