Skip to main content
SpringerLink
Log in

Seeking the user interface

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

User interface design and coding can be complex and messy. We describe a system that uses code search to simplify and automate the exploration of such code. We start with a simple sketch of the desired interface along with a set of keywords describing the application context. If necessary, we convert the sketch into a scalable vector graphics diagram. We then use existing code search engines to find results based on the keywords. We look for potential Java-based graphical user interface solutions within those results and apply a series of code transformations to the solutions to generate derivative solutions, aiming to get solutions that constitute only the user interface and that will compile and run. We run the resultant solutions and compare the generated interfaces to the user’s sketches. Finally, we let programmers interact with the matched solutions and return the running code for the solutions they choose. The system is useful for exploring alternative interfaces to the initial and for looking at graphical user interfaces in a code repository.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  • Akhin, M., Tillmann, N., Fahndrich, M., de Halleux, J., Moskal, M.: Search by example in touch develop: code search made easy. In: Proceedings SUITE 2013, pp. 5–8 (2012)

  • Bajracharya, S., Ngo, T., Linstead, E., Dou, Y., Rigor, P., Baldi, P., Lopes, C.: Sourcerer: a search engine for open source code supporting structure-based search. In: Proceedings ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications 2006, pp. 682–682 (2006)

  • Bajracharya, S., Ossher, J., Lopes, C.: Sourcerer: an infrastructure for large-scale collection and analysis of open-source code. Sci. Comput. Program. 79, 241–259 (2014)

    Article  Google Scholar 

  • Begel, A.: Codifier: a programmer-centric search user interface. In: Workshop on Human-Computer Interaction and Information Retrieval (2007)

  • Bragdon, A., Reiss, S.P., Zeleznik, R., Karumuri, S., Cheung, W., Kaplan, J., Coleman, C., Adeputra, F., LaViola Jr. J.J.: Code bubbles: a working set-based interface for code understanding and maintenance. In: Proceedings SIGCHI Conference on Human Factors in Computing Systems, pp. 2503–2512 (2010a)

  • Bragdon, A., Reiss, S.P., Zeleznik, R., Karumuri, S., Cheung, W., Kaplan, J., Coleman, C., Adeputra, F., LaViola Jr. J.J.: Code bubbles: rethinking the user interface paradigm of integrated development environments. In: ACM/IEEE International Conference on Software Engineering, pp. 455–464 (2010b)

  • Bull, G., Gao, J.: Classification of hand-written digits using choriograms. In: IEEE International Conference on Image Computing Techniques and Applications (2011)

  • Caetano, A., Goulart, N., Fonseca, M., Jorge, J.: Javasketchit: issues in sketching the look of user interfaces. In: AAAI Spring Symposium on Sketch Understanding, pp. 9–14 (2002)

  • Chan, W.-K., Cheng, H., Lo, D.: Searching connected api subgraph via text phrases. In: Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, FSE ’12, pp. 10:1–10:11. ACM, New York, NY, USA (2012)

  • Chou, S.-C., Chen, J.-Y., Chung, C.-G.: A behavior-based classification and retrieval technique for object-oriented specification reuse. Softw. Pract. Exp. 26(7), 815–832 (1996)

    Article  Google Scholar 

  • Chou, S.-C., Chen, Y.-C.: Retrieving reusable components with variation points from software product lines. Inf. Process. Lett. 99, 106–110 (2006)

    Article  MATH  Google Scholar 

  • Coates, A., Carpenter, B., Case, C., Satheesh, S., Suresh, B., Wang, T., Wu, D.J., Ng, A.Y.: Text detection and character recognition in scene images with unsupervised feature learning. In: 2011 Internataional Conference on Document Analysis and Recognition ICDAR, pp. 440–445 (2011)

  • da Silva, P.P.: User interface declarative models and development environments: a survey. In: Patern, F. (ed.) Proceeding of the 7th International Conference on Design, Specficiation, and Verification of Interactive Systems, pp. 207–226. Springer, Berlin (2000)

  • de Sa, M., Carrigo, L., Duarte, L., Reis, T.: A mixed-fidelity prototyping tool for mobile devices. In: Proceedings of the Working Conference on Advanced Visual Interfaces, pp. 225–232 (2008)

  • Drummond, C.G., Ionescu, D., Holte, R.C.: A learning agent that assists the browsing of software libraries. IEEE Trans. Softw. Eng. 26(12), 1179–1196 (2000)

    Article  Google Scholar 

  • Duda, R.O., Hart, P.E.: Use of the hough transformation to detect lines and curves in pictures. Commun. ACM 15, 11–15 (1972)

    Article  MATH  Google Scholar 

  • Ernst, M.D., Lencevisius, R., Perkins, J.H.: Detection of web service substitutability and composability. In: WS-MaTe 2006: International Workshop on Web Services—Modeling and Testing, pp. 123–135 (2006)

  • Frakes, W.B., Pole, T.P.: An empirical study of representation methods for reusable software components. IEEE Trans. Softw. Eng. 20(8), 617–630 (1994)

    Article  Google Scholar 

  • Gajos, K.Z., Weld, D.S., Wobbrock, J.O.: Decision-theoretic user interface generation. In: Proceedings of the 22nd AAAI Conference on Artificial Intelligence (AAAI-08) (2008)

  • Goodger, B., Hickson, I., Hyatt, D., Waterson, C.: Xml user interface language (xul) 1.0 specficiation. Technical report. http://www.mozilla.org/projects/xul/xul.html (2003)

  • Grechanik, M., Fu, C., Xie, Q., McMillan, C., Poshyvanyk, D., Cumby, C.: A search engine for finding highly relevant applications. In: Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering (2010)

  • Greenberg, S.: Toolkits and interface creativity. J. Multimed. Tools Appl. 32, 139–159 (2007)

    Article  Google Scholar 

  • Haiduc, S., Bavota, G., Oliveto, R., Marcus, A., Lucia, A.D.: Evaluating the specificity of text retrieval queries to support software engineering tasks. In: Proceedings of the 2012 International Conference on Software Engineering, pp. 1273–1276 (2012)

  • Haiduc, S., Bavota, G., Marcus, A., Oliveto, R., Lucia, A.D., Menzies, T.: Automatic query reformulations for text retrieval in software engineering. In: Proceedings of the 2013 International Conference on Software Engineering, pp. 842–851 (2013)

  • Hall, R.J.: Generalized behavior-based retrieval. In: Proceedings International Conference on Software Engineering ’93, pp. 371–380 (1993)

  • Harris, C., Stephens, M.: A combined corner and edge detector. In: Proceedings of Fourth Alvey Vision Conference, pp. 147–151 (1988)

  • Hartmann, B., Abdulla, L., Mittal, M., Klemmer, S.R.: Authoring sensor based interactions through direct manipulation and pattern matching. In: Proceedings of chi 2007: ACM Conference on Human Factors in Computing Systems, pp. 145–154 (2007)

  • Hill, E., Pollock, L., Vijay-Shanker, K.: Automatically capturing source code context of NL-queries for software maintenance and reuse. In: International Conference on Software Engineering (2009)

  • Hoffmann, R., Fogarty, J.: Assieme: finding and leveraging implicit references in a web search interface for programmers. In: Proceedings UIST 2007, pp. 13–22 (2007)

  • Hummel, O., Janjic, W., Atkinson, C.: Code conjurer: pulling resusable software out of thin air. IEEE Softw. 25(5), 45–52 (2008)

    Article  Google Scholar 

  • Janjic, W., Atkinson, C.: Leveraging software search and reuse with automated software adaptation. In: Proceedings SUITE 2013, pp. 23–26 (2012)

  • Janjic, W., Stoll, D., Bostan, P., Atkinson, C.: Lowering the barrier to reuse through test-driven search. In: SUITE ’09, pp. 21–24 (2009)

  • Ko, A.J., Aung, H., Myers, B.A.: Eliciting design requirements for maintenance-oriented ides: a detailed study of corrective and perfective maintenance tasks. In: Proceedings of the 27th International Conference on Software Engineering, pp. 126–135 (2005)

  • Landay, J.A., Myers, B.A.: Sketching interfaces: toward more human interface design. Computer 34(3), 56–64 (2001)

    Article  Google Scholar 

  • Lemos, O., Bajracharya, S., Ossher, J., Morla, R., Masiero, P., Baldi, P., Lopes, C.: Codegenie: using test-cases to search and reuse source code. In: ASE ’07, pp. 525–526 (2007)

  • Lemos, O., Bajracharya, S., Ossher, J., Masiero, P., Lopes, C.: Applying test-driven code search to the reuse of auxiliary functionality. In: Proceedings ACM Symposium on Applied Computing, pp. 476–482 (2009)

  • Lemos, O.A.L., Bajracharya, S., Ossher, J., Masiero, P.C., Lopes, C.: A test-driven approach to code search and its application to the reuse of auxiliary functionality. Inf. Softw. Technol. 53(4), 294–306 (2011)

    Article  Google Scholar 

  • Li, X., Wang, Z., Wang, Q., Yan, S., Xie, T., Mei, H.: Relationship-aware code search for javascript frameworks. In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, pp. 690–701. ACM, New York, NY, USA (2016)

  • Lin, Z., Zhao, J., Xie, B.: A graph database based crowdsourcing infrastructure for modelling and searching code structure. In: Proceedings of the 6th Asia-Pacific Symposium on Internetware on Internetware, INTERNETWARE 2014, pp. 15–24. ACM, New York, NY, USA (2014)

  • Little, G., Miller, R.C.: Keyword programming in java. In: Proceedings ASE 2007, pp. 84–93 (2007)

  • Liu, D., Marcus, A., Poshyvanyk, D., Rajlich, V.: Feature location via information retrieval based filtering of a single scenario execution trace. In: Proceedings of the Twenty-Second IEEE/ACM International Conference on Automated Software Engineering, pp. 234–243 (2007)

  • Lu, M., Sun, X., Wang, S., Lo, D., Duan, Y.: Query expansion via wordnet for effective code search. In: Proceedings of the 2015 IEEE 22nd International Conference of Software Analysis, Evolution and Reengineering, pp. 545–549 (2015)

  • Maarek, Y.S., Berry, D.M., Kaiser, G.E.: An information retrieval approach for automatically constructing software libraries. IEEE Trans. Softw. Eng. 17(8), 800–813 (1991)

    Article  Google Scholar 

  • Martie, L., der Hoek, A.V.: Toward social-technical code search. In: 6th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), pp. 101–104 (2013)

  • Martie, L., der Hoek, A.V.: Sameness: an experiment in code search. In: Proceedings of the 12th Working Conference on Mining Software Repositories, pp. 76–87 (2015)

  • Martie, L., LaToza, T.D., van der Hoek, A.: Codeexchange: supporting reformulation of code queries in context. In: Proceedings of the 30th International Conference on Automated Software Engineering (2015)

  • McMillan, C., Grechanik, M., Poshyvanyk, D., Xie, Q., Fu, C.: Portfolio: finding relevant functions and their usage. In: Proceeding of the 33rd International Conference on Software engineering. mgpxfport (2011)

  • Meixner, G., Patern, F., Vanderdonckt, J.: Past, present, and future of model-based user interface development. i-com 10(3), 2–11 (2011)

    Article  Google Scholar 

  • Meyers, B.A., Ko, A.J., Coblenz, M.J., Aung, H.H.: An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE Trans. Softw. Eng. 32(12), 971–987 (2006)

    Article  Google Scholar 

  • Mishne, A., Shoham, S., Yahav, E.: Typestate-based semantic code search over partial programs. SIGPLAN Not. 47, 997–1016 (2012)

    Article  Google Scholar 

  • Neumann, L., Matas, J.: Real-time scene text localization and recognition. In: IEEE Conference on Computer Vision and Pattern Recognitioin (2012)

  • Nguyen, T.A., Csallner, C.: Reverse engineering mobile application user interfaces with remaui. In: Proceedings of Automated Software Engineering, pp. 248–259 (2015)

  • Nichols, J., Faulring, A.: Automatic interface generation and future user interface tools. In: ACM CHI 2005 Workshop on The Future of User Interface Design Tools (2005)

  • Nylander, S.: Semi-automatic generation of device adapted user interfaces. In: UIST conference companion (2005)

  • Park, J.-W., Lee, M.-W., Roh, J.-W., Hwang, S.-W., Kim, S.: Surfacing code in the dark: an instant clone search approach. Knowl. Inf. Syst. 41(3), 727–759 (2014)

    Article  Google Scholar 

  • Paul, S., Prakash, A.: A framework for source code search using program patterns. IEEE Trans. Softw. Eng. 20(6), 463–475 (1994)

    Article  Google Scholar 

  • Plamondon, R., Srihari, S.N.: Online and off-line handwriting recognition: a comprehensive survey. IEEE Trans. Pattern Anal. Mach. Intell. 22(1), 63–84 (2000)

    Article  Google Scholar 

  • Podgurski, A., Pierce, L.: Retrieving reusable software by sampling behavior. ACM Trans. Softw. Eng. Methodol. 2(3), 286–303 (1993)

    Article  Google Scholar 

  • Raneburger, D., Popp, R., Vanderdonckt, J.: An automated layout approach for model-driven wimp-ui generation. In: Proceedings of the 4th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS ’12), pp. 91–100 (2012)

  • Reiss, S.P.: A component model for internet-scale applications. In: Proceedings ASE 2005, pp. 34–43 (2005)

  • Reiss, S.P.: Automatic code stylizing. In: Proceedings ASE ’07, pp. 74–83 (2007)

  • Reiss, S.P.: Semantics-based code search. In: International Conference on Software Engineering 2009, pp. 243–253 (2009a)

  • Reiss, S.P.: Specifying what to search for. In: Proceedings SUITE 2009 (2009b)

  • Reiss, S.P.: Plugging in and into code bubbles. In: Proceedings Workshop on Developing Tools as Plug-ins 2012, pp. 55–60 (2012)

  • Reiss, S.P.: Browsing software repositories. http://www.cs.brown.edu/people/spr/rebuspaper.pdf (2014a)

  • Reiss, S.P.: Seeking the user interface. In: Proceedings of 29th ACM/IEEE International Conference on Automated Software Engineering, pp. 103–114 (2014b)

  • Seifert, J., Pfleging, B., del Carmen Valderrama Bahamndez, E., Hermes, M., Rukzio, E., Schmidt, A.: Mobidev: a tool for creating apps on mobile phones. In: Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services, pp. 109–112 (2011)

  • Sisman, B., Kak, A.C.: Assisting code search with automatic query reformulation for bug localization. In: Proceedings of the 10th Working Conference on Mining Software Repositories, pp. 309–318 (2013)

  • Stolee, K.T., Elbaum, S.: Toward semantic search via smt solver. In: Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, pp. 1–4 (2012)

  • Stolee, K.T., Elbaum, S., Dobos, D.: Solving the search for source code. ACM Trans. Softw. Eng. Methodol. 23(3), 1–45 (2014)

    Article  Google Scholar 

  • Sugumaran, V., Storey, V.C.: A semantic-based approach to component retrieval. Adv. Inf. Syst. 34(3), 8–24 (2003)

    Article  Google Scholar 

  • Takuya, W., Masuhara, H.: A spontaneous code recommendation tool based on associative search. In: Proceedings of the 3rd International Workshop on Search-Driven Development: Users, Infrastructure, Tools, and Evaluation, pp. 17–20 (2011)

  • Thomas, S.W.: Mining Unstructured Software Repositories using IR Models. Ph.D. thesis, Queen’s University, Canada. Ph.D. Dissertation (2012)

  • Thummalapenta, S., Xie, T.: Parseweb: a programmer assistant for reusing open source code on the web. In: Proceedings ASE ’07, pp. 204–213 (2007)

  • Toshev, A., Taskar, B., Danillidis, K.: Shape-based object detection via boundary structure segmentation. Int. J. Comput. Vis. 99(2), 123–146 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  • Vanderlei, T.A., Durao, F.A., Martins, A.C., Garcia, V.C., Almeida, E.S., de L. Meira, S.R.: A cooperative classification mechanism for search and retrieval software components. In: Proceedings SAC ’07, pp. 866–871 (2007)

  • Wang, S., Lo, D., Jiang, L.: Code search via topic-enriched dependence graph matching. In: 18th Working Conference on Reverse Engineering, pp. 119–123 (2011)

  • Wang, J., Dang, Y., Zhang, H., Chen, K., Xie, T., Zhang, D.: Mining succinct and high-coverage api usage patterns from source code. In: Proceedings of the 10th Working Conference on Mining Software Repositories (MSR 2013), pp. 319–328 (2013)

  • Wang, S., Lo, D., Jiang, L.: Active code search: incorporating user feedback to improve code search relevance. In: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, ASE ’14, pp. 677–682. ACM, New York, NY, USA (2014)

  • Wang, S., Lo, D., Jiang, L.: Autoquery: automatic construction of dependency queries for code search. Autom. Softw. Eng. 23(3), 393–425 (2016)

    Article  Google Scholar 

  • Wightman, D., Ye, Z., Brandt, J., Vertegaal, R.: Snipmatch: using source code context to enhance snippet retrieval and parameterization. In: Proceedings of the 25th annual ACM symposium on User Interface Software and Technology, pp. 219–228 (2012)

  • Wu, V., Manmatha, R., Riseman, E.M.: Textfinder: an automatic system to detect and recognize text in images. IEEE Trans. Pattern Anal. Mach. Intell. 22, 1224–1229 (1999)

    Article  Google Scholar 

  • Yao, H., Etzkorn, L.: Towards a semantic-based approach for software reusable component classification and retrieval. In: ACMSE ’04, pp. 110–115 (2004)

  • Ye, Y.: Programming with an intelligent agent. IEEE Intell. Syst. 18(3), 43–47 (2003)

    Article  Google Scholar 

  • Ye, Y., Fischer, G.: Supporting reuse by delivering task relevant and personalized information. In: Proceedings International Conference on Software Engineering ’02, pp. 513–523 (2002)

Download references

Acknowledgements

This work is supported by the National Science Foundation Grant CCF1130822. Additional support has come from Microsoft and Google.

Author information

Authors and Affiliations

  1. Department of Computer Science, Brown University, Providence, RI, 02912, USA

    Steven P. Reiss & Qi Xin

  2. Google, 1600 Amphitheater Parkway, Mountain View, CA, 94043, USA

    Yun Miao

Authors
  1. Steven P. Reiss
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yun Miao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qi Xin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Steven P. Reiss.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Reiss, S.P., Miao, Y. & Xin, Q. Seeking the user interface. Autom Softw Eng 25, 157–193 (2018). https://doi.org/10.1007/s10515-017-0216-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10515-017-0216-3

Keywords

Search

Navigation