Media History

Georgetown University
Graduate School of Art and Sciences
Communication, Culture & Technology Program

Professor Martin Irvine
CCTP 711: Semiotics and Computing Technologies
Fall 2019

This course is a seminar motivated by the following key questions for inquiry and learning:

(1) In studying the designs and uses of our technologies and media, what can we learn from a better understanding of the core human "symbolic capacity" -- in language, writing, images, abstract symbol systems (math, logic)?

How and why is the capacity for symbolic cognition (thought, imagination, communication with others) the foundation for all our technologies of meaning (for representing, mediating, communicating, storing, and transmitting symbolic forms)?

Everything we do in computing and digital media is based on, and is designed to be in the service of, one or more of our core human symbolic systems (language, writing, images, graphics, all representation systems in math and science). Why is this fact lost in all the marketing and industrialization of computer systems, computing devices, and everything designed for digital media?

(2) What can we learn about our computational, media, and communication technologies when we describe them as artefacts of human symbolic cognition? How are these media and technologies designed for symbolic functions?

(3) Why it matters! How can everyone, especially non-technical people, come to a better understanding of computing, digital media, and everything "technical" by understanding where the technologies come from, how and why they are designed the way they are, and how our current technologies fit in a longer story of human symbolic thought?

To provide methods and concepts for opening up these questions, this course will introduce major topics of current research on signs and human symbolic cognition, the processes of meaning-making in language and other sign systems, and the nature of symbolic and cognitive technologies (media, communication, computation and software). Our main research model comes from C. S. Peirce's approach to signs and symbol functions (semiotics), communication, and technology. Our readings will draw from an exciting body of new, converging research: from humanistic and social science disciplines (linguistics, semiotics, anthropology, philosophy) and from sciences and technical disciplines (cognitive science, computation, HCI design and interface theory, and AI).

The course will be led as a research seminar with students developing their own weekly and ongoing research and contributing their own findings and discoveries in the class sessions. We will use the seminar as a learning lab where we will map out the major interdisciplinary questions and research areas and ways to apply them to our core concerns in CCT.

The Interdisciplinary Research Context

The seminar will draw from ongoing research and theory in several intersecting fields: linguistics; the study of meaning systems and human symbolic artefacts (semiotics); communication and information theory; cognitive science approaches to media theory and human-computer interface (HCI) design; computational thinking and theories of computation; cognitive anthropology and archaeology; artificial intelligence; and philosophy of language and mind. We will study closely the recent models for computation as methods for representing and automating human symbolic abstraction, and some of the major open questions about computation, AI, and "cognitive computing." We will investigate the nature of symbolic cognition in all human meaning systems from language, writing, music, and images to computer code and digital media, and uncover how the functions of technical mediation in our recent technologies are extensions of core human cognitive abilities.

Course Objectives and Outcomes

This course will provide students with an overview of important, new interdisciplinary research and theory on symbolic cognition, symbolic systems, and technologies of meaning that can be used as (1) a foundation for developing productive areas of new research and/or (2) for reframing questions and problems as conventionally approached in specific disciplines. By the end of the course, students will have acquired the ability to pose questions and think critically with the intellectual resources in this interdisciplinary knowledge base, and to apply concepts and methods for analyzing symbolic systems, computation, and digital media technologies. These interdisciplinary approaches will also enable students to make their own syntheses of knowledge for application in interface design, media representations, technologies for learning and social communication, and applications in organizational communication.

Classroom location: CCT Studio, Car Barn 318

Course Format and Syllabus Design

The course will be conducted as a seminar and requires each student’s direct participation in the learning objectives in each week’s class discussions. The course has a dedicated website designed by the professor. The web syllabus provides a detailed week-by-week "learning map" with links to weekly readings (in a shared Google Drive folder). Each syllabus unit is designed as a building block in the interdisciplinary learning path of the seminar.

To facilitate learning, students will write weekly short essays in a Wordpress site that reflect on and apply the main concepts and approaches in each week’s unit. Students will also work in groups for in-class exercises and for collaborative presentations.

Grades:

Grades will be based on:

(1) Weekly Class Participation: Weekly short writing assignments (in the course Wordpress site) and participation in class discussions (25%). Weekly short essays must be posted by 10:00AM for each class day so that students will have time to read each other's work before class for a better informed discussion in class.
(2) A group collaborative presentation on a research question posed in the course (25%).
(3) A final research project written as a rich media essay or a creative application of concepts developed in the seminar (50%). Due date: one week after last day of class.
(Final projects will be posted on the course Wordpress site, which will become a publicly accessible web publication with a referenceable URL for student use in resumes, job applications, or further graduate research) .

Professor's Office Hours
Wed. and Thurs. 12:00-2:00, and by appointment. I will also be available most days after class meetings.

Academic Integrity: Honor System & Honor Council
Georgetown University expects all members of the academic community, students and faculty, to strive for excellence in scholarship and in character. The University spells out the specific minimum standards for academic integrity in its Honor Code, as well as the procedures to be followed if academic dishonesty is suspected. Over and above the honor code, in this course we will seek to create an engaged and passionate learning environment, characterized by respect and courtesy in both our discourse and our ways of paying attention to one another.

Statement on the Honor System
All students are expected to maintain the highest standards of academic and personal integrity in pursuit of their education at Georgetown. Academic dishonesty, including plagiarism, in any form is a serious offense, and students found in violation are subject to academic penalties that include, but are not limited to, failure of the course, termination from the program, and revocation of degrees already conferred. All students are held to the Georgetown University Honor Code: see https://honorcouncil.georgetown.edu/system.

Instructional Continuity
In the event of a disruption of class meetings on campus from inclement weather or other event, we will continue the work of the course with our Web and online resources, and will arrange for online discussions and meetings with the professor by using the Google Hangout interface in our GU Google apps suite. I am also always available via email, and respond to student messages within a few hours or less.

Required:

  • W. Brian Arthur, The Nature of Technology: What It Is and How It Evolves. New York, NY: Free Press, 2009. [ISBN 1416544062]
  • Peter J. Denning and Craig H. Martell. Great Principles of Computing. Cambridge, MA: The MIT Press, 2015.
  • Luciano Floridi, Information: A Very Short Introduction. Oxford, UK: Oxford University Press, 2010. ISBN: 0745645720
  • Lev Manovich, Software Takes Command: Extending the Language of New Media (London; New York: Bloomsbury Academic, 2013).

Recommended:

  • George Dyson, Turing’s Cathedral: The Origins of the Digital Universe. New York, NY: Pantheon/Vintage, 2012. ISBN: 1400075998
  • James Gleick, The Information: A History, a Theory, a Flood. New York, NY: Pantheon, 2011.
  • Ray Jackendoff, Foundations of Language: Brain, Meaning, Grammar, Evolution. New York, NY: Oxford University Press, USA, 2003.
  • Janet H. Murray, Inventing the Medium: Principles of Interaction Design as a Cultural Practice. Cambridge, MA: MIT Press, 2012.
  • Noah Wardruip-Fruin and Nick Montfort, eds. The New Media Reader. Cambridge, MA: MIT Press, 2003. ISBN: 0262232278

Links to Online E-Text Library and University Resources

Learning Objectives:

  • Foundational background for the course. Introduction to key concepts, methods, approaches.
  • Introduction to the schools of thought and major testable hypotheses that we will work with:
  • Establishing the context of research and theory in fields working in the question of human symbolic capabilities, symbolic cognition, actions, and embodiments in media and technology.

Introduction to the framework for our guiding research question:

  • How do we construct a model of meaning processes, symbolic thought and action, sign and symbol systems that allow us to unify all forms of symbolic activity from language and symbolic artefacts like artworks and music to mathematics, media technologies, computation, and software programs?

Course Introduction: Requirements and Expectations

  • Format of course, requirements, participation, weekly assignments, projects, outcomes.
  • Classroom rules: how to use PCs and mobile devices: no social media or attention sinks during class.

Using Research Tools for this Course (and beyond)

In class: Introductory Lecture and Presentation (Prof. Irvine) (Google Slides)

In-class discussion of basic concepts, examples, and case studies:
sign systems, symbolic-cognitive artefacts and computational devices

  • Sign and symbols systems: what, how, why?
  • Artefacts we think with: introducing symbolic and "artefactual" cognition
  • Defining terms: symbol/symbolic, sign, medium, artefact, interface, cognition, computation
  • Looking at: writing/texts, images, digital media, sign processes, "natural" algorithms

Weekly discussions on the course Wordpress site

  • Use the course Wordpress site for weekly discussion and your journal or ideas and questions.
  • Instructions for your weekly writing assignment (read carefully before posting!)
Learning Objectives:

"Booting up" our thinking and research with key concepts from the fields that we will be drawing from for interdisciplinary model. In this orientation to studying semiotics and symbolic cognition, students will gain a sense of the disciplines, terminology (specialized vocabularies), key concepts, arguments, and guiding hypotheses in the relevant fields.

Interdisciplinary Challenges

When beginning learning in a new field or research approach, we are always entering intellectual conversations and debates mid-stream and already in progress (sometimes for hundreds of years!). To understand and participate, we must always begin with getting a sense of the major questions, schools of thought, the problem set, the scope of the research domain, and the range of disciplines that converge on a problem. We have to sort out and map out the inherited concepts, terms, methods, and discourses, the best we can, and learn what motivates the current research questions.

As you go through the course, engage directly with the assumptions, arguments, concepts, and array of disciplinary vocabularies in these representative major statements, which everyone working in the relevant field is assumed to know. When reading, you should ask for yourself:

  • Why are these arguments, terms, and concepts important for our overall research program, and how can we use them to think with? how can we test the validity of different (often rival or competing) hypotheses?
  • What conversations, debates, and dialogues are these statements participating in? what were/are the contexts, disciplinary/professional conversations that we need to know to understand how/why the arguments are framed as they are?
  • What are the key "take aways" in terms of getting what the arguments and assumptions are about, and using them as possible approaches in our own learning and research?

Don't be concerned that you don't understand the terms, concepts, and arguments in the readings. No one does on a first reading. We will build out the contexts and backgrounds for understanding the assumptions and arguments in these key statements and why they are important. Through the course, we'll work through the consequences of these concepts and methods and ways to critique them in light of the most research research and theory.

Using the Course Reader

This is an anthology of excerpts from major statements that represent the approaches, arguments, research, and concepts that converge around our central topic. This is our first "common core" group of major inherited and recent statements that define the interdisciplinary "problem space" that we are investigating. Each of these readings (and the cluster of main arguments represented by the respective leaders in the various fields) have generated wide discussion and debate (some over many years), and students need to encounter them first hand to do significant thought and further research. (Use Google Scholar on any of these authors, texts, or concepts and you'll long chains of discussion, debate, and publication trails, but the important task is to begin learning the main issues, research approaches, and why they're important.)

Readings: Introduction the Multidisciplinary Field of Semiotics

  • Daniel Chandler, Semiotics: The Basics. 3rd. ed. London: Routledge, 2017. Excerpts.
    • Read pp. 7-38 for this week. This is a good introduction to terms and concepts if you are approaching the subject for the first time. It is written for humanities students, and we will supplement this approach with a broader interdisciplinary view that also applies to all forms of media and technology. We will be following the models and methods of C. S. Peirce, which are briefly covered here.
  • Martin Irvine, "Introduction to Sign Systems, Symbolic Cognition, Semiotics, and Technology" (Part 1)
    • This introduction is written specifically for our multidisciplinary context, and introduces some major terms and concepts from C. S. Peirce's semiotic theory for studying computation, media, and cognitive technologies.
    • Don't worry if you can't complete this reading this week; we will use it as a basic reference for our key concepts.
  • Selections from our Course Reader: Semiotics, Symbolic Cognition, and Technology: A Reader of Key Texts ( PDF).
    • You can survey and browse this Reader as a preview, but do not read in full. We will refer to this text throughput the course.
  • Links to shared Google Drive folders for weekly readings and research sources (GU login required)

Discussion and "workshop" in class:

  • We will have an open seminar discussion in class this week: survey the background texts for key terms and concepts, and we will work through the contexts and backgrounds in class, and apply the ideas to interesting examples and cases. You can cite an example you'd like to analyze in detail in your WordPress post, or simply come with questions that you'd like to cover.

First writing assignment (link to Wordpress site)

  • Even though these texts and concepts are new for you, does this "bigger picture" view begin to change the way you can think about the symbolic basis of all forms of expression, symbolic thought, media, communication, and computational technologies?
  • Try working with an example of a symbolic form with the introductory background (e.g., piece of music or shorter section of a song/composition, a photograph or image, an artwork, code in a web page or mobile app, a section of a text or speech, sentences in everyday conversation). Can you describe the process of pattern recognition in the instance you are studying: how do we go from perceptible structures to meanings (shared and collective)?

Learning Objectives and Main Topics:

Gaining a foundation in the implications of the research generated by the study of symbolic cognition in intersecting disciplines and sciences--anthropology, evolutionary psychology, archaeology, and cognitive science more broadly--and how this developing knowledge base allows up to ask better informed questions about human sign systems, technologies as cognitive artefacts, and mediated communication systems.

Background:
The human ability for making meaning in any kind of expression and embodying meaningful, collectively understood expression in material media technologies depends on shared symbols and symbolic cognition. Research in many fields continues to discover more and more about the consequences of being the Symbolic Species in Terrence Deacon's term. This week, you will learn the main concepts from cognitive science and archaeological research for describing the "cognitive continuum" from language and symbolic representation to multiple levels of abstraction in any symbolic representation (spanning writing, mathematics, symbolic media like images and combinations in film and multimedia, and computer code). One promising way of studying media, communication, and computational technologies is uncovering a "cognitive continuum" of accumulating capabilities for symbolic representation, abstraction, and material externalizations for storing and "off-loading" collective cognition in symbolic artefacts (all forms of media and communications using material and technical means), including computational processes and digitization.

Overview of Topics and Themes:
Symbolic Cognition, Sign Systems, Mediation > Cognitive Technologies


Within a broad cluster of fields--ranging from neuroscience to cognitive linguistics, cognitive anthropology, and computational models of cognition and artificial intelligence research--there has been a major convergence on questions and interdisciplinary methods for studying cognition, human meaning-making and the "symbolic faculty" generally, including all our cumulative mediations and externalizations in "cognitive technologies." Cognitive science has been closely related to computer science in seeking computational models for brain and cognitive processes, and proposing hypotheses that explain cognition as a form of computation.

Language > Symbol Combinatoriality > Abstraction > Mathematics > Machines > Computation

Symbolic

There's wide agreement that any account of symbolic systems needs to explain the human parallel cognitive-symbolic "architectures" that are based on:

  • (1) the functions of language and other symbol systems for enabling abstraction (generalized concepts), learning, and representations in memory of what its learned
  • (2) rules for combining sign/symbol components (an underlying syntax for forming complex and recursive expressions of meaning units),
  • (3) the intersubjective preconditions of symbolic structures "built-in" to meaning systems for collective and shared cognition, and
  • (4) material symbolic-cognitive externalizations (e.g., writing, images, artefacts, human built environment) transmitted by means of "cognitive technologies" (everything from writing and image making to digital media and computer code) which enable human cultures and cultural memory.

This recent interdisciplinary research convergence is a "game changer" for the way we think about human meaning systems, symbolic cognition, communication, and media technologies.

Readings

  • Kate Wong, “The Morning of the Modern Mind: Symbolic Culture.” Scientific American 292, no. 6 (June 2005): 86-95.
    [A short accessible article on the recent state of research on the origins of human symbolic culture and the relation between symbolic cognition and tool making and technologies.]
  • Terrence W. Deacon, The Symbolic Species: The Co-evolution of Language and the Brain. New York, NY: W. W. Norton & Company, 1998. (Excerpts from chapters 1, 3, 9, 11, 13.)
    [Deacon's work initiated a wide and ongoing interdisciplinary debate. He presents an important argument for the co-evolution of language, symbolic cognition, and culture with the human brain based on connecting the research from evolutionary brain science, anthropology, archaeology, and paleontology. Read for his main argument about language, symbols, and brain co-evolution. Many details and data from evolutionary neuroscience are, of course, outside our field. Do your best to work through Deacon's main hypotheses and conclusions. When following his main points, remember that we're not so much concerned about what is "right" or "wrong," but about the new hypotheses that come into view with his cross-disciplinary synthesis. (He adapts Peirce for his own interpretation and misunderstands Chomsky, but we can let the argument play out to see where it goes.) There continues to be lots of research on all these topics, if you want to go further.]
  • Merlin Donald, "Evolutionary Origins of the Social Brain," from Social Brain Matters: Stances on the Neurobiology of Social Cognition, ed. Oscar Vilarroya, et al. Amsterdam: Rodophi, 2007.
    [Donald's research is a major contribution to evolutionary brain science, and this short article is a summary of his key findings and conclusions. Although his argument about linking of all symbolic capabilities to mimetic cognition (the ability to imitate and represent others' activity in a social group) is debatable, the central questions he confronts are important for any theory of human symbolic cognitive development.]
  • Colin Renfrew, “Mind and Matter: Cognitive Archaeology and External Symbolic Storage.” In Cognition and Material Culture: The Archaeology of Symbolic Storage, edited by Colin Renfrew, 1-6. Cambridge, UK: McDonald Institute for Archaeological Research, 1999.
    [Important argument to supplement Merlin Donald's view about the evolutionary origins of the symbolic brain: material culture is part of the externalizing cognitive process.]
  • John C. Barrett, "The Archaeology of Mind: It's Not What You Think." Cambridge Archaeological Journal 23, no. 01 (2013): 1-17.
    [Barrett provides a good summary of the "state of the question" where evolutionary sciences and archaeology intersect on understanding the origins of symbolic cognition and use of artefacts.]

Presentation (for class discussion): "Cognition, Symbols, Meaning" (Part 1)

 

Documentaries and Recent News

Weekly Writing Assignment (link to Wordpress site)
Choose at least one topic to focus your thoughts and questions about the readings for this week:

  • What are some of the main hypotheses and research conclusions (so far) in the research literature above for the question of how we have evolved as a "symbolic species"?
  • What are the consequences of this research and major working hypotheses for studies of language, communication, symbolic cognition, tools, machines, and technology?
  • Do any of these recent research findings open up new ways for you to understand recent technologies in a deeper and longer historical continuum?

Learning Objectives and Main Topics

Learning the main concepts, terminology, and assumptions developed in contemporary linguistics as essential foundations for understanding and describing language, and by extension, all human symbolic capabilities, sign systems, and communication.

The Importance of Linguistics for Interdisciplinary Thinking and Research

The human capacity for language is the starting point of many disciplines and research programs in all aspects of communication, symbolic culture, and media. Many of the research questions and most of the terminology for the study of language and human symbol systems has been established by the various specialties of modern linguistics. The terms and categories for analysis in linguistics have also been widely used by other disciplines, and all students studying media, communication, and computation need to be familiar with the concepts and research programs in the major branches of linguistics. 

We can only do a top-level overview here, but familiarity with the core concepts and research questions will allow you to advance to other questions and topics in your own research. Each of the major topics of linguistic research and theory (especially syntax/grammar and semantics) involve major, ongoing research programs, schools of thought, and competing arguments and terminology. All aspects of linguistic research are now part of larger research programs in cognitive and brain sciences, as well as ongoing investigations in social sciences, philosophy, and humanities fields. Getting a grounding in the central research questions, concepts, and methods will open up many new paths for your own thinking and research.

Our Reference Model: Ray Jackendoff's "Parallel Architecture" Model of Language

We will use Ray Jackendoff's "Parallel Architecture" as a reference model for explaining the structure and principles of natural language, and explore ways to extrapolate from this model to other sign systems (e.g., music, images, semantic representation in software) to uncover how they may be like, or different from, language. Jackendoff's model is driven by unifying syntax, semantics, and the lexicon (the words in a language), and the model assumes that there are cognitive and symbolic "interfaces" between language and other sign systems. Jackendoff was Noam Chomsky's student at MIT in the 1960s, and while maintaining an expanded "generative grammar" framework, he has usefully developed his own research conclusions different from Chomsky's, and has synthesized many schools of thought and research programs in linguistics. He is also a musician, and has written important works on the structure of music and the parallels with, and difference from, language. His models and question-framing can be productively combined with C. S. Peirce's macro model of signs as generative rule-governed processes. When thought through together, Jackendoff's and Peirce's models provide a way to re-model all sign systems and symbolic functions as combinatorial structures with components working in parallel to create the "output" expressions that we understand as meaningful (any symbolic expression in words, images, sounds, and their combinations).

Why Learning Well-Researched Linguistic Methods are Important for All Our Fields

In CCT and beyond, we find many areas of intensive inquiry on all human communication and meaning systems in many disciplines covering the whole history of meaning systems and symbolic representation from the earliest records of symbolic expression to computational language processing and artificial intelligence. Many fields assume that the meaning systems of interest to the field (e.g., text, images, music, designed artefacts, multimedia, computer code) work "like a language" or "are a language" without clearly defining what a language is (or what model of Language is being assumed). In other words, don't we first need to clarify what the properties or features of language or a language are, for which other meaning or media systems can be like?

Semiotics is founded on the inquiry into generalizable principles for meaning-making and meaning systems, and often posits that language is our "primary modeling system." Is the "like a language" analogy or parallelism more than a vague observation? You can see that it's fundamental for us to begin with a model or description of language that is as clear and well-described as possible before assuming that other meaning systems can be "like" (a) language.

Contemporary linguistics provides a knowledge base of key concepts and terms for the descriptive levels, rules, and constraints that specify how language works and what a natural language is. Getting as precise as we can at the detail levels will help us think more clearly about important questions like, "are visual image genres a language?", "what do we mean by a 'computer language', code, and 'language processing'?", "is music a language; and/or individual music genres?", "how do we understand film, video, and multimedia genres with multiple combined 'languages'?", "how is language connected to intelligence, symbolic cognition, and other forms of human cognition"?.

Key terms and concepts: 

  • syntax, semantics, lexicon
  • generative grammar, open rule-governed combinatoriality
  • pragmatics (the contexts and situations of language use, shared assumptions, speech and discourse genres, and speech acts)
  • cognitive linguistics (the intersection of language and cognitive/brain science research)

yoda

Readings: Language as a Symbolic Cognitive System: The Linguistics Model

Introductions (in this order)

  • Steven Pinker, Video presentation on Language and the Human Brain (start here)
    • A well-produced video introduction to the current state of knowledge on language and cognitive science from a leading scientist in the field.
    • See also: Steven Pinker's website at Harvard University for a view of his work and career.
  • Martin Irvine, "Introduction to Linguistics and Symbolic Systems: Key Concepts" (intro essay).
  • Andrew Radford, et al. Linguistics: An Introduction. 2nd ed. Cambridge, UK: Cambridge University Press, 2009. Excerpts.
    • [This is an excellent text for an overview and reference. Review the Table of Contents so that you can see the topics of a standard course intro to linguistics. Read and scan enough in each section to gain familiarity with the main concepts. You don't have to read the whole selection of excerpts; focus on the Introduction to Linguistics as a field, and sections on Words (lexicon) and Sentences (grammatical functions and syntax).]
  • Ray Jackendoff, Foundations of Language: Brain, Meaning, Grammar, Evolution. New York, NY: Oxford University Press, USA, 2003. Excerpts and introduction to "the Parallel Architecture" model of Language.
    Also included in these excerpts is Jackendoff's extensive bibliography of references cited in the book.
    Read pp. 3-17 (from Chap. 1), and 38-44 (from Chap. 3) and begin Chap. 5 on the Parallel Architecture; we will continue with Jackendoff next week.
    • Jackendoff has developed a useful synthesis of many important developments in linguistics in a "unification" model of language components that he calls the "parallel architecture." This book is a highly detailed, argumentatively nuanced, exhaustively researched compendium of the major issues in linguistics (especially from the cognitive and generative approaches) that is difficult to summarize, but there are important "take aways" for us. In reading Jackendoff's work, you will be jumping in midstream to a highly-detailed 30-year ongoing debate about syntax and semantics, but he usually provides a good orientation to the issues. He also provides useful conceptual hooks and links for ways to talk meaningfully about the structures of language in relation to other symbolic systems like music and visual narratives.
    • See also: Ray Jackendoff's website at Tufts University for a view of his work and career.

Supplemental: for Further Background and Research [Optional]

  • Steven Pinker, "How Language Works." Excerpt from: Pinker, The Language Instinct: How the Mind Creates Language. New York, NY: William Morrow & Company, 1994: 83-123.
    [Accessible introduction to linguistic analysis of syntax and the role of language in cognition; underlying theory to his video presentation above.]
  • Steven Pinker, "The Cognitive Niche: Coevolution of Intelligence, Sociality, and Language." Proceedings of the National Academy of Sciences 107, Supplement 2 (May 5, 2010): 8993–99.
    [A short accessible essay (written on the occasion of appraising Darwin's continuing value) summarizing current issues and state of knowledge on human evolution and cognition.]

For discussion (in class):
Experiment with Visualizing Sentence Structure through Computational Parsing

  • XLE-Web: A sentence parsing tool and syntax tree generator that maps both the "Constituent Structure" and "Lexical Functional Grammar" models of generative grammar. Choose "English" and map any sentence for its constituent (c- ) and functional (f- ) structure! (Uses linguistic notation from two formal systems.) (The term "parse" comes from traditional grammar for breaking sentences down into their "parts of speech" [pars/partes: Latin for "part/parts"]). Syntax parsers are used in computational linguistics and all complex text analysis in software and network applications. Web search algorithms and Siri voice recognition and interpretation software have to use parsers for generating a map of the probable grammatical structure of natural language phrases before processing probable semantic searches. So here you get a visualization of what Siri and Google have to do in milliseconds behind the scenes before the software can initiate a search or other command.
  • Try the test sentence "I like dark beer but dark beer doesn't like me." You will see how the parser generates several grammatical structure options because of the ambiguities in switching subject and object with the verb [like]. Then try any sentence; try some very complex ones (compound subjects, many conjunctions and clauses, etc.).

Presentation for class discussion (continued): "Cognition, Symbols, Meaning" (Part 2: On Language)

Weekly Writing Assignment (link to Wordpress site)
Choose at least one topic to focus your thoughts and questions about the readings for this week:

  • Drawing from the readings for this week (and any connections to prior weeks), how would you answer these basic questions for someone who has little or no knowledge of linguistics as a science or field of research: What is language? What is "a language," What are the essential features that enables a language to be a language? 
  • From these foundations, we will go on to ask other important questions:
    What are the implications of using the features of language as the model for other symbolic systems (visual, audio, and multimedia combinations) and for most forms of communication and media?
    Even before we investigate the knowledge base from sciences and disciplines that converge on the study of other symbolic systems, can you think through what it would mean to study other sign systems by assuming that they work like a language (e.g., making meaningful rule-governed combinations of symbolic elements in expressions that are understood collectively)? Try working with one or two of the main linguistic concepts to see if they provide extensible models for understanding and describing other meaning systems (like visual genres or music genres) in the forms of media and communication we use every day.
  • What seems intuitively clear at this point, and what is difficult or unclear?

Learning Objectives and Topics:
Expanding from the Principles of Language to all Human Symbolic Systems

Developing an understanding of language and other sign systems by using concepts and research from the "sign systems" (semiotics) perspectives in the larger cognitive science framework. Becoming familiar with the main traditions of semiotic theory and practice, and how these intersect with linguistics and other traditions of thought on communication and media studies (over the nest few weeks of the course).

The terms and concepts used in the related disciplines are not consistent, so we will use C. S. Peirce's key terms and concepts, and ways that they have been expanded and re-interpreted for analyzing communication, media, and computational technologies. Peirce's approach has great heuristic value (that is, what we can discover, or "de-blackbox" by thinking with and applying the concepts and models). In this unit, students will learn how to connect linguistics research with more generalizable semiotics (sign systems theory) and research on symbolic cognition.

We will investigate ways to describe the features of language that are extensible to other symbolic systems and to the technologies that mediate them. We will expand from Ray Jackendoff's insightful "parallel architecture" model of language (which usefully rolls up many research questions in cognitive linguistics) for understanding the cognitive functions of signs and symbols in the Peircean semiotics tradition. We will begin surveying the kinds of knowledge resources we need to bring to the main question of human meaning-making in all symbolic forms, the individual and collective cognitive processes required and involved, and the role of externalized symbolic media, artefacts, and technologies in our cultural/historical representations and transmission of meaning.

Readings

Continuing/Concluding:

  • Ray Jackendoff, Foundations of Language, selections on the "Parallel Architecture" model of language as a combinatorial system. Chap. 5.5, pp. 123-128; Chap. 7, pp. 196-200.
    • [Go to the last sections of the excerpts; continued from last week. This is Jackendoff's key hypothesis for integrating how speech sounds (phonology), semantics, and the lexicon work in a unified parallel (not linear) structure. Do your best in thinking with the linguistic concepts from last week's readings and how Jackendoff works with them. Think about what linguists mean by "interfaces" between the language components. As we will see, his model provides a useful "bridge" between the generative linguistics research community and the study of other sign and symbol systems in semiotics (which mainly leave out syntax and big questions in combinatorial rules).]

Introductions and primary texts for semiotics and sign systems research

In Class:
Presentation for discussion: "Cognition, Symbols, Meaning" (Parts 2 and 3)

  • Begin studying the presentation on your own. We will only begin part 3 this week.

Introductory Discussion on Symbolic Feature Mapping: The Semiotic Matrix

  • Introduction to using this spreadsheet as a tool to think about the features, functions, and properties of our symbolic systems and how the "symbolic-cognitive architectures" are distributed across sign systems (individually or in combination like film and video). Any map is incomplete: this is for heuristic purposes only (an aid for discovery). The Google shared sheet is set for comments by all students in the course; make your own analysis and contribute to the map. We will develop the map over the next few weeks.
  • In-class group exercise on analyzing meaning structures in everyday genres (written genres, images, music genres, web pages, photograph genres, movie genres).

Weekly Writing Assignment (link to Wordpress site)
Choose at least one topic to focus your thoughts and questions about the readings for this week:

  • Choose an example of an everyday symbolic genre (movie scene/shot; musical work or section of a composition; image or artwork as (1) an instance of its genre[s]) and as (2) an implementation of one or more sign system. Using the terms, concepts, and methods in the readings so far, describe as many of the features that you can for how the meanings we understand (or express) are generated from the structures of the symbolic system(s). Does the "Parallel Architecture" model open up ways to detect and describe the features and functions of meaning systems beyond language?
  • Hint: it's useful to work on an example that is new to you or not yet very well known. Symbolic forms that we know well (a movie, artwork, or song, for example) often seem "automatic" in how we understand the kinds of meanings going on, and therefore difficult to analyze.
Learning Objectives and Main Topics

Learning the main concepts and approaches in C. S. Peirce's model for explaining signs and symbolic thought: sign relations and symbolic processes for applications to all symbolic systems and semiotic technologies.

Merging Peirce's philosophy with other disciplines, we find that semiotics can be used as a de-blackboxing method for understanding how meaning systems and technologies of meaning work. Semiotics shows that any representation or materially embedded expression an interface to the meaning system(s) in which it appears and makes it possible both as a social artefact and an instance in a system of technical mediation.

Readings:

Additional Sources (Optional):

  • Peirce, Lecture on Triadic Relations and Classes of Signs, Lowell Institute, Boston, 1903.
    [This is a famous text from a series of public lectures that Peirce gave in 1903. He provides a complex description of 10 classes of signs logically extended from the basic functions of the icon, index, and symbol.]
  • Richard J. Parmentier, Signs in Society: Studies in Semiotic Anthropology. Bloomington, IN: Indiana University Press, 1994. Excerpts.
    [A useful summary of Peirce's main concepts. Don't get stuck in the terms that Peirce invented to try to account for all the categories of sign functions. In chapter 1, focus on the first 3 sections (through "Language and Logic").]

Research and Reference Sources for C. S. Peirce

Weekly Writing Assignment (link to Wordpress site)
Choose at least one topic to focus your thoughts and questions about the readings for this week:

  • Working in your teams, annotate The Students' Annotatable Peirce (sharable Google doc) with your comments and questions. We will follow up in class with a discussion of how to apply the concepts in a method for semiotic analysis of any example.
  • For your individual thinking, choose and post an example of a symbolic expression or artefact (any medium) to describe with Peirce's concepts (we will focus on art and music examples in class, but you can choose any example to discuss). (You do not have to do extensive analysis; we will work on it in class.)

Learning Objectives

  • Learning the major terms, concepts, and technical applications for communication and information theory as defined and used in electronics, computation, and digital media for signals and digital-electronic states.
  • Learning how and why the engineering definition of information in our electronic and digital context is essential for the way all information and media technologies work (based on discrete states represented by bits, units of binary values, transmitted in time to different physical locations in space).
  • Learning why these technical implementations for signals are necessarily “pre-semantic” (prior to meaning correlations) or “semantic-agnostic” (not-knowing or unaware of meaning-correlations), but must presuppose motivated meanings by senders and receivers of messages and any kind encoded information. Without human meaning intentions there would be no sending and receiving of messages encoded in signals!
  • Learning why the engineering transmission model (sender-signal-receiver) is not applicable or sufficient for describing the communication of meaning, values, and intentions that we encode in sign/symbol units.
  • Learning how to complete the description of human meaning making in electronic media representations with the current knowledge and conceptual models provided by linguistics, semiotics, and cognitive science.

Key Terms and Concepts:

  • Information defined as units of probability and differentiation (differentiability) from other possibilities.
  • The Transmission Model of Communication and Information.
  • The dominant conceptual metaphors (and their consequences):
    conduit (“channel”), container (and “content”), source, destination.
  • The “bit” (binary unit) as minimal encoding/encodable/encoded unit (2 possible values in base 2 number system with one of two values represented = 1 bit of information), and how/why binary code units map onto electrical circuits and electronic states (a value can be represented in an electronic state).
  • Discrete (digital/binary) vs. Continuous (analog) signals or information sources.

Video Lessons (for background)

Readings: The Engineering Model of Information and Semiotic Foundations

  • Martin Irvine, "Introduction to the Technical Theory of Information" (Information Theory + Semiotics)
  • Luciano Floridi, Information, Chapters 1-4. PDF of excerpts.
    [For background on the main traditions of information theory, mainly separate from cognitive and semantic/semiotic issues.]
  • James Gleick, The Information: A History, a Theory, a Flood. (New York, NY: Pantheon, 2011).
    Excerpts from Introduction and Chapters 6 and 7.
    [Readable background on the history of information theory. I recommend buying this book and getting as deeply into the issues that Gleick explains as possible.]
  • Ronald E. Day, "The ‘Conduit Metaphor’ and the Nature and Politics of Information Studies." Journal of the American Society for Information Science 51, no. 9 (2000): 805-811.
    [Models and metaphors for "communication" have long been constrained by "transport", "conduit" ("pipes"), and "container/content" metaphors that provide only the linear signal processing view of a larger contextual and meaning-motivated process. If the "conduit" and "container/content" metaphors are not useful, how can we best describe "communication/information" and "meaning" that accounts for all the conditions, contexts, and environments of "meaning making"? Are network and other systems metaphors better than the linear point-to-point metaphors?]
  • James Carey, "A Cultural Approach to Communication" (from James W. Carey, Communication as Culture: Essays on Media and Society. Revised edition. New York and London: Routledge, 1989. )
    ["Communication is a symbolic process whereby reality is produced, maintained, repaired, and transformed." An influential essay by a major founder of the modern field of Communication Studies. Carey repositions the study of communication and mediating technologies in cultural-symbolic meaning systems. Drawing from the tradition of philosophy known as American Pragmatism (C.S. Peirce, William James, John Dewey), Carey concludes with views close to those of Peirce on the inclusive view of symbolic thought and all media of representation and communication.]

In-Class:

Weekly Writing Assignment (link to Wordpress site)
Choose at least one topic to focus your thoughts and questions about the readings for this week:

  • The signal-code-transmission model of information theory has been the foundation of signal and message transmission systems in all communication technology from the telegraph and telephone to the Internet and digital wireless signals. Why can't we extrapolate from the "information theory" model to explain transmission of meanings? Where are the meanings in our understanding of messages, media, and artefacts? Hows does semiotics complete the information-communication-meaning model by accounting for the contexts, uses, and human environments of the presupposed meanings which are not properties of the signals that are designed to register the "information" of a "transmitted message"?
  • From what you've learned about symbol structures so far, can you describe how the physical/perceptible components of symbol systems (text, image, sounds) are abstractable into a different kind of physical signal unit (electronic/digital) for transmission and recomposition in another place/time? (Hint: as you've learned from Peirce and semiotic theory, meanings aren't properties of signals or sign vehicles but are relational structures in the whole meaning-making process understood by senders/receivers in a meaning community.)
  • Following on with a specific case: how do we know what a text message, an email message, or social media message (complete with images and emoji tokens) means? How does the design of the electronic, digital media system enable us to convert signals into material-perceptible patterns (representations) that become part of a symbolic system? What kinds of communication acts understood by communicators are involved that must be supplied on the symbolic, not signal, level?

Learning Objective and Main Topics:

We have studied how sign and symbol systems require material and physical representation, and that signs and symbols are collective and link individual minds to others and to a shared representable world. Semiotics, then, merges with studies of collective and extended cognition in cognitive science. A new field of "cognitive semiotics" has now been established, and most of this research extends C. S. Peirce's model of sign systems and sign processes. This week, we will focus on learning these main concepts:

  • Learning the approaches to distributed and extended cognition as they apply to cognitive technologies, technical mediation, symbolic artefacts, and an enlarged view of semiotics.
  • Gaining a working knowledge of the main research questions in these fields of cognitive science and philosophy as a starting point for pursuing individual research or further inquiry.
  • Learning how to apply the hypotheses to analyzing and explaining our own uses of cognitive technologies, symbolic artefacts, and sign systems (writing, image technologies, music, film).

Major Terms and Concepts:

  • Extended Mind
  • Distributed cognition
  • Embodied cognition
  • Cognitive off-loading
  • Cognitive scaffolding
  • Distributed agency

Readings: Introductions to Distributed and Extended Cognition:

  • Martin Irvine, Introduction to the Theory of Extended Mind and Distributed Cognition.
  • Andy Clark and David Chalmers. "The Extended Mind." Analysis 58, no. 1 (January 1, 1998): 7–19.
    [The first version of the argument for the hypothesis that set off a large research conversation and debate. Also reprinted in Clark's Supersizing the Mind: Embodiment, Action, and Cognitive Extension (below)]
  • Andy Clark, Supersizing the Mind: Embodiment, Action, and Cognitive Extension (New York, NY: Oxford University Press, USA, 2008).
    [Excerpts from the Forward by David Chalmers, pp. ix-xi; xiv-xvi (attend especially to the comments in the last 3 pages of the Forward); Introduction and Chapter 1.3: "Material Symbols" (especially the concept of "cognitive scaffolding").]
  • James Hollan, Edwin Hutchins, and David Kirsh. “Distributed Cognition: Toward a New Foundation for Human-computer Interaction Research.” ACM Transactions, Computer-Human Interaction 7, no. 2 (June 2000): 174-196.
    [This is an important summary of research conclusions from leaders in cognitive science and its relations to technology and HCI. Hutchin's justly famous book, Cognition in the Wild (1996), provided empirical validation for understanding cognition as involving and requiring a larger system of human interactions outside and beyond individual minds/brains.]
  • Jiajie Zhang and Vimla L. Patel. “Distributed Cognition, Representation, and Affordance.” Pragmatics & Cognition 14, no. 2 (July 2006): 333-341.
  • Itiel E. Dror and Steven Harnad. "Offloading Cognition Onto Cognitive Technology." In Cognition Distributed: How Cognitive Technology Extends Our Minds, edited by Itiel E. Dror and Stevan Harnad, 1-23. Amsterdam and Philadelphia: John Benjamins Publishing, 2008.

Optional and Supplementary: Research Sources and Going Further

  • Riccardo Fusaroli, Nivedita Gangopadhyay, and Kristian Tylén. "The Dialogically Extended Mind: Language as Skillful Intersubjective Engagement." Cognitive Systems Research 29-30 (September 2014): 31–39.
    [Extended and collective cognition is fundamentally intersubjective, as our use of symbolic artefacts and the fundamentally dialogic basis of ordinary language use shows. ]
  • Hutchins, Edwin. "The Cultural Ecosystem of Human Cognition." Philosophical Psychology 27, no. 1 (February 2014): 34–49.
    [A valuable up-to-date summation of the state research and theory by a leader in cognitive science research.]

Weekly Writing Assignment (link to Wordpress site)
Develop this topic to focus your thoughts and questions about the readings for this week:

  • Applying some of the concepts in this weeks readings, use a common representational practice (e.g., writing and drawing on a board, paper, and/or screen) or a technical artefact (a group of functions in a PC or mobile device, not the whole bundle) and analyze the way we distribute, extend, and off-load cognitive functions that working together produce what we experience as thinking, communicating, completing a cognitive task, and/or representing and expressing intersubjectively accessible meanings.
  • (Consider how our repertoire of sign and symbol systems -- as implemented in physical representations and technical mediation -- support distributed and extended cognition, and how -- or whether -- the recent theories confirm or extend Peirce's earlier views.)

Learning Objectives and Main Topics:

This unit focuses on the key concepts in computation and core models for programming, software, computer and digital media. We will approach the questions from a non-specialist perspective, but it's important for everyone to get a conceptual grasp of the core ideas in computation because they are now pervasive throughout many sciences (including the cognitive sciences), and are behind everything we do daily with computational devices, information processing, and digital media (for example, the Google algorithms for searches, all the apps in mobile devices, the software functions for displaying and playing digital media). We will focus on foundational concepts that can be extended for understanding today's environment of "computation everywhere" and "an app for everything." 

  • Understanding the main conceptual foundations of computation, and how is related to the continuum of human symbolic cognition and generating meaning in levels of symbolic abstraction.
  • Understanding computation in relation to information theory, semiotics, and communication processes.
  • Learning the key concepts in "computational thinking" and the design principles in computation and software code.

Introductory Videos:

Readings:

  • Martin Campbell-Kelly, "Origin of Computing." Scientific American 301, no. 3 (September 2009): 62–69.
  • Jeannette Wing, Computational Thinking (Video)
    [Introduction to a way to make computing accessible to non-CS students.]
  • Jeannette Wing, "Computational Thinking." Communications of the ACM 49, no. 3 (March 2006): 33–35.
    [Short essay on the topic; Wing has launched a wide discussion in CS circles and education.]
  • Denning and Martell, Great Principles of Computing, Chapters 4-6.
    • The book combines the work done in prior studies and Denning's long-term project on "Great Principles" for the ACM. Earlier statements that can help for background. See:
      Peter J. Denning, "The Great Principles of Computing." American Scientist, October, 2010.
  • Subrata Dasgupta, It Began with Babbage: The Genesis of Computer Science. Oxford, UK: Oxford University Press, 2014. Excerpts: Prologue and Chapter 1.

Main Reading for Introduction to Coding Tutorial:

  • David Evans, Introduction to Computing: Explorations in Language, Logic, and Machines. Oct. 2011 edition. CreateSpace Independent Publishing Platform; Creative Commons Open Access: computingbook.org/.

    Focus on this text as the core reading to prepare you for the concepts in the Code Academy tutorial. Focus on chapters 1-3 (Computing, Language, Programming); others as reference and as your time and interest allow at this point. You can always return to other chapters for reference and self-study.
    [This is a terrific book based on Evans' Intro Computer Science courses at the University of Virginia. The book is open access, and the website has updates and downloads.]

For Reference: Background on the Technical Design of Today's Computers

  • David A. Patterson, and John L. Hennessy. Computer Organization and Design: The Hardware/Software Interface. 5th ed. Oxford, UK; Waltham, MA: Morgan Kaufmann, 2013. Excerpts from Chapter 1.
    [Excellent overview of important concepts for system architecture from PCs to tablets. For beginning computer engineering students, but accessible.]
  • Ron White, How Computers Work. 9th ed. Indianapolis, IN: Que Publishing, 2007. Excerpts.
    • Part One: Boot Up Process (History and Architectures)
      [Use for reference and conceptual understanding of basic PC computing architecture. This is extensible to mobile devices with further modularization and miniaturization of components.]
    • The Basics (hardware and software architectures)
    • Software Applications
      [Use this book as a reference for the operational nuts and bolts of hardware components and the computational principles behind them. Note that the systems architecture described is applicable to the miniaturization and modular design of mobile devices as functions become integrated in fewer chips and smaller space.

Main Assignment: Hands-On Learning Project On Code Academy

  • Create a free account on Code Academy: www.codecademy.com
    • Sign up for the self-paced tutorial lessons on the Python programming language "track": (www.codecademy.com/tracks/python)
    • The tutorial will prompt you through the lessons. Go as far as you can this week.
    • These lessons will guide you through basic computing concepts and also let you write some basic code and see the results when it runs.
    • Python is now the most widely used "teaching language" in introductory computer science courses. The concepts learned about programming methods here are extensible to most other computing contexts (including Web interactive content and mobile apps).
  • And if you have time and interest, you can follow up with the Web Fundamentals "track":
    ( http://www.codecademy.com/tracks/web)
    (other tracks: http://www.codecademy.com/tracks).

Follow the Code: Signs for Meanings, Signs for Operations

Presentation (In-Class): Computing as Symbolic Process

 

Weekly Writing Assignment (link to Wordpress site)
Choose at least one topic to focus your thoughts and questions about the readings for this week:

  • Describe what you learned from working through the CodeAcademy tutorial and making connections to the computing principles introduced this week. Were any key concepts clearer? What questions can you describe now after having a little more background.
  • Can you see how a programming language (and thus a software program or app) is designed to specify symbols that mean things (represent values and conceptual meaning) and symbols that do things (symbols that are interpreted in the computer system to perform actions and/or operations on other symbols). Computation (or "running" software) is a way of defining transitions in information representations that return us interpretable symbol sequences in chains of "states" that combine meanings and actions. (This is what the software layers running on your device right now are doing to render the interpretable text, graphics, images, and window formatting from the digital data sources combined in a Web "page," image or video file, and many other behind-the-scenes sources.)

Learning Objective and Main Topics:

  • Learning the background history for the models of computation that led to the development of interfaces for human symbolic interaction with programmable processes.
  • Understanding the important steps in the transition of computing from the post-War environment to the expansion of computation applied to knowledge and cognitive needs.
  • The conceptual origins for the technical development of graphical interfaces in the "human computer interaction" (HCI) design discipline.
  • Learning the concepts behind the technical architectures in all our devices that support user interfaces as semiotic and cognitive interfaces.

Readings

Historical and Conceptual Background: How do we get from early computers to today?

  • Martin Irvine, Introduction to Symbolic-Cognitive Interfaces: History of Design Principles (essay). Read Parts 1-2.
    [I've synthesized a lot of background history for today's design concepts. Includes a research bibliography if you want to follow up on any of these topics.]
  • Mahoney, Michael S. "The Histories of Computing(s)." Interdisciplinary Science Reviews 30, no. 2 (June 2005): 119–35.
    [Important background on the different research and development communities behind concepts and applications for computing.
    Note definitions of computing as symbolic processing, pp. 127-129.]
  • John S. Conery, “Computation Is Symbol Manipulation.” The Computer Journal 55, no. 7 (July 1, 2012): 814–16. [2 pp.] (The final version of a paper presented at the ACM Ubiquity Symposium on Computation: Ubiquity 2010, November, 2010.)
  • Lev Manovich, Software Takes Command, pp. 55-106, on the background for Allan Kay's "Dynabook" Metamedium design concept. Excerpts in pdf.
  • Bill Moggridge, ed., Designing Interactions. Cambridge, MA: The MIT Press, 2007. Excepts from Chapters 1 and 2: The Designs for the "Desktop Computer" and the first PCs.
    [This is valuable collection of interviews with the main designers of the interfaces and interaction principles that we use every day. Includes interviews with and background about Doug Engelbart and the design team at Xerox PARC with Allan Kay (Stu Card and Larry Tessler).]

Computers and Interfaces for Knowledge- and Meaning-Making:
Historical and Conceptual Backgrounds for Computer Interface Designs

  • Collection of original documents in pdf. You do not need to read these texts fully, but review them for their historical significance and as they are referenced in the readings for this week. (Graduate students should have access to the primary texts in their original form.)
    • Contents:
    • Vannevar Bush, "As We May Think," Atlantic, July, 1945.
      • A seminal essay on managing information and human thought by a leading computer engineer and technologist during and after World War II. Bush's pre-modern computing extrapolations lead to the concepts of GUIs (graphical user interface for computers), hypertext, and linked documents. His conceptual model, though not yet implementable with the computers of the 1940-50s, inspired Doug Engelbart and other computer designs that followed in the 1970s-80s and on to our own hypermediated era.
      • Wikipedia background on this essay, and Bush's concept of the "Memex", precursor to hypertext information systems.
    • Ivan Sutherland, "Sketchpad: A Man-Machine Graphical Communication System" (1963)
      • Expanding on techniques for screen interfaces for military radar, Sutherland was way ahead of his time, and it took many years for the whole combinatorial array of technologies to catch up for implementing the concepts. The Sketchpad concepts inspired Engelbart, Alan Kay (Dynabook), and the development of all screen interactive technologies that we know today.
    • Douglas Engelbart, "Augmenting Human Intellect"
      • Engelbart's research and development teams at Stanford in the 1960s-70s were influenced by Vannevar Bush's vision and motivated by a new conception of computers not simply as business, government, or military machines for instrumental ends but as aids for core human cognitive tasks that could be open to everyone. Engelbart's expansion of computation into user interaction led to:
        • The "Desktop" metaphor for a virtual interface to a user's content
        • The mouse and pointer cursor
        • Display editing and outline processing
        • Multiple remote online users of a networked processor
        • Multiple windows
        • Intra-file linking, Hypertext and Hypermedia linking ("clickable" linking between media objects)
      • Also available online at the Doug Engelbart Institute:
        HTML annotated edition of "Augmenting Human Intellect: A Conceptual Framework" (the Doug Engelbart Institute site).
      • For the original design concepts for the mouse, see Engelbart's Patent Application (with diagrams) for an "An X-Y Position Indicator for a Display System" (aka, mouse).
      • Background on Doug Engelbart at the Computer History Museum and the influence of Vannevar Bush's ideas.
    • Xerox PARC, Alan Kay, and the Dynabook/ Metamedium Concept for a "Personal Computer"
      • Alan Kay and Adele Goldberg, “Personal Dynamic Media” (1977), originally published in Computer 10(3):31–41, March 1977. (Cambridge, MA: The MIT Press, 2003), 393–404. 
      • Alan Kay's original paper on the Dynabook concept: "A Personal Computer for Children of all Ages." Palo Alto, Xerox PARC, 1972).
        [Wonderful historical document. This is 1972--years before any PC, Mac, or tablet device.]
      • Interview with Kay in Time Magazine (April, 2013). Interesting background on the conceptual history of the GUI, computer interfaces for "interaction," and today's computing devices.

Video Documentary: Demo of Ivan Sutherland's Sketchpad, Lincoln Labs, MIT (c.1963)

Reference Library (shared folder): major texts on computation and computer history (GU students only)

Weekly Writing Assignment (link to Wordpress site)
Choose at least one topic to focus your thoughts and questions about the readings for this week:

  • Referring to two or more of the readings and sources, describe the key concepts and technical implementations that enabled computation to be re-conceived and "interfaces" designed in relation to more universal human cognitive and semiotic uses or needs?
  • Referring to two or more of the readings and sources, describe the developing concepts of "interfaces" and "interactions" for using computers (even before possible technical implementations) and how many have been realized or still not implemented? What other paths remain to be realized in the concepts and possibilities for computing but have not (yet) been implemented commercially in all the software and device interfaces we now take for granted?

Learning Objective and Main Topics:

Continuing from last week, we will study the key principles and concepts of contemporary graphical, interactive software and digital media design.

Key terms and concepts:

  • Medium/media as social-technical implementations of communication and meaning functions maintained by roles in a larger cultural, economic, and political system.
  • Interface as a metasymbolic physical-material "relay" for "users" as social-cognitive agents of interactive (non-terminating) computer systems that "mediate" our sign and symbol systems in digital media form.

Readings:

Metamedia, Remediation, and Interactive Design as Models of/for Semiotic Processes

  • Brad A. Myers, “A Brief History of Human-Computer Interaction Technology,” Interactions 5, no. 2 (March 1998): 44-54.
    [This excellent synthesis of the history was written ten years ago, and we continue to use the interface design principles summed up here. Think about how the different design concept "leaps" (with supporting technologies as they became available) were motivated by semiotic-cognitive uses and finding ways to bring more cognitive agency to using computer systems and digitized symbolic media types.]
  • Peter Wegner, “Why Interaction Is More Powerful Than Algorithms.” Communications of the ACM 40, no. 5 (May 1, 1997): 80–91.
    [Do your best to catch the key concepts here (this is a famous statement in computer science). "Interactive Computing" is the current paradigm for combining non-terminating software (software that keeps running) -- an operating system, software applications like word processors and Web browsers -- with graphical interface "inputs," controls, and ongoing "output" representations. Interactive computing is based on designing computable processes for human sign actions in parallel and concurrent processes that can be redirected while in process. Our contemporary "interactive" software model is now the opposite of the Turing-Von Neumann model of one sequential process at a time. We have designs for combining and integrating "symbols that mean" and "symbols that do" in many kinds of continuous, concurrent processes that produce many transformable states of interpretable representations (symbolic patterns) for ongoing interpretations (as Peirce described).]
  • Jay David Bolter and Richard Grusin, Remediation: Understanding New Media. Cambridge, MA: The MIT Press, 2000. Excerpt from Introduction and Chapter 1.
    [This is a famous book in media studies. "Re-mediation" (re-represention in a digital medium) is a concept parallel with metamedia design, which includes interfaces for the whole GUI system as meta-interfaces (an interface for combining interfaces). Attend to their argument about the "double logic of re-mediation". Interpreted semiotically, "remediation" is "re-tokenization" with additional delegated subsystem semiotic actions.]

Semiotic Foundations of Interface Design Principles

  • Janet Murray, Inventing the Medium: Principles of Interaction Design as a Cultural Practice. Cambridge, MA: MIT Press, 2012. Excerpts from Introduction and Chapter 2. For reference: Glossary of Terms.
    • This book is an excellent recent statement of the contemporary design principles developed in the cognitive design tradition, which assumes that computer interfaces are designs for semiotic systems.
    • Murray explains four key affordances of digital interactive interfaces. Follow her explanations for how the design concepts for computational and media interactions are operationalized (i.e., made routine for use) in actual implemented interface designs. Complete her view with the fuller description of interfaces as metamedia that project connecting points (nodes) for the meaning systems used in representations.

Important Background on History and Concepts (read over next two weeks)

  • Martin Campbell-Kelly and William Aspray. Computer: A History of The Information Machine. 3rd ed. Boulder, CO: Westview Press, 2014. Excerpts from Part 4.
    [Necessary historical background on the technologies (conceptual viewpoint) we now take for granted in our contemporary understanding of "medium": the Personal Computer, the Internet, and World Wide Web.]
  • Peter J. Denning and Craig H. Martell. Great Principles of Computing. Cambridge, MA: The MIT Press, 2015, chapters 7 (Memory) and 9 (Design). [Concluding background on computer system design.]

Weekly Writing Assignment (link to Wordpress site)
Reflecting on your reading over the past few weeks, develop you thinking on these points:

  • First, using the concepts and methods from the readings (and any connections with prior weeks), describe the important concepts and supporting technologies that enabled computing to become the mediating/ mediated/ metamedia platforms that we take for granted in our everyday symbolic-cognitive technologies?
  • Second, what ideas from Alan Kay's design concepts for the graphical interface, computers as learning aids, and user access to software have yet to be implemented? Could you propose an "interface" feature or function that you would like to see in our PCs and mobile devices (including any suggestions on what it might take to implement it)?

Learning Objectives and Main Topics:

Merging the learning throughout the seminar weeks on sign systems, distributed cognition and agency, computation, and mediation in symbolic-cognitive processes. Semiotics can be expanded to describe how computation, software, and programming work as symbolic-cognitive artefacts, and thus help reveal that computation is not external to meaning (as described in an information theory model) but is a set of methods for abstraction and symbolic reflexivity that are co-constitutive in the creation of meaning (in material/conceptual forms) parallel to other sign processes but at automated orders of symbolic abstraction.

Synthesizing Topics and Questions for Semiotics and Computing

  • Computational devices (large or small) are intentionally designed cognitive-symbolic artefacts with structures and affordances for extending and distributing human collective cognition and agency.
  • The physically implementable designs for digitization (on Information theoretic principles) and computing processes are in service of, and motivated by, the uses of one or more of our sign/symbol systems (e.g., language, mathematics, graphical systems, images, sounds, mixed media systems).
  • Semiotic principles for representations in physical and material sign components as produced in/for computational systems (signals to symbolic structures in visual and audio media symbol systems).
  • Semiotic foundations of digital metamedia interfaces: interfaces to meaning systems and interfaces to actions in software, digital media objects, and the physical computing system.
  • Semiotic foundations of computation in software, programming (code), and computing system architecture.

Macro-level topics for synthesizing ideas and approaches:

  • The symbolic foundations of abstraction, reflexivity, and meta processes. Computation, programming, software, digitization, and digital interfaces employ the reflexive, abstractive, meta structures that appear to be "built-in" (constitutive) structural features of sign systems and their functions in symbolic cognition.
  • Kinds of symbol systems and symbol processes. While computation (software) implemented in a computer (actual physical hardware) is widely accepted as symbolic or (literally) as a symbol system, we have no consensus on how to clearly describe what kind of symbol process this is. All parties acknowledge that this is a different use of terms from the generative meaning process models in Peirce (semiosis) or in cognitive and generative linguistics. We know computation works because it is instrumentalized all around us. How and why it works as a symbolic system is as difficult to describe and explain as natural language.
  • Information systems and/vs. meaning systems. Understanding ways to describe computation, software, and all interfaces for digital media as a set of methods for handling the physical structures of "information" patterns as token structures for meaning systems and human intentions established outside computation. Digital Information designs and techniques as a semiotic subsystem for computable and digitally transmittable representations.

This Week's Seminar Discussion:

We will use this week's class meeting like a lab to go over ways to combine the concepts and methods we have been studying with a session on semiotics and/of computation and digital media.

Readings:

Models of computation:

  • Peter Denning, "What Is Computation?" Originally published in Ubiquity (ACM), August 26, 2010, and republished as "Opening Statement: What Is Computation?" The Computer Journal 55, no. 7 (July 1, 2012): 805-10. Note the important revised definition of "computation" by leaders in computer science (a tacit or implicit semiotic definition):
    • "The computational model of representation-transformation refocuses the definition of computation from computers to information processes. This model shows that representations are more fundamental than computers because representations appear in many situations where no computer is present."
      "This is actually a fundamental shift. It relinquishes the early idea that “computer science is the study of phenomena surrounding computers” and emphasizes that “computer
      science is the study of information processes”. Computers are a means to implement some information processes." (pp.808-809)
  • Herbert A. Simon, The Sciences of the Artificial. Cambridge, MA: MIT Press, 1996. Excerpt (11 pp.).
    Section on Computers as Artefacts and Symbol Systems in the "Newell-Simon" theory.
    [This concept of "physical symbol systems" (by a founder of a prominent school of thought in AI) is different from Peirce and cognitive linguistics (explain why), but important to know about for comparing models or merging models and descriptions.]
  • Peter Andersen, et al., eds, The Computer as Medium. Cambridge: Cambridge University Press, 1993. Excerpts.
  • Janos J. Sarbo, Józef I. Farkas, and Auke J. J. van Breeman. Knowledge in Formation: A Computational Theory of Interpretation. Heidelberg; New York: Springer, 2011. Selections.
    [Read the Introduction and survey Chapters 1 and 3. This book is an excellent overview of some current applications of Peirce's models of signs, interpretation, and logic to computational problems (as in NLP, Natural Language Processing, and approaches to modeling conceptual and semantic information). When you survey the Table of Contents, you'll see the range of applications developed by these authors working at the intersection of computing and semiotics.]

Optional and Supplemental: For Research and Reference

  • Matthias Scheutz, ed. Computationalism: New Directions. Cambridge, MA: MIT Press, 2002.
    This is an excellent collection of essays by major thinkers in computer science. Many take up the issue of computation as a symbolic process, and other key questions about computation and meaning.
    Read around to get a sense of the issues covered, but especially the essays by: Brian Cantwell Smith, Philip Agre, and John Haugeland. A good launchpad for further research.
  • Bibliography: Semiotics and Computation [for continuing your research]
    • Many texts in the bibliography are available in the etext library (GU student login only).

For Class Discussion:

Weekly Writing Assignment (link to Wordpress site)
Use this topic to focus your thoughts and questions about the readings for this week:

  • We’ve seen examples of how people in many sciences and disciplines are beginning to re-envision the meaning of computing, information, and all things digital in the larger, more integrative context of human social symbolic-cognitive life. The recent convergence of research and theory in linguistics, cognitive science, computing, information and communication theory, and semiotics around the core questions of the function of human sign and symbol systems and symbolic cognition in general opens up new knowledge and new research paths.
  • Though we have only touched on a small (but I hope representative) sample of these developments, can you describe how learning in this context of approaches has contributed to your own understanding and insights about human meaning systems and the technologies designed as cognitive-symbolic artefacts? Has anything changed or become clearer in your own thinking and motivation as a result of seeing how you belong in this larger, more inclusive picture of communication, culture, and technology?

Learning Objective and Main Topics:

Learning how to apply and extend the concepts and methods of the seminar to examples of mediations for meaning systems.

Concluding Case Study:

The Google Cultural Institute and Google Art Project as a "Meta" Mediation and Representational Project:

Consider all the ways to talk about the "Google Arts & Culture" platform as a project for mediating a complex semiotic system. How many layers of interfaces are involved? How can the platform function as an interface to the meaning system(s) that it is described as mediating? Where is the agency and connection to collective symbolic cognition for complex cultural categories?

Google seems to present the Google Art Project and related cultural projects as technology implementations. But what about all that's being mediated as the precondition for using the technology this way? Can Google (the company, the technologies) re-mediate the "museum" or "cultural transmission" function? What are the parallels and differences with the Google Books project?

We will also consider the longer history of meta-media interfaces to art, art history, and the system of art genres and art concepts which form the background of contemporary digital media re-mediations.

 

Websites:

Background on Project and Technologies:

Readings:

  • In our Key Texts in Semiotics and Technology Reader:
    • Roman Jakobson on communication and codes. Section 4, pp. 20-21.
    • V. N. Volosinov, on Signs in Society and Ideology. Section 8, pp. 30-34.
    • Umberto Eco on Encyclopedic levels of meaning: "Dictionary" and/vs. "Encyclopedia." Section 13, pp. 42-44.
    • Read these background texts to become familiar with other semiotic terminology that we will use in this week's case study discussions.
  • Interpreting artworks and art history with metamedia interfaces, 17th century to today:
    • Martin Irvine, "André Malraux, La Musée Imaginaire (The Museum Idea) and Interfaces to Art". An introduction to major "interface" concepts for art history before computing, with excerpts from Malraux's text.
    • The English translation of the first chapter of Malraux's The Voices of Silence (1953) was unfortunately first done as "The Museum Without Walls." Malraux was addressing the question of the governing concepts of the museum and the photographic technology used to mediate the institutional function of art history and access to cultural knowledge. There is a long tradition of developing and questioning mediations, representations, and interfaces to art and cultural artefacts, and will explore some of them as background for our contemporary projects.
    • Notice how the photographic reproductions of artworks function on different levels as icons, indices, and symbols (in Peirce's terms). The token instances exemplify their types. The iconic function is seen in the correlations to genres and categories that they stand for, and are also used as indices, that is, references to actual, physical artefacts located somewhere (in a museum). The whole presentation is symbolic (in Peirce's term) in providing a representation, the meaning of which occurs only in further signs and symbols (descriptions, interpretations, relations and patterns with other works, etc.).
    • All the mediating functions that Malraux describes continue in contemporary metamedia interface technologies. All Internet and Web-based presentations are based on photographic reproductions in which artefacts are detached from historical and cultural contexts.
    • Optional/Supplementary: Martin Irvine, "From Samuel Morse to the Google Art Project: Metamedia, and Art Interfaces" (more in-depth analysis of semiotic interfaces).

Presentations (For Conceptual Review and Applications)

For Class Discussion:

Weekly Writing Assignment (link to Wordpress site)
In class: Group discussion

  • The purpose of this concluding group discussion is to allow you to combine and synthesize the interdisciplinary approaches, concepts, and methods that we've studied, and apply them to a complex case of technical mediation designs for interfaces to meaning systems. The Google Art Project (Google Arts & Culture) is only one instance among many of a common question and design problem for mediating and creating interface to the symbolic artefacts that cultural groups interpret with many layers of meanings and values. Try to weave together as many of the key ideas and ways of approaching semiotics and cognitive technologies that you find implemented (or not and remain problematic) in this specific case; consider, for example:
    • the array of sign systems involved, as interfaces to systems of meaning "outside" the represented tokens and beyond the physical-perceptible features of the interface,
    • externalized, distributed and collective symbolic cognition,
    • the features and functions of technical mediation (in a history and continuum),
    • presentational substrates (signs and symbols require structured physical/material substrates),
    • the mediating and interpreting function of software, data algorithms, digital information structures for memory and transmission
    • the mediation of both kinds of agency: offloading and directing symbolic intention and action, and action or agency delegated to artefacts, media, and technologies
    • the use (or problematic neglect) of interface principles from Engelbart and Kay to the summary in Janet Murray's Inventing the Medium,
    • the presentation and organization of symbolic structures in the two levels of the interface itself: how does the interface in the sense of the front-facing perceptible view of the platform in the screen project that we observe and interpret features function as an interface (2) to the meaning systems that motivated it and in which it participates as a connecting node?
    • The major question: all cultural artefacts function in levels and networks of encyclopedic-level knowledge and symbolic correspondences. Can this kind of technical platform enable discovery of the kinds and categories of Interpretants (in Peirce's sense) that members of the interpretive community assume are relevant and necessary for understanding the meaning, value, and function of an artefact/artwork?

Instructions and Roundtable Discussion of Final Projects

  • General Instructions for Final Projects: on the Wordpress site.
  • In-class discussion of projects: we will have a roundtable discussion of your current state thinking and research, and a chance to get feedback and suggestions from the class.
  • Final projects are due to be posted as a Wordpress essay 10 days after the last day of class.